Interchange, Replication, Interfaces

Standard: Interchange Agreement

Agencies will define requirements and develop agreements before sharing data between primary authoritative data sources, whether those sources are internal or external to the agency.

Supports Policies: Interchange, replication and interfaces

Scope and Interpretation

Interchange may involve data transfer/duplication or data access at a system level. It must involve two or more prime authoritative data sources and may be one or more data flows.

Where similar data is collected from various sources into a new combined data source, this is considered to be replication in the manner of a data warehouse. See Standards: Replication.

Ongoing interchange arrangements require a single agreement but regular reviews of its operation must be included. One time interchanges still require an agreement to ensure the process is documented.

Agreements will include a clear statement of objectives and requirements, and will detail:

• The legislative environment that permits and/or limits the interchange

• The purpose of the interchange

• What data is included in the interchange

• A review of how fit the data is for that purpose

• Quality requirements

• Security requirements

• A review schedule, if the interchange is repeated over time

• Notification of any changes to the either the primary authoritative data source or to the interchange use

Agreements between agencies require the approval of the Crown Data Steward. Agreements within an agency require the approval of the Business Custodian.

The Privacy Act applies to the interchange of personal information, particularly where the interchange involves data matching.

Rationale

Interchange agreements within agencies, however brief, will ensure that all interfaces are known to the Business Custodian and documented accordingly. Externally the same applies, so that the Crown Data Steward is aware of all interchanges and all are adequately documented.

Standard: Replication

Replication of data or documents will be controlled by the Business Custodian(s) involved and will only come from prime authoritative data sources. All replication arrangements will be auditable to ensure that a true replica is made.

Supports Policies: Interchange, replication and interfaces

Scope and Interpretation

Permanent replication of data or document store content will most frequently take place for disaster recovery purposes or where data is to be included in a data warehouse.

In some cases data may be usefully replicated from many distributed sources into a single source, that can be used as an aggregate in the manner of a data warehouse.

Permanent replication of data to servers in different geographic areas may sometimes be justified as a technical solution where network bandwidth is short. This kind of solution should be transparent to system users. Data or document stores physically segmented and replicated on the basis of business organisational structures are to be avoided. The excessive complexity inherent in such systems puts government data and document assets at unnecessary risk.

Temporary replication may be used for data matching programs where a snapshot copy of the required data is used for comparison with another database, and then discarded. It may also be used to check out documents or data to a local system for off-line work. In these cases the data or documents in the prime authoritative data source must be locked from update until the changes are checked in.

Temporary replication is also commonly used to populate databases within test and development systems from the production base. This must not be undertaken unless access rules followed e.g. person data could be encrypted or scrambled. See Policies: Access Rules

Replication places responsibility on the Business Custodian to ensure checks are in place to monitor the accuracy of the replication. This may include the periodic restore of a database from tape backup, or regular comparison of databases where one is replicated online by the other. When two agencies are involved the Business Custodians will co-operate to ensure a consistent approach. See also Standards: Interchange Agreement.

Wherever practical other methods of interchange should be used to avoid the build up of unnecessary duplicate data stores.

Rationale

Replication dramatically increases the risks of: data corruption, confusion between data sources, use of incomplete data, security breaches, confusion over retention and consequent increased potential for data loss. Replication of data and document stores therefore requires constant monitoring to ensure an accurate result. . It should only be undertaken to meet specific business requirements.

Standard: Interfaces

Electronic interfaces between systems must use mechanisms based on open industry standards as specified in the government information technology policies and standards.

Supports Policies: Interchange, replication and interfaces

Scope and Interpretation

Interfaces between systems need to be robust and exhibit the following characteristics:

• Use appropriate security for the data or documents being used

• Use standard communication protocols

• Use an encryption standard if a public network such as the internet is involved

• Not rely on creating duplicate data or document stores unless replication is the purpose of the interface

• Be transparent to system users

• Show that data and document integrity is retained

Rationale

Interfaces based on closed proprietary standards are likely to be more expensive to run and are prone to becoming obsolete. Use of standard open protocols will normally allow a wide range of cheaper systems and equipment to be used.

Using standard mechanisms will increase the ability of individual agencies to respond to whole-of-government initiatives.

Standard: Migration

Data and document stores will be constituted such that all content, structure and metadata can be migrated to a different environment without loss of integrity.

In the event of a migration or major upgrade, migration plans will be produced by the Physical Custodian and subject to approval by the Business Custodian.

Supports Policies: Interchange, replication and interfaces

Scope and Interpretation

Each agency must retain the ability to upgrade or migrate to a new platform/operating system or new application , or even to remove its databases or business documents and their contextual information from an application/store, without loss of information. Where the material is held electronically, migration should operate on a copy of the original dataset, so that the original dataset remains intact.

Documents migrated out of the current document store or DMS must still be linked to their metadata, either stand-alone or in another application. An approved document store design must permit transfer of documents to another storage repository complete with their metadata, with or without the application managing the current document store or DMS.

For any upgrade, migration or transfer, the process report will include alterations to relationships and content etc, including:

• Original data

• What data has changed

• What data has been added

• What changes are agreed

• Who decides

• Back-up and roll-back plans

• Documentation on the original system

• What security has been applied

Each agency must have standards for data and document management, conversions, and migrations, that are sufficiently robust to ensure that integrity of data is maintained or enhanced through migration processes. Agencies should take all reasonable steps to ensure the future integrity and accessibility of data and business documents, whether or not those data or documents were created within an agency or inherited from another agency.

Agencies must assess and select software to meet these standards, and ensure that any data migrations meet these standards. This standard also applies to manual systems.

Rationale

Migration of data carries with it the risk of loss of contextual information and/or change of some data or metadata and security permissions. Each agency needs to have in place standards and procedures to ensure that the integrity of data is maintained through system and/or platform migration.

This strategy assists an agency to be independent of any software provider. It reduces the risk of inability to upgrade software and of inability to access data or documents.

[ Previous | Next ]