Skip to content.
|Networking government in New Zealand.
You are here: Home » Standards » Interoperability (e-GIF) » Best Practice

Exchanging Data Between Agencies

Ensuring e-GIF, XML and Legal Compliance: A Checklist

Context

The current version of this document is Version 6.1, released in January 2007.

1.1 Purpose

The purpose of this document is to provide a checklist of technical considerations for people developing systems that involve data exchange between agencies and the associated agreements. Typical written agreements could take the form of: Data Sharing Agreement (DSA); Memorandum of Understanding (MOU); Information Sharing Agreement (ISA); and Data Matching Agreement (DMA).

The e-Government Interoperability Framework (e-GIF) specifies that exchange files will be prepared using EXtensible Markup Language (XML) technologies. Agreeing the technical issues will help parties who exchange data to understand the scope, nature and responsibilities (individually and collectively) for the transaction.

The ultimate objective of an agreement to share or exchange data is to protect the intentions of, and limit the purpose for, the data exchange process itself. The objective of using XML and an exchange standard is to support interoperability, with ‘no surprises’ for the receiving party.

1.2 Audience

The target audience of this document is primarily Project Managers, Business Analysts and the Technical Teams who implement business processes involving data exchange on behalf of a Business Unit Manager.

Checklist: Items to Consider

2.1 Business Purpose of the Data Exchange

This document is not intended to cover business or project management issues. Suffice to say that all the usual project risks are present and amplified by the need for two or more agencies to work together. The presence of strong business sponsors in each agency will be essential for success.

2.2 Legal Authority

It is important to determine if there is any legal impediment to the data exchange. This could be related to privacy, contractual, security or other issues. Legal advice should be sought to consider the following:

  • If legislation administered by the agency authorises the exchange
  • If existing contract or other legislative provisions apply
  • If the exchange is authorised by an existing provision of the Privacy Act (1993), or falls within the exception provisions
  • The impacts on personal privacy. Here, the completion of a Privacy Impact Assessment or Information Matching Privacy Impact Assessment (IMPIA) may be useful.

Where no existing legislative provision exists, then the legal opinion should consider:

  • Is personal information involved and what are the implications?
  • Is there a need for legislative authority?
  • If so, what form the legislation should take
  • Will the information disclosure be for the purposes of an information matching programme?

2.2.1 Personal Information

Cabinet has mandated that the Office of the Privacy Commissioner must be consulted on any proposed legislation involving disclosure of personal information http://www.dpmc.govt.nz/cabinet/manual/index.html.

If in doubt about privacy or legal authority issues relating to a data exchange, you can find out more about the subject here: http://www.privacy.org.nz/ .

Related risks associated with data exchange implementations include:

  • Identifying the wrong individual in a data match and taking action against them – there are examples of people with the same name and birth date being repeatedly confused. Resulting damage to the reputation of both the individual and the agency can be serious.
  • Unreasonable invasion of privacy even when there appears to be legal authority
  • Propagating errors through to other databases.

It is essential that these risks are managed through appropriate legal advice and through detailed project plans and application testing.

2.3 Data management Policies

Good practice guidelines are laid out in the New Zealand Government Data Management Policies and their related standards: http://www.e.govt.nz/standards/e-gif/data-management-policies. They are useful tools that can be applied to all data management including data exchange programmes.

The policies set a clear framework for all aspects of data management in government. Here the Interchange, Replication and Interfaces policy is perhaps the most relevant, but others, such as those dealing with ownership, stewardship and custodianship of data resources, will also assist when preparing agreements between agencies.

2.4 e-GIF Standards and Name and Address

In government, the current principal standard for XML - personal data exchange is eXtensible Name and Address Language (xNAL). The e-Government Interoperability Framework (e-GIF) requires Public Service departments, when upgrading or acquiring new systems or business processes involving name-and-address exchange, to use xNAL: Organisation for the Advancement of Structured Information Standards (OASIS) xNAL Version 2 or, if possible, its New Zealand profile, xNAL (NZ).

The SSC ICT Branch holds regular workshops on request, to help agencies prepare for using xNAL. This document was drafted in response to requests from xNAL-workshop participants and from Local Government. The checklist technical detail is based primarily on using xNAL, but is largely applicable to other data standards.

Transport and messaging standards also exist in the e-GIF for many of the examples listed below under that heading.

2.5 Data File Issues

With regard to the data files exchanged, it is important to agree on:

  • File Format - The content, format and information model, including length, formatting of elements, and structure for more complex types.
  • A Schema Language and Parsing Protocols
  • To parse - Before sending; or if not, when and what fragments can be sent unparsed or partially parsed
  • Controlled Value Lists (CVLs) - Which are vocabulary or code lists
  • Schemas - Including all possible elements and attributes – for example, it is essential to specify precise XML Path Language (XPATH) locations of elements that can be stored in multiple XPATH locations
  • Each element or tag’s data - With agreed sample XML files or file fragments from all parties – free-text elements containing proprietary data are excellent examples
  • Free-text fields’ data - Typically supplementary information, these can also house data when there is no feasible alternative; for example in xNAL: NameLine, AddressLine, Supplementary, Unparsed, any#other
    Avoid, if at all possible, the use of elements and attributes that cannot be resolved by popular parsers.
  • Syntax - For information outside of the scope of the standards-compliant parts of the file – many data files contain proprietary as well as standard structure and data; best current practice suggests splitting the proprietary data structure from the XML-schema structure inside the file record in order to better interpret errors exposed by parser and validation tools
  • Namespace – Agree and declare an unambiguous Namespace for elements and attributes in the data file.

2.6 Transport and Messaging

Typical data-exchange scenarios include:

  • Direct access to the file when copied to portable media
  • Direct connection to a database for export/import of data to a web service
  • Programmatic access via an agreed Application Program Interface (API)
  • Access to download a file
  • Direct access to partner application, e.g. terminal access.

When setting up an exchange, it is important to consider:

  • Security - Ensure the confidentiality of data throughout the process of an information exchange.
  • Authentication - User or partner identity is recognised and accepted.
  • Authorisation - The process of determining what types of activities are permitted. Usually, authorisation is in the context of authentication; once a user is authenticated, they may be authorised for different types of access or activity.
  • Non-Repudiation - A feature of a system that provides objective evidence that a transaction occurred.
  • Reliable Delivery - A feature to ensure that a message is delivered to a partner in a predictable way, and that the message is only received by the partner and is processed once.
  • Message Neutrality - The protocol does not place any constraints on the types of data or message that can be conveyed, e.g. Binary information or different character sets.
  • Message Scalability - The ability to add additional message exchanges without changes to the established pipeline.
  • Partner Scalability - The ability to add further partners in a seamless manner.
  • Real Time Messaging - The ability to participate in a business conversation with immediate results.
  • Compression - The ability to reduce total size of the communication while preserving the information exchange.

2.6.1 Scope of Agreement

The circumstances and rules for each of these data access scenarios must be agreed in writing. With regard to XML-file transport and messaging, it is important to agree specified versions of:

  • the message transport, such as:
    • ebXML Messaging Service (ebMS)
    • Simple Object Access Protocol (SOAP)
    • Secure Electronic Environment (SEE)
    • Simple Mail Transfer Protocol (SMTP).

www.e.govt.nz/standards/e-gif/e-gif-v-3-1/standards/chapter3.html

  • the email file attachment standard, such as:
    • Multipurpose Internet Mail Extensions (MIME)
    • Secure/Multipurpose Internet Mail Extensions (S/MIME).

http://www.e.govt.nz/services/see

www.e.govt.nz/standards/e-gif/e-gif-v-3-1/standards/chapter3.html

  • the file transfer standard, such as:
    • File Transfer Protocol (FTP)
    • Hypertext Transfer Protocol (HTTP)
    • Transaction Level Security (TLS).

www.e.govt.nz/standards/e-gif/e-gif-v-3-1/standards/chapter3.html

  • the security protocol, such as:
    • Hypertext Transfer Protocol Secure sockets (HTTPS)
    • Transaction Level Security (TLS).
    • Secure Sockets Layer (SSL)
    • Secure/Multipurpose Internet Mail Extensions (S/MIME).

www.e.govt.nz/standards/e-gif/e-gif-v-3-1/standards/chapter3.html

  • the file compression standard, such as:
    • zip
    • GNUzip (gzip),
    • Tape Archive (TAR),
    • tape archive gzip (TAR.GZ).

www.e.govt.nz/standards/e-gif/e-gif-v-3-1/standards/chapter3.html

2.7 Implementation Issues

With regard to the implementation, it is important to agree on:

  • Compliance testing - Including the process and procedure for testing a sample file against an agreed set of data and parsing mechanisms
  • Pilot implementation - That repeatedly tests agreed datasets against the interoperability objectives.
  • Change-control process - So that all parties know of any changes before a data exchange.
  • Levels of support - That each partner will provide including contacts and operational procedures.
  • Auditing processes - For both parties, given the sensitive nature of personal data.
  • Intellectual property rights (IPR) - When modifying and versioning schemas, for example to optimise re-use in a particular sector (for government agencies, all IPR belongs to the Crown).
  • Promoting modified schemas - For other agencies to access and reference, using namespace declarations chosen for persistence and uniqueness.
  • Governance process - To ensure that files are reviewed regularly to accommodate business needs (which may have changed) and that any modifications are versioned and re-released.
  • Charging to recover costs - Should be avoided wherever possible as it adds overhead and can become an unhelpful source of argument. Instead, agreement should be reached on the proportions of development and maintenance costs to be borne by each agency. Often, a simple commitment to pay costs as they fall and to consult and agree on any changes to data or file formats, which may incur maintenance costs, will be sufficient.
  • Timeliness/frequency and tolerance range - Of the data exchange.
  • Non functional requirements (NFR) - This has a direct impact on, and is constrained by, partner implementation limits and partner’s ability to implement. For example, a parser may have file size limitations, or a partner may only be able to implement a certain type of parser due to external factors like security or operating environment.

2.8 General Considerations

  • Quality of information – There may be a mismatch of requirements if the relative value of the data to each agency involved in the exchange is different. For example, there may be limited validation applied to the original source data, or it may not be possible to directly match records by key value.
  • Trust - Quality issues may lead to a down grade of the level of trust in the data and any services that depend on it.
  • Security – Perceptions of security risk may differ between the partners in a data exchange. By connecting systems within the infrastructure of each agency, overall security becomes a function of all partners’ security. It is important to align security expectations and employ technological solutions that meet the needs of all parties.
  • Prime Data Source - Most information collected by Government Agencies has a natural home based on agency function. While sometimes it may not be possible to go direct to that agency for technology or relationship reasons, it is normally better to do so, as it is likely to have the most up to date and accurate information.
  • Business process - The total end to end business process across partners, of which the data exchange is a part, should be mapped out to define the best final solution. This can lead to process enhancements, such as real time and customer self service and may reduce the amount of information that needs to be transferred between the agencies.

References

e-GIF References:

Legal Compliance

Data Exchange Checklist

What is the Business Purpose?

Risk: High

The business purpose and rationale behind establishing a MOU or Data-Share agreement must be clearly articulated and understood. You must be able to account for data privacy and security issues and have clear business rules about all the facets of the data exchanged.

Is there an Alternative Means of Achieving the Same Purpose?

Risk: High

Is there an alternative way of meeting the business requirement without exchanging data between agencies? You must consider all options before proceeding with agreements to share data. It is a very good idea to test the value of a data exchange before proceeding with formal agreements. This can be done by processing sample data and verifying expected results. It may be that the theoretical value is not realised in practice.

What are theLegal Compliance Risks?

Risk: High

Legal compliance can be a complex field and you will need to clearly establish the basis for your data exchange. The risk can be managed by seeking competent legal advice, which will typically cover the following:

  • Are there contractual or other legal impediments?
  • Does legislation administered by the agency authorise the exchange, and if not, is there a need for legislative authority, and what form should it take?
  • Is the exchange authorised by an existing provision of the Privacy Act or does it fall within the exception provisions?
  • The impacts on personal privacy?
  • Will the information disclosure be for the purposes of an information matching programme?

Once the legal position has been established, implementation risks need to be managed through the project plan. Typical risks include:

  • The probability of identity mismatch and the seriousness of the consequences
  • The chances of the data exchange being perceived as an unreasonable invasion of privacy
  • The propagation of data errors from one system to another.

Data Management Policies

Risk: Medium

Data exchanges should comply with best practice data management standards and policies to ensure the security and protection of the data. All data collected by Government Agencies is owned by the Crown, so the Stewardship obligations undertaken by agencies must be exercised responsibly at all times.

e-GIF Standards

Risk: Medium

Relevant e-GIF standards exist for:

  • EXtensible Markup Language (XML)
  • Transport and Messaging
  • Customer Information (primarily name and address) – xNAL

Data File Issues

Risk: Low

Agreements need to be reached about the following:

  • File Format
  • Which Schema Language and Parsing Protocols
  • When to parse
  • Content of controlled value lists
  • Schemas
  • Element data
  • Free text fields
  • Syntax
  • Namespace

Transport and Messaging

Risk: Low

Choose appropriate protocols from the standards recommended by the e-GIF, and consider:

  • Security
  • Authentication
  • Authorisation
  • Non-Repudiation
  • Reliable Delivery
  • Message Neutrality
  • Message Scalability
  • Partner Scalability
  • Real Time Messaging
  • Compression

Implementation Issues

Risk: Medium

A range of issues need to be considered including:

  • Compliance testing
  • A pilot implementation
  • Change control
  • Levels of Support
  • Auditing
  • Intellectual property rights
  • Schema modification over time
  • Overall governance process
  • Cost recovery
  • Timeliness/Frequency and Tolerance Range
  • Non-Functional Requirements

General Considerations

Risk: Medium

General considerations which should not be missed:

  • Quality of information
  • Trust
  • Security
  • Prime Data Source
  • Business Process