Authenticity, Integrity, Retrievability

Standard: Referential Integrity

Agencies must put in place systems to:

• Maintain the context of data elements in database structures

• Maintain relationships between documents in a set

Supports Policies: Authenticity, Integrity and Retrievability

Scope and Interpretation

Within a database referential integrity implies that defined relationships between data elements and data structures are maintained when data content is added, updated or deleted. Rules to achieve this should be enforced by the database management system wherever possible. In addition, appropriate edit rules and data entry screens must be defined during system definition and enforced by the application system maintaining the data.

At a user level referential integrity is more likely to be maintained when data elements on screens are appropriately named. Staff training programmes must ensure that users understand the meaning of the data elements they work with as represented in the application system.

Documents in a set need to be retrievable as a group if required. This could be done through a classification scheme, metadata or simply by keeping the documents physically (or logically - say in a common directory) together.

In a document store referential integrity must be maintained within the metadata environment as with any database.

Aids to maintaining referential integrity include:

• Planned, thorough technical system and user testing of all new application code that could add or update data elements

• Adequate procedures to discover and recover from hardware or software data corruption

• Planned thorough technical evaluation of system and data or document management software upgrades before they enter production

• Protection of live data in the production environment through adequate security and migration procedures. The Physical Custodian must fully control access to production data and systems and the movement of data and software from testing environments to production.

Rationale

If referential integrity within a database breaks down then the data content quickly becomes unusable.

Loss of context within a document set or store will quickly lead to unreliable retrieval and the store will no longer be viable.

Any ambiguity about what a particular data element represents will result in unreliable data, since the meaning may change depending on who entered it.

Standard: Integrity of Application Software

The integrity of any application software operating on approved data and document stores will be monitored at appropriate intervals, and action taken to repair and prevent defects.

Supports Policies: Authenticity, Integrity and Retrievability

Scope and Interpretation

Agencies must be able to demonstrate that software operating on a data or document store can maintain the integrity of the contents including:

• Maintain data element content and interact with the database management system to ensure correct data-relationships are also maintained

• Implement business rules as defined by configuration ( e.g. security, retention)

• Retain profile data and history of the document object

• Retrieve data and documents accurately and consistently

• Report accurately and consistently

Rationale

The agency must ensure that the software used to access and maintain prime authoritative data sources is adequately tested. Otherwise the integrity of the source will always be in doubt.

Standard: Integrity of configuration

The configuration of each agency's approved data and document stores will be regularly reviewed and updated to ensure it still meets business needs.

Supports Policies: Authenticity, Integrity and Retrievability

Scope and Interpretation

Configuration can apply to both manual and automated systems and includes:

• Adequate design for capturing metadata

• Consistency and relevance of classification

• Consistency and relevance of validation sets

• Business rules such as retention, disposal, access rights, security-classification and privacy

• Exceptions and any requirements for changes to business rules

• Verification at intervals as determined by the Business Custodian

• For automated system, adherence to industry best practice and government IT standards for technical configuration management of hardware and software

Rationale

As business requirements evolve it is essential to regularly review the initial and ongoing configuration, both at business and technical levels, of the systems managing data and document stores. Otherwise the agency risks serious loss of integrity over time in its data and document assets.

Standard: Integrity of content

To optimise the integrity of data and metadata:

• System data and metadata capture must be maximised, and user input must be minimised

• Where users enter data into a data store, or create metadata in a document store, validation at the time of input is required wherever practical

• Processes must be in place to monitor and correct errors in the data and metadata

• Any changes to the use of a of a data or metadata field design purpose must be agreed with the Business Custodian and documented and effects on downstream systems taken into account

Supports Policies: Authenticity, Integrity and Retrievability

Scope and Interpretation

Systems must be designed to minimise user input, and maximise the validity of remaining user input. Examples include: use of data known to the system e.g. date and time of user input, use of standard pick lists, or data derived from user actions on the system.

National or international standard pick lists should be used where these exist e.g. list of countries, list of occupations etc. Currently recognised standards could be available via NZGO. Where an agency deviates from the standard list the additions and substitutions must be documented.

Document metadata validation should be by the use of pick lists or controlled vocabularies for metadata fields and by spell checking on fields where pick lists are not feasible.

The internal content of documents should be to standards required by the agency, e.g. any agency style guide, use of templates/layouts, correct spelling, use of terms and abbreviations etc. Users must spell check the contents of their documents, and agency workstations should all be set to a standard dictionary, ideally New Zealand English (or failing that, Australian English). Only official abbreviations or contractions should be used, from a discrete set maintained by the agency and/or an external government approved source. The gazetted list of New Zealand place names is available from Land Information New Zealand.

Care must be taken in the use of validation tools to ensure they are fit for use by those entering data. Public access via e-government initiatives may allow the public to enter data directly into some systems.

Quality standards must be regularly monitored for compliance, and updated as required.

Each agency must develop or adopt data monitoring standards and obtain regular samples.

Changes to terminology should be recognised and incorporated into the data and document retrieval systems, so that retrieval requests are accurate and comprehensive.

Rationale

Validation increases accuracy, reduces opportunities for errors, and improves retrieval and implementation of business rules such as security and retention. Unauthorised and undocumented alterations to the usage of data and metadata fields pose a high risk of corruption and loss of both current and historical data.

Consistent application of data quality standards will result in a high level of success when searching for and retrieving information from document stores. Similar benefits are gained when searching for or manipulating data in data stores, or interchanging data with another agency.

Use of standard pick lists ensures better opportunities for matching within and between agencies.

Standard: Integrity of Process

Agencies must be able to demonstrate that their processes do capture required data elements and business documents, that business rules and standard operating procedures are in place for their management, and that they are implemented.

Supports Policies: Authenticity, Integrity and Retrievability

Scope and Interpretation

This covers the whole broad area of defining standards, developing guidelines, providing training and demonstrating that staff members comply with those guidelines. See also Standards: Skills and training, Individual responsibilities, Process maps.

Rationale

In order to demonstrate the authenticity of data element or document content, agencies must be able to show that the processes involved in creating and maintaining content deliver to business and statutory requirements.

Standard: Skills and training

Staff will be trained in their responsibilities when working with Crown data and document assets. These responsibilities will be written or referred to in key agency documents such as Job Descriptions and Performance Agreements, for staff at all levels.

Supports Policies: Authenticity, Integrity and Retrievability; Document identification and capture

Scope and Interpretation

For each agency to implement effective data and document management, individual users must understand and accept responsibility for applying key principles, rather then just work on a "step-by-step" basis.

Training on data and document management should be provided for existing staff and should be part of the orientation/induction process for new staff. Training includes:

• Reasons why data and business documents are important to the Crown

• Reasons why careful management is required

• The correct entry and usage of data in database systems

• The appropriate management of business documents

• Access to up-to-date manuals and guidelines

• Education in the importance of generic legislation e.g. the Official Information Act, the privacy Act, and legislation specific to the organisation

• Education in the importance of the Treaty of Waitangi, its relevance and practical application to the capture and management of data and documents.

The inclusion of data and document management responsibilities in documents such as Job Descriptions and Performance Agreements is designed to ensure all employees understand that this is a key component of their work. These responsibilities should also part of the "rules of the house", meaning the core rules on how employees will conduct themselves in an agency. See also Standards: Discovery implications.

Rationale

Staff must have the necessary knowledge and skills to do the task. It is important to ensure that appropriate training and documentation are available when and where needed.

Standard: Version Control

Agencies will determine business rules for version control of data elements, business documents, and data sets. Rules will be built into systems or expressed as guidelines for users.

Supports Policies: Authenticity, Integrity and Retrievability; Auditability

Scope and Interpretation

Each agency must build guidelines on when versions should be created, and users must create versions according to those guidelines. The focus of the guidelines should be on capturing evidence of business transactions. See also Standards: Integrity of process.

Versions of individual data elements are not necessarily held on operational systems however they may be retained in a time series within a data warehouse. At least some of the versions of a document that are generated during its development will normally need to be saved and identified as part of its history.

Data and document stores would not normally be versioned as a whole unless a migration to a different environment is required. Individual data sets may require a version history when used as a basis to generate other material e.g. a policy document.

Approved document stores must permit the user to generate documents that can be treated as versions of the same document. Each version will have its own unique identifier that might be generated or user entered free text. Users should create new versions of documents where there is substantive change to the document, typically linked to a defined process.

Some versions may become "read only".

Guidelines will aid users in deciding when to create a new version. Guidelines will tell users whether the prime authoritative data source is to be kept in electronic or physical form.

Each agency must have available all required versions of a data element, business document or dataset, and the status of each version will be known. See also Standard: Process maps.

Rationale

Version control assists historical evidence of business transactions to be maintained over time and is essential where this is a statutory requirement.

As data and documents are created and modified, the changes themselves provide additional evidence (e.g. development of policy, published version).

Reduces risks of wrong versions being referred to, and ensures correct versions can be identified and retained.

Standard: Document Templates

Agencies will develop and maintain a set of standard document templates and styles for their document types.

Supports Policies: Authenticity, Integrity and Retrievability

Scope and Interpretation

These templates will:

• Assist each agency to standardise information capture

• Be universally available and up to date

• Be clearly labelled

Each agency must ensure standard document templates are designed and made available, and audit their usefulness and usage. The focus is on frequently used documents like letters, memos, web pages. and distributed widely e.g. within the agency, to the public, to the web or intranet. The template can also address issues of suitability for use in different processes, eg the effect of shading/colour on scanning.

NZGO could be a repository for a set of generic optional templates which agencies could modify for their own use. Such templates would need to conform to open standards and not be restricted to proprietary software.

Rationale

Templates ensure that standard business documents are quicker and easier to create and edit, whilst consistently conforming to agency standards. This and reduces the risk of inappropriately formatted documents in use, reduces the need for users to have advanced formatting skills, ensures consistent look and feel and use of styles, and improves recognition.

Templates ensure that documents are partially structured, and therefore improved for use by search engines and knowledge management tools.

Standard: Retrievability

Data element content and business documents will be retrievable in formats that meet open international standards.

Supports Policies: Authenticity, Integrity and Retrievability

Scope and Interpretation

Data elements and documents will be stored in an easily accessible format for their retention periods, and if necessary migrated to other storage media and/or software to maintain their accessibility.

Easily accessible formats conform to open standards and are software independent. Practically the choice will usually fall to current industry standards supported by multiple vendors to ensure contestability and future choice. See E-Government IS Policies and Standards.

Changes to low level formats controlled by operating systems and storage hardware are generally infrequent and can be managed through standard migration processes, provided open rather than proprietary technologies are used.

For electronic documents:

• Effective and efficient searching and retrieving tools will be incorporated as part of the approved document store design

• All documents saved into an approved document store can be retrieved through that document store, regardless of where they are currently stored (location on the network, on-line, near-line and off-line)

• Agency rules must control the retention of documents, even if processes external to the approved document store actually implement the deletion or transfer of documents to other storage media

Acceptable speed of response will be stated in agreements between the Business Custodian and the Physical Custodian.

Rationale

There is little point in preserving data or documents if they cannot be easily retrieved in a usable format. The use of open industry standard formats for storage and retrieval mitigates the risk of expensive recovery being needed to retain the use of older information.

[ Previous | Next ]