Retention
Standard: Backup, Recovery & Restore
All agencies will have a backup regime for data and document stores to insure against system failure or human error. Backup operations will be regularly monitored for completeness and tested for retrievability. The regime will be developed and actioned by the Physical Custodian, and subject to approval by the Business Custodian.
Support Policies: Retention
Scope and Interpretation
This standard applies primarily to electronic data and document stores. Paper based document stores could be backed-up by copying documents and transferring them to another site. This would only be practical in special circumstances.
Back-up
Back-up schedules must be determined and applied to all data and electronic business documents held by government agencies. This will normally be done by specifying schedules for the various network database and application servers, and by ensuring that all data elements and documents fall to at least one of these.
The following sample schedule could be used for a network server:
- A full system and application software back-up to be stored securely off-site and updated at every system/application software change. The back-up must be re-done at least annually if there are no changes.
- Incremental data back-ups at least daily and more frequently if possible - normally stored on-site
- Full data back-ups at least weekly to be sent off-site maintaining at least 4 generations
With this scenario the maximum amount of data that can be lost by machine failure is 1 day. The maximum amount that can be lost by the destruction or loss of the site housing the system hardware is 1 week.
Where the prime source of agency data is held on an isolated PC e.g. field workers using portable computers, the user must back-up data and documents at least daily to a second drive or separate medium for as long as it remains the only copy.
The exact details of the back-up schedule for any one machine will be determined by the most sensitive and important data held on it.
The requirements for data back-up should be part of the agreement with the Physical Custodian.
Recovery
Data available on-line and being constantly updated should not be totally reliant on recovery from back-up, except in cases of catastrophic failure. Restoring a back-up is not the same as recovery. Restoring a back-up will result in data loss and cause expensive re-work as well as errors. Database management software that allows automatic recovery from temporary system or software failures should be used wherever possible.
Back-up of network file systems should operate at an appropriate frequency and level of granularity to meet business requirements for recovery of single files, Emails etc. with minimum data loss.
Restore
The actual recoverability of each grade of back-up must be periodically tested in case it turns out to be unrecoverable when needed. Both the data and necessary support systems must be fully recoverable at an independent site from that housing the normal production systems. The Business Custodian will need to determine how much data input or document development work the agency can afford to lose in a disaster that would require a full restore. One day is a typical choice for cost effective tape based systems. See also Standards: Disaster recovery.
Rationale
Electronic storage has become more and more reliable as technology has been refined, however the cost of back-up is far outweighed by the cost of losing data and documents essential to the functions of the agency.
Standard: Storage Media
Data and document storage will conform to the following standards:
- Storage conditions for electronic data or documents will conform with government Information Technology standards
- Physical storage conditions will conform with National Archives Storage Standard NAS 9901
Support Policies: Retention
Scope and Interpretation
Storage technology is constantly changing. Examples include: magnetic storage - video, audio and data tape, disk etc., CD, microfilm, paper with chemical inks or fused toner etc. While the agency may reproduce data or documents in a variety of media for operational purposes, it must actively choose media for retention. The decision to use a particular technology will normally be a trade off between price and requirements, but it must be fit for the business purpose:
- Online and near line storage must meet system performance requirements in terms of access, integrity, and failure rates laid down by the Business Custodian
- Off line storage media must meet business requirements for access and retention
- Where a medium, e.g. magnetic tape, is chosen for back-up or off line storage, rules must be in place to ensure that data is refreshed at a predetermined frequency. In the case of magnetic tape archives an annual refresh is usually advised.
- Agencies must monitor the processes that place data and documents into the medium chosen for the prime source and ensure that they are working. For example, processes that rely on individuals to "print and file" are especially risky and need close monitoring.
Rationale
Adequate storage provisions are essential to protect all Crown data and business document assets, whether it be specialised storage for paper documents or software and hardware for material in electronic format.
There is a risk that context and content can be lost in transfer to another storage medium, unless the transfer is correctly managed and audited.
Standard: Disaster Recovery
All agencies will have a fully tested disaster recovery plan to reconstitute data and document stores to ensure timely re-establishment of the business. Plans will be produced by the Business Custodian and subject to agreement by the Crown Data Steward.
Support Policies: Retention
Scope and Interpretation
Disaster recovery planning is a component of business continuity planning and includes strategies to ensure:
- Back up of all electronic data and document stores to a standard that permits their reconstitution. See also Standards: Backup Recovery and Restore.
- Restoration of data and document stores and their operating software to established time-frames in the event of a disaster
- Vital records (see Glossary: Vital Records) will be identified in approved data or document stores.
- Vital data and vital document stores to be restored in time-frames to permit the rapid re-establishment of the business of the agency. Where data and documents are stored only in a physical medium such as paper, consideration must be given to duplicating vital records.
- Minimal damage to or loss of irretrievable physical material. See also Standards: Storage Media.
- Users and technical staff are trained to required levels of expertise and available to respond in the event of a disaster
- Hardware and software required to support a restored system are in place or will be available
- Testing schedule to ensure plans are regularly exercised and assumptions are challenged. A full test must include full restoration at the disaster recovery site and simulation of total expected usage at that site.
Rationale
In the event of a system failure each agency must be able to retrieve or recreate data and documents relating to its core activities.
Many agencies are now almost totally dependant on computer systems to continue everyday business. Without a tested disaster recovery plan, government agencies place their functions at risk of very serious disruption during an emergency.
Identification of vital data and documents permits an agency to focus its main effort.
Standard: Retention requirements
Agencies must identify, describe and comply with their retention requirements for data elements and business documents.
Scope and Interpretation
This applies to all data elements and business documents owned by the Crown.
Each agency must:
- Clearly identify retention requirements, including access rules over time as stated in government legislation
- Clearly identify any additional business needs for retention
- Document the retention rules for the types of data elements, documents and data stores created in the course of an agency's business activities and work with National Archives to develop retention and disposal schedules.
- Monitor compliance with identified retention requirements
- Review retention requirements when changes to the agency functions, structure, classification, technology or legislation occur.
Each agency must have a clear record of its official data and documents, and must apply policies and business rules to retain them for defined periods, without duplication of effort across systems. Access restrictions may change over time, and this information must be included in the retention rules.
The Business Custodian will define relationships between retrieval requirements and systems for off-line or remote storage, and will negotiate these with the Physical Custodian. Practically, items will usually be moved from operational systems in a related group e.g. a client record, project documents etc. However it may be found that sub-groups of data elements may be usefully moved separately - for example parts of a client record relating to a specific business area may be required on-line for a limited period only, while other parts may need much longer retention.
Retention and disposal schedules must meet the criteria stipulated by National Archives. These criteria address three main issues:
- How long to keep what material
- Who has custody
- Who has access
- Retention and disposal schedules must be authorised by the Chief Archivist. Data and documents may only be disposed of with such approval
Rationale
Agencies must comply with the statutory requirements in the Archives Act that only the Chief Archivist can approve the destruction of Crown material. The most efficient way to achieve this is via a Retention and Disposal Schedule.
Rules based retention ensures:
- Data and documents are kept in a systematic way and are available for appropriate periods
- Timely access to accurate information
- Data and documents are managed in accordance with their statutory, evidential, and business value
- Access restrictions are administered over the retention period
- The appropriate policies, business rules and operation standards are demonstrably applied to official data and documents. Removes the risk of breaching statutory requirements including the Archives Act
- Enables systematic identification and prompt destruction for those no longer needed.
Standard: Transfer between Agencies
Where all or part of a data or document store is to be transferred between agencies, those agencies must develop an explicit agreement for that transfer. With the exception of transfers between an agency and National Archives, the explicit agreements will be between agency business custodians and will be ratified by the Crown Data Steward.
Support Policies: Retention
Scope and Interpretation
Transfer in this context means the complete removal of material from one agency to another.
As agencies acquire or discontinue functions, datasets related to those functions may be transferred to another agency. The agencies concerned must draw up an agreement detailing:
- Which agencies are involved in the transfer
- What is to be transferred - all or part of the dataset
- What if any part of the dataset is not to be transferred, and which agency has responsibility for it
- Any existing access restrictions including security-classifications
- Any existing Retention and Disposal Schedules
- Any applications required to maintain the integrity of dataset
The transferring Business Custodian will also inform National Archives of the transfer.
Where datasets are no longer in operational use, the agency may negotiate either retention within the agency or transfer to National Archives. Only material approved by the Chief Archivist as requiring retention may be transferred to National Archives. Transfers to National Archives must be to standards set by National Archives.
In the past, most agencies have transferred paper documents to the physical custody of National Archives when several years have passed since they were in operational use, but with electronic material that is readily reproduced this paradigm can be reviewed. Data and documents do not have to be transferred to the custody of National Archives, and each agency can negotiate to retain custody to an agreed standard of storage and access. Agencies may wish to negotiate which agency has the responsibility for retention, whichever medium is used for storage
Rationale
Ensures that the Crown is aware of where its datasets are and who has responsibility for them. Minimises the risk of orphan datasets.
Data and documents will be held in the physical custody of whichever agencies are best able to meet the business needs and regulatory requirements for their retention. Each agency must negotiate retention with National Archives.
Standard: Destruction Protocols
No data or business documents will be destroyed while they are needed to fulfil the statutory or business requirements of the crown:
Any deletion or destruction process must be secure, deliberate, authorised and auditable.
Scope and Interpretation
Each agency must at all times be able to identify the existence and status of documents, and, where applicable, demonstrate accountability in the deletion or transfer of those documents. The deletion or destruction process will be auditable, and there must be no unauthorised deletion of data or documents.
- For any processes in an automated system that permit the deletion or overwriting of data elements, agencies must validate those processes against retention and disposal requirements and against the risk of accidental destruction
- Agencies must provide advice to staff on what document types do not require retention of non-substantive versions
- Data and documents may only be deleted or destroyed in accordance with Retention and Disposal Schedules agreed with National Archives and according to business rules. See Standards: Retention requirements.
- Responsibility for deletion of data and documents rests with the Business Custodian
- Data and documents must not be automatically destroyed without review
- Staff searching for documents that have been destroyed should be able to identify the fact of and the reason for deletion of documents from the system
- For electronic documents, deletion will comply with agency policies and standards for secure destruction of electronic data
- The Privacy Act applies to data used in authorised data matching programmes.
Rationale
Agencies may not destroy Crown data or document assets without agreement from National Archives. A formal destruction process will ensure legislative requirements are met, e.g. ensure that the reason and authority for destruction are available.
Auditable destruction of data and business documents:
- Demonstrates accountability in managing Crown assets
- Saves time wasted searching for documents which no longer exist
- Satisfies Ombudsman's Office requirements for declining requests for official information
[ Previous ]
