APPRAISAL AND DISPOSAL OF ELECTRONIC RECORDS

Raimo Pohjola Deputy Director-General, National Archives of Finland

Principles, problems and solutions

General Electronic transactions are no different in nature from their paper counterparts: They need to be recorded, captured in a fixed form, maintained and made accessible as records. The role and purposes of record keeping in the electronic environment are the same as in the paper world. Similarly, the same principles and criteria are applied while appraising electronic records. Because principles and criteria of appraisal have already been treated here, I concentrate on certain special characteristics of the appraisal of electronic materials.

Is there a need to appraise and dispose electronic records? It is common knowledge that the preservation of electronic records does not require much space. This has led to the conclusion that there is no need for the disposal of them; all materials could be preserved permanently. The problem of growing volumes could be met simply by increasing the capacity of hardware. The media for storing electronic information are inexpensive and their capacity is continuously increasing. This view pays no attention to the fact that the methods and techniques of electronic preservation are still quite undeveloped today. The materials that are to be preserved have to be copied and migrated at certain intervals which means costs on a continuous basis. The volume of the materials will naturally increase, on the long run, also the costs of preservation. Even more. If the materials determined for long-term preservation are not separated from other materials, it means that materials that are not really worth long-term retention will be copied and recopied all over again and again. There are other problems with large volumes of information, too. It will become difficult to use the materials. The user will find it difficult to locate the materials he or she is really interested in. The increase of volume means difficulties also in retrieval which forces to develop the capacity of IT-systems. Large volumes of information are also a burden to the hardware.1 That is why it is important to be able to separate materials having different retention periods from each other easily.

Long-term strategy of early appraisal In many countries, the appraisal of paper records does not take place until at the end of the records life-cycle. This is not the case in Finland, however. We try to plan appraisal 30

beforehand so that when records are created or received in an organization it is already known what are the respective retention periods. As regards electronic records early appraisal is doubly important and it is commonly recommended in order to avoid the risk of records becoming incomplete or unreliable. Changes in IT-systems can cause losses or degradation of records which have not been migrated to a new system with sufficient planning. It is also possible that retrospective disposal procedures on IT-systems will cause changes that are harmful for the systems. Databases may lose information that should be retained. This risk is especially marked with regard to relational databases as Matti Pulkkinen has pointed out in Finland.2

The life-cycle concept The so called life-cycle concept of records was applied, to a certain extent, already in paper context. The concept received attention especially in Canada. In principle, this kind of approach has been a part of the Finnish model for a long time. Especially since the early 1980s it has been an established goal in Finnish archival practice to plan the records management of administrative authorities comprehensively so that it would cover all the stages of the records from creation to disposal or permanent preservation. I shall give here an example of how the different phases of the life-cycle of electronic records have been planned in Finland. The example is from the report of a working group for electronic records, dating from the mid-1990s:3 Active phase

S S S S

The record is captured in the record keeping system by registering it: a kind of procedural authorization takes place with the capture and checking of the possible electronic signature. The integrity and authenticity of the record is reviewed in the internal information system of the organization. The system is being audited and controlled by inspections made by an independent third party and any functional problems that may arise are recorded. And, finally, a very important thing: it is determined how long the record will be kept in the system - the active phase lasts ca. 10 years (in reality, possibly 6-7 years). The systems will change so rapidly that long active preservation is problematic.

Passive (semi-active) phase

S S S S

The preservation of records takes place off-line. The records and their metadata are in ASCII-format. Two copies of the materials that are recopied at five-year intervals. Disposal of the records whose retention periods expire.

31

Historical phase

S S

S S

The records that are determined to be preserved permanently are transferred to the custody of the archives service of the organization in question or to an archival institution. The records are preserved as data (e.g. statistical information) as long as it is technically and economically possible. The materials which are not heavily structured and which can be exploited by using clear cut search criteria can be taken to permanent preservation in micro form. Paper printouts may be considered when the materials in question are not extensive. The printout procedure has to be carefully planned and regulated. The evidential weight of the record is based on the place of preservation and on the process of the handling of the record which must be backed by an appropriate documentation.

Preserving content, structure and context In the electronic environment, it is important to preserve not only the information content of records but also their structure and context. Preserving mere information content is not enough. Contextual and structural information is needed to make the documents understandable and usable as records. Electronic documents do not have the physical characteristics of traditional records that establish the relationship between a record and its functional and administrative context. Thus electronic records are dependent not only on well documented administrative context but also on information describing how the document is recorded. This has to be done at the time of creation of the electronic documents and calls for good electronic record keeping procedures to be put in place.4 The preservation of these three components constitute a problem in appraisal, too.

The planning of appraisal The planning of appraisal takes place, naturally, in connection with the planning of the whole archiving system. Appraisal of electronic records and managing of electronic records should be seen as a corporate process which has implications for broader information management and information technology strategies.5 This kind of planning requires that the tasks (and operative processes) of the organization as well as the records created or received in connection with them are mapped and inventoried. The determination of retention periods for the records thus inventoried is an integral part of this process. With respect to electronic records it is important, besides the things mentioned above, to pay attention to the reviewing of the IT-systems. This includes, among other things:6

S S S S S

which are the IT-systems where electronic records are maintained in, do these systems meet business needs and preservation requirements, is all metadata necessary for appraisal generated and are file/directory structures adequately organized, do IT-systems conform to desired standards, policies and procedures, where are records physically located and how can they be retrieved. 32

Appraisal in system designing The most efficient way to create a management system for electronic records is to take the so called functional requirements into consideration already at the planning stage of the system. It should have built-in facilities which support records management and appraisal process. Retention times could be allocated to particular electronic record types and the system should track and record the progress of records and the conditions related to them as they age. The records management system should not allow for automatic destruction of records, but should ensure the proper organization and management of the appraisal process. The system should alert the record manager at the point where human intervention or approval is required.7

The appraisal of electronic materials: first experiences The appraisal of electronic materials is nothing new in itself. A lot of experience in this field has been gained, especially in North America but also in Europe. During the 1970s and 1980s computerized data processing was seen as a cycle of input, processing and output. Machinereadable files - as they were then called - were generally characterized as input, processing and master or output files. Master files or historical files represented the final stage of the systems process. They were usually stored on computer tapes. Master files produced mainly records which were analogous to paper records. The appraisal of this kind of materials started usually with a review of the computer system that produced the machine-readable files. An important instrument in this was the systems documentation that could be, however, quite poor in quality. In this context the determination of records creators was, on its part, usually quite easy. 8 The principles and criteria that were applied in appraising these materials were mainly the same that were applied in appraising traditional paper records. The essential thing in this process was to select materials for permanent preservation from a corpus of materials that was already accrued. The handling of the materials that had been selected for permanent preservation required, however, new solutions and practices. New copies of the tapes had to be made at certain intervals. The files were stored on tapes as so called sequential flat-files which were hard- and software independent. The development of information technology has, however, made the problems of appraising electronic materials more acute. The electronic records of today are not necessarily analogous to paper records any more. They include multimedia systems, integrated relational databases and complex databases that cross organizational boundaries. A relational database can be used to produce almost an unlimited number of database views. In this kind of context new problems arise in connection with such basic concepts as records creator, fonds and record. In the discussion of the 1990s there seems to be a common consensus about certain new principles which were first presented by Charles Dollar and besides him also David Bearman.9 The principles are as follows:

33

S S S S

Archivists should adopt a new approach. It is not possible any more to appraise records but those functions, transactions and competences that produce records. Archivists should define the functional requirements for the life-cycle management of recorded information. Archivists should intervene in the record creation process as early as possible. The appraisal function ought to be taken into consideration already at the planning stage of IT-systems. Metadata systems should include elements that describe the different stages of the life-cycle of electronic records.

The support to appraisal provided by the Finnish appraisal tradition As Jari Lybeck pointed out in his presentation, the Finnish appraisal tradition has been quite pragmatic. The appraisal of electronic records will require a different approach. There is, however, at least two things in the Finnish tradition which will help the adaptation to the electronic environment. First, Finnish appraisal is pre-appraisal in nature which means that the retention period of a record is determined at an early stage of the life-cycle, preferably even before the actual record itself is created. The instrument that is applied in appraisal is the so called records management schedule. According to the Archives Act, every administrative authority must have such a schedule. Its importance will be further enhanced by the new Freedom of Information Act that will come into force before the end of this year. A special decree on good management of information will also be issued on the basis of this law. This decree will give records management schedules an important role in the realization of the principle of freedom of information in the records management of administrative authorities. It can be said that the approach represented by records management schedules is closely related to the so called macro-appraisal. A records creator and its tasks constitute the starting point for appraisal. Within tasks it is possible further to identify the operative processes attached to an individual task. While making a schedule it is a common practice to review how tasks and processes fit in the organizational structure of the records creator. The assessment of tasks, processes and organizational units helps to notice records management and processual problems that cross task- or organizational boundaries. The records management schedule shows what series and other information materials of record nature accrue in connection with the management of a records creators tasks and how these materials are registered and archived. The schedule also shows which records are to be preserved permanently and which can be destroyed after the expiration of different retention periods. Records management schedules are composed by the administrative authorities themselves but the decisions on permanent retention of records are made in the National Archives Service. Records management schedules will be used also in the appraisal of electronic records. Certain basic information on IT-systems should be recorded in records management schedules. This information includes:10

34

S S S S S S S S S S S

name of the system, whether the system is a personal register, whether the information is classified or not (and to what degree if it is classified), general description (purpose; connected tasks and processes; basic functions of the system; main changes made in the system since its taking into use), information content of the system, sources of information, unit/person who is responsible for the information in the system, information given to other authorities from the system, software, hardware and media, unit/person who is responsible for the technical maintenance of the system.

This information serves as a kind of short introduction to the IT-systems of an authority. It does not remove, of course, the need for a proper technical and other documentation that must be attached to IT-systems.

First magnetic tapes in the National Archives The first magnetic tapes were transferred to the National Archives in 1987. They contained information from the Finnish Population Register from the 1970s, that is, from the earliest period of the computerized central register when the register was not yet a database solution. The appraisal procedure was typical for the period. First, the content of the tapes was mapped by using the documentation that was available. After that the tapes themselves were examined; most of them were found to be readable. In some cases, however, we had to rely on copies. The number of tapes taken into custody were substantial which was due to the fact that they originated from the early stages of the new application. The information on the tapes was not unique because similar information was available, in paper form, in local population registers and parish registers.

Finnish a ppraisal d ecisions on electronic m aterials In Finland, public authorities and institutions had to compose records management schedules for themselves by the end of the year 1989. It was quite common that electronic materials were taken into consideration rather poorly in these schedules. In 1992 the National Archives issued guidelines on the appraisal of electronic materials. The perspectives employed in these guidelines included operative use of the materials, legal aspects and the needs of scientific research. The emphasis was on the research perspective.11 In 1994 the National Archives had a working group that made a survey on the electronic materials in the central administration and research institutions. These materials included mostly different registers and databases. The result of the 1994 survey was that the volume of the aforesaid materials was ca. 200 GB of information and that the annual rate of growth of the materials was ca. 10 %. The working group gave also some recommendations concerning the appraisal and disposal of these materials:12

35

The National Archives Service should make a distinction between permanent preservation on the one hand and continuous or for the time being preservation on the other hand. Consequently, permanent preservation would mean that the National Archives Service is committed to preserve the materials in all circumstances. The most economical way to preserve a large database would be to take it in as COMmicrofilm. The working group estimated that the annual volume of one 1000 pages would justify the COM-solution. In case the research use of the materials would be difficult, COM-microfilms could be accompanied also by electronic preservation on a continuous basis. If electronic preservation were the prerequisite of efficient research use and if also evidential aspects required electronic form that would be a justification for permanent electronic preservation. Continuous or for the time being preservation in electronic form would take place in cases where evidence is not a major issue

In practice the situation is such that permanent preservation of electronic materials is necessary in certain cases. These materials should be preserved separately from the materials whose retention takes place on a continuous basis. The working group proposed also for the internal use of the National Archives Service certain criteria for the appraisal of electronic materials. The basic guidelines are that the information system should be connected with the mandated functions of the records creator and that databases containing reference information should be preserved only in cases when they constitute a part of the registry system of the records creator. The working group examined also the need for permanent or continuous preservation of the systems that were included in the survey. It turned out to be so that in surprisingly many cases the process of composing a records management schedule had brought in information that had provided grounds for the making of appraisal decisions. The survey itself resulted in many appraisal decisions.This appraisal activity concerned active systems and materials already in existence. When the systems had been at the planning stage, no attention was paid to appraisal at all. However, the introduction of the appraisal aspect even at this later stage could have a positive effect in that it caused a new awareness with regard to the means of long-term preservation of the materials. It should be pointed out that most of the institutions that were surveyed were big research institutions with a good IT-infrastructure producing different electronic registers. As regards this kind of research institutions, the National Archives has decided that they themselves preserve as long as possible the electronic materials that are designated for permanent or continuous preservation because they have the best expertise with regard to the use of the materials. There have been cases, however, were an institution has not taken its responsibility as the keeper of the materials seriously enough.

36

In most cases decisions on permanent retention have concerned entire registers. The selection of individual sets of information has proved to be a very laborious practice. Besides that there is the possibility that this kind of selection causes damage to the database in question. Before finishing my presentation I would like to present three concrete cases from the field of appraisal. 1. The appraisal and disposal of the population register of the Finnish Population Register Centre have been developed so that cross-sections are taken from the register concerning the years that end with 0 and 5 showing the situation of the register on the 1st of January. Magnetic tapes containing these cross-sections are transferred to the National Archives. Basic information on the population has been determined for permanent preservation by a special law issued this year. The Population Register Centre will keep this information in its own database and takes care, naturally, of the information service required. This means that the tapes transferred to the National Archives are essentially safety copies. They are there to make sure that this strategically important information will be preserved in all circumstances.13 2. It is a well established practice in Finnish universities to keep student registers in electronic form. Already in 1990 it was decided that certain fields of information are printed out annually on paper from these registers for permanent preservation. The original tapes are preserved by the universities as long as they regard it necessary.14 3. The third case concerns tax records of the local offices of Inland Revenue.15 It used to be so that the annual tax lists, which were paper records, were determined to be preserved permanently. They were also free for access by the members the public as soon as they were composed. The annual tax returns, on their part, were classified information. Their retention period was first 10 years, later 6 years. Tax returns are, obviously, interesting material for scientific research. That is why it was decided to take a sample of tax returns to permanent preservation. Long discussions in the National Archives resulted in the decision to use a combined sampling method which includes all the years ending with 0 and 5. Within these years the records concerning people born on the 8th, 18th and 28th of each month shall be preserved permanently. The use of the date of birth as the basis of sampling was problematic because the tax returns were kept in the offices of the Inland Revenue in alphabetical order. When the use of information technology increased in the offices it became possible to print out alphabetical indexes on the tax returns that were sampled. This made the sampling procedure easier in practice. The management of taxation information has changed radically. Taxation authorities get the information they need increasingly in electronic form. Only part of the population nowadays fill in tax return forms. The National Board of Taxation has the same kind of large database as the Population Centre. The new appraisal decision from the year 1996 stipulates that tax lists shall be printed out in micro form for permanent preservation, starting from 1995. The offices of Inland Revenue will preserve tax lists that are in paper form for 10 years. As regards the information comparable to tax returns, it was decided that it shall be taken into permanent preservation in micro form. The information is selected using the combined sampling method I described earlier.

37

In Finland, a new law on using electronic methods in administrative transactions is about to be issued in the near future. The new legislation will increase electronic transactions between authorities and citizens as well as between authorities themselves. The authorities have to develop their IT-infrastructure and services accordingly. The more and more authorities adopt electronic records and document management systems their development becomes a real challenge. Records management requirements, including those of appraisal and disposal, have to be taken into consideration while planning these new systems.

EU-cooperation In the so called Blackbook (Archives in the European Union) it was acknowledged that the problem of growing volumes of documents is a common archival problem today. Despite the fact of growth there is no absolute or even dominant method of appraisal and disposal.16 The problem gets even more faceted when electronic transactions become everyday reality. On the other hand, the field of electronic transactions seems to provide a fruitful area for study. The technology used is international, the problems are common. The DLM-Forum has proved to be a productive form of cooperation. The conclusions of the DLM-Forum '99 are promising. It is not necessary to repeat them here. Even the products of the first DLM-Forum in 1996 - the proceedings and the Guidelines on the handling of electronic information - have become important tools of work. As regards the framework of appraisal policy presented in the Black Book already in 1994,17 it is obvious that the analysis holds. To my mind, the recommendations of the Black Book are still valid. According to this and taking into consideration the development that has taken place, we should examine the possibility of composing model records schedules, by using the expertise of different national experts. Firstly it would be reasonable to limit the attempt to a certain of types of records that are familiar to most Member States. Further, we could aim at a recommendation on the principles, criteria and process of appraisal as well as retention periods. This could lead, as a recommendation, to a manual of records management for public sector archives which would be valuable for the government departments and archival institutions of the Member States.#

1 2 3

Pekka H enttonen, Atk -aineistojen ar kistoinnin haaste ita, Arkisto, Arkistoyhdistyksen julkaisuja 6 (1999) , p. 44. Ibid., pp. 44-45.

Shk inen asia kirja ja sen k yttn liittyvt tietoturva llisuustarpe et. Tiveke 2 .3:n osa tehtv 2 :n rapo rtti (3.101996), pp. 30-31.
4

Manag ement and appraisal of Electron ic Records. Stand ards for the man agemen t of Governm ent records (Public Record Office, 1998), p. 7.
5

Ibid., p. 15.

38

6 7 8

Ibid., p.19. Ibid., p. 35.

Charles M . Dollar, Archival Theory and Information Technologies. The Impact of Information Technologies on Archival Pron ciples and M ethods (Macerata, 1992), p. 56.
9

See for exa mple ibid., 5 8-59, 76 -77; Dav id Bearm an, Archival M ethods, Archives and Museum s Informatics, Technical Re port 3: 1 (Sprin g 1989 ), cit. in David B earman A rchival Strateg ies,The Ame rican Archivist , 58:4 (Fa ll 1995), pp. 382-383, 399; David Bearman,Archival Strategies, pp. 405-406.
10

Arkistolaito ksen suo situs arkiston muod ostussuu nnitelma n laadin nan, k ytn ja yll pidon p eriaatteiksi (Draft 4.10 .1999), p .5.
11 12 13 14

Valtionarkiston ohje ATK-aineistojen silytysarvon mrittelyst (Valtionarkisto, 1992). Atk-aine istojen arvo nm ritysprojekti. Lo ppurap ortti (23.5.1994), p. 5-6. National Archives to the P opulation Register Cen tre 2.5.1990, no 1 5/41/89. Na tional Archives.

See for example the decisions 9.2.1990, no 98/33/88 (the University of Joensuu) and 5.10.1990, no 92/32/89 (the University of Tamp ere). National Archives.
15 16 17

Decision 28.5.19 96, no S 22/9 5. National Archives. Archives in the European Union (Brussels-Luxenbourg, 1994), p. 13. Ibid., pp. 15-16.

39