Three French government ministries have pooled resources to build a digital archiving software to secure and maintain digital documents. The software, called Vitam, is an open back-end which could be used by any administration. It makes use of new technologies to promote re-use. Agility is a priority in keeping the project aligned with business needs and ensuring that it remains economically viable.
Policy Context
- French public archives are regulated in Book II of the Code du Patrimoine (Heritage Code), which is designed to guarantee the protection, authenticity and quality of archives.
- Each administration is responsible for maintaining its own archive repository. The General Code for Regional Administrations (CGCT – Code General des Collectivités Territoriales) specifies that local administrations have a legal obligation to set up their own local archives and bear the related costs.
- Official digital documents are legally considered to be administrative documents and must be archived accordingly.
- The Interministerial Committee for Archives (Comité Interministériel aux Archives de France – CIAF) has designed a general framework (Référentiel Général de Gestion des Archives – PDF) setting out the rules and recommendations administrations need to build and maintain their archives.
Technology solution
The origins of Vitam
- Vitam was validated by the Interministerial Committee for Archives on 1 July 2013, and approved by DINSIC (the state IT authority) on 4 September 2013.
- On 5 March 2015, three ministries and the Prime Minister’s Office signed a convention under which Vitam is officially funded as an interministerial project.
- On 9 March 2015, launch of the project, DINSIC and CIAF were identified as lead
Supporting e-administration
The French government has put in place an ambitious e-government policy under which more and more administrative procedures will be digitalised. French administrations need to manage a growing volume of native digital documents that now have a legal value in their own right. They must be preserved accordingly, and their evidential value maintained.
In 2011 the French Ministries of Foreign Affairs and International Development (MAEDI), Culture and Communication (MCC), and Defence began discussions on the development of a common archiving solution. These administrations have important needs for long-term and permanent archiving. The Ministries of Defence and Foreign Affairs have to maintain their own critical archiving processes and sensitive data (active, semi active records and archives). The Ministry of Culture, meanwhile, manages inactive records with permanent value for France’s other ministries.
“On their own, none of the ministries could finance a permanent archive on this scale,” says Jacques Marzin, the former government CIO, on the SGMAP website. “This explains why we managed to build such a collective project that is so important for the nation.”
Twelve market solutions were tested in an effort to find the right solution. However, none of them met all the needs defined by the Vitam project teams, a presentation reveals (PDF).
Vitam : five projects
Vitam is managed by DINSIC and the Interministerial Committee for Archives (CIAF). Vitam is composed of five distinct projects:
• Develop a digital archiving free software capable of supporting large volumes of documents;
• Deploy the solution in the three ministries and adapt it to their respective archiving processes and rules. The three projects are:
- Saphir (MAEDI)
- Adamant (MCC / National Archives)
- ArchiPel NG (Defence)
• Promote the re-use of the solution by other administrations through a project called AD-ESSOR. This is supported by the Ministry of Culture and the Interministerial Committee for Archives. The goal is to make installation as easy as possible and encourage its adoption by other administrations. In due course, some new software components will be added and detailed documentation is already published, says Jean-Séverin Lair, the program’s director.
Vitam : the features
Vitam is intended to be a technological back-end that provides all the essential functions for archiving. Administrations who use Vitam are responsible for their own front-ends and must therefore use Vitam with their own archiving rules, explains Jean-Séverin Lair. Administrations are also responsible for the IT infrastructure needed to support Vitam. A development platform with 400 virtual machines and more than 100 TB of storage is currently hosted by the Ministry of Agriculture.
Vitam provides functions to manage the entire lifecycle of every document (active, semi-active and inactive stages from short-term to long-term archiving), as required to meet all the use cases identified by administrations in France. The type of archiving chosen for a particular document depends, for example, on how frequently it is accessed and the acceptable delay, Lair explains. Active and semi-active stages suits documents that require regular access and therefore a rapid response from the archive system. Permanent archiving is for documents that are accessed more rarely, and hence do not require high speed. Vitam must also guarantee that the evidential value of every archived document is preserved.
The basic functions of Vitam span the ingestion of a document, its conservation (or destruction), and its maintenance. These functions are:
- Ingest documents based on defined standards;
- Index the archived documents, and provide access as required. Both metadata and logs are searchable;
- Ensure the perservation of the documents. Documents are translated into pre-defined standard formats to ensure that they remain accessible over time. The translations are based on specific rules set by the administrations;
- Manage the storage infrastructure.
“Vitam has been developed to support large volumes of documents, so it can address both large administrations and groups of smaller communities,” says Lair. The platform will be able to manage petabytes of data.
Vitam has been designed to be used via web services. Document ingestion, metadata searching and document access are accessible via RESTful APIs. Administrations can integrate these into their own applications, such as messaging or human resources (HR) systems.
Administrations can participate in:
- The validation process;
- Drafting specifications;
- Testing the software;
- Creating a platform using the Vitam back-end; and
- Sharing code back to the community.
Agility and open source
“Vitam is entirely composed of open source software,” says Lair. 14 modules have been developed by Vitam’s development teams, and 70,000 lines of Java code have been produced. According to Lair, the application relies on open source libraries and uses only two packaged applications: MongoDB, a NoSQL database which stores the metadata, and ElasticSearch, used to search metadata.
The source code was published on GitHub on January 16, 2017 under a CeCILL licence, a French and standard licence for free software created by the three research organisations CEA, CNRS and INRIA . “We are committed to updating it,” Lair says.
According to Lair, free software was a “natural” foundation for this project, and Vitam “would not have been possible without the use of free software”. Openness encourages the reuse of the solution and its adoption by other administrations. It also makes customisation easier.
“Open source was the best way to share the developed software, and thus to make the financial effort of the French government profitable. Open source offers a wide range of innovative technologies useful for developing the solution. Finally, it is also the best way to extend the user community and diversify the needs and the functionalities. It also facilitates the maintenance and durability of the software solution”, Lair adds.
Lair explains that the program makes extensive use of agile techniques to respond quickly to the needs of administrations. Being agile helps to prevent delays in the progress of the project and ensures that “the deliverables remain aligned with the initial needs”, explains the SGMAP website.
The resources contributed by the three ministries to the Vitam development team include IT personnel, archivists and other business experts. The interministerial team currently has 14 members, and will eventually grow to 60 people.
Agile methods allow:
- Fast and regular deliveries of software elements (three weeks for sprints, a release every three months, and one major version per year);
- Frequent updates to individual functions, rather than having to wait for milestones;
- Use of the latest technologies (such as big data) that require special expertise.
Technology choice: Open source software
Return on investment
16 million euros were allocated to the development of the Vitam software. In total, the project will cost 56 million euros (resources and all ministerial projects included), Lair explains.
The tender was divided into several lots, separating the development and integration processes to optimize profitability, according to the SGMAP site. This separation makes it possible to include a broad scope of commercial suppliers, “especially SMEs”.
Development was awarded to HIT-Part, Linagora and Smile, and integration and software quality to Thalès. Smile was also included in the tender for some development expertise
Vitam’s roadmap is:
- 9 March 2015 : Launch of Vitam
- 12 August 2015 : Publication of the two calls for tender (development and integration)
- 30 September 2015 : End of submissions
- Beginning 2016 : Publication of tenders
- July 2016 : First deliverables (as APIs)
- September 2017 : Completion of the first production version
- September 2018 : Completion of the second production version
- September 2019 : Completion of the third production version
Lessons learnt
Scope: National