Eurostat collects and publishes huge amounts of data each year, and exchanges many datasets with other large organisations. This exchange was constantly suffering from a lack of interoperability, as data needed to be converted from one organisation's convention into another, a process which consumes both time and money. Different organisations were also using very different tools to work with the data, which caused further problems. In late 2001, Eurostat got together with a number of EU committees to discuss the need for greater interoperability within the European public sector. In 2005, IDABC agreed to fund the SDMX Open DATA Interchange (SODI) project. Thanks to previous cooperations between Eurostat and other international institutions, the SDMXÂ (Statistical Data and Metadata Exchange)Â standard quickly found a large group of sponsors, all of which hoped to benefit from the greater interoperability afforded by using a single standard, and the tools built on it. These tools were developed by Eurostat and other sponsoring institutions, and many of them were published under the EUPL license. The SDMX Converter is an example of the successful development and publication of a tool that is essential for working with the SDMX standard.
In 2001, Eurostat's section forÂ Advanced Technologies for the European Statistical System in Luxembourg sat down with various committees composed of the European Member States to brainstorm about the possibilities of developing a European knowledge management system. Although this focus changed quickly, one core issue remained: Europe's need for more interoperability in the information sector. About four years later, these brainstorming sessions were followed by an intensive planning phase which eventually led to the X-DISÂ program (XML for DATA Interoperability in Statistics). The focus here was no longer on knowledge management, but rather on interoperability of statistics within Europe, and on the usability of these information beyond the organisation which initially gathered it. This was to be achieved by introducing the a standard for all statistical data and related applications, which came to be called SDMX (Statistical Data and Metadata Exchange). In the framework of X-DIS program, the SDMX Open DATA Interchange (SODI) project became an important pillar.
Eurostat is the statistical institute of the European Commission, producing and illustrating data of a wide range of indicators,Â from economic ones to social, for example. Within the framework of the IDABC program (Interoperable Delivery of European eGovernment Services to the Administrations, Businesses and Citizens), Eurostat is in charge of several programs that aim at fostering European eGovernment services. One such program is the X-DIS project, a â€œproject of common interestâ€.
Description of target users and groups
Besides Eurostat, the SDMX standard is sponsored by the Bank of International Settlements (BIS), the European Central Bank (ECB), the International Monetary Fund (IMF), the Organization for Economic Co-operation and Development (OECD), the Statistics Division of the United Nations, the World Bank.
The institutions all benefit from the standard, and work together on increasing its use, as well as on improving the applications built on it. The costs for tools developed as part of the SODI project are shared with all institutions, which keeps the financial burden for each organisation at a minimum. The data collected by these institutions can be used by all of them, which facilitates the data collection and its use greatly.
Description of the way to implement the initiative
For the introduction of the SODI project at Eurostat, the team gradually implemented the standard and the related tools and applications.
The X-DIS team took a step-by-step approach to introducing the use of the SDMX Open Data Interchange (SODI). This allowed Eurostat to adopt its own procedures. Ensuring a smooth transition was also necessary since Eurostat works closely with the Member States in collecting data, so that a significant number of organisations were affected by the introduction of the new standard and the related tools. Fortunately, this process went smoothly, and the team did not encounter significant problems.
Specialised tools such as the SDMX converter and other software developed by the X-DIS project usually do not require significant changes in the IT infrastructure of the user organisations. This is particularly the case for applications built on platform-independent technologies such as Java.
The main technical backbone of SDMX is XML, which stands for Extensive Markup Language. It is a framework in which markup languages are developed, which allows for the for visualization of data in text form. With regard to statistical data, SDMX makes it possible to illustrate statistical data with tables and graphs, which can be updated easily and quickly. Through the use of the standardized SDMX format, the different institutions can make use of data more efficiently, and share their datasets much more easily, as they do not have to convert data into their own format â€“ a process which is usually time-consuming and error-prone. Especially in an international context, this substantially facilitates the work, as data has to be collected just once, and can then be shared and used by all. The standard's certification by the International Standards Organisation as ISO 17369 means that the standard meets official criteria and will be applicable in the future as well.Technology choice: Standards-based technology, Open source software
Main results, benefits and impacts
The use of the SDMX standard has greatly accelerated the process of publishing statistical data provided by Member States. Where this process was previously hindered by the use of diverging standards, SDMX allows for a much quicker publication, since Eurostat no longer has to invest time in converting the data of the Member States from various formats into its own mode of display.
In addition, the use of a common standard amongst various international institutions is very beneficial. It makes significant savings possible, as it becomes very easy for an organisation using the standard to use data which has already been gathered elsewhere for its own purposes, with minimal effort. The data is collected only once, and then shared between the various organisations, which reduces the amount of funding needed for collecting data. The process of sharing is not only enabled, but also accelerated by the use of SDMX as a common standard
Track record of sharing
SDMX is an initiative that is sponsored by the Bank of International Settlements (BIS), the European Central Bank (ECB), the International Monetary Fund (IMF), the Organisation for Economic Co-operation and Development (OECD), the United Nations (UN), the World Bank and Eurostat. The cooperation between these organisations originated before the SODI project, as they were had already been exchanging data and expertise. The common initiative came into existence after a series of executive meetings, which were further prepared in technical staff meetings in Paris, Maqua explains. Eurostat is the main beneficiary of the greater interoperability provided by the SDMX initiative, â€œbecause in contrast to the other institutions we receive data and pass it on, which is basically only the case with the ECB although it has a much narrower target areaâ€,Â explainsÂ Leonhard Maqua from the Eurostat team. Nonetheless, the initiative also brings many benefits to the other institutions as well. As Maqua says, â€œwe do something, they do something, and afterwards we can all use it together, because SDMX is a common standardâ€.
In the case of the SDMX converter, Eurostat was in charge of the development of the tool. It contracted a consortium of firms, which developed the converter in close cooperation with the Eurostat team. For testing purposes, the team worked closely together with the Bank of International Settlements (BIS) in order to test the software thoroughly and in areas where Eurostat could not have put the program through its paces by itself.Â
Another example can be seen in the European Central Bank's Statistics Dashboard tool, which visualises statistical data in a way that is very clear and easily understandable. Although this tool was not developed by Eurostat, it will intensive use at Eurostat. This in turn will help in further improving the software. Cross-institutional developments are thus not considered an obstacle for the initiative, but rather help all participating institutions in some form or another.
Beyond the group of sponsoring organisations, the United Nation's Food and Agriculture Organisation has also started using the standard, which was a rather positive surprise to the Eurostat team.
For the success of the X-DIS program and the SDMX initiative, careful planning was one of the most important ingredients. Maqua highlights that â€œwe invested one and a half work years into the planning phase of the projectâ€, which enabled the project team to give clear predictions of the requirements for such a program.Â
With basically no advertisement and only with the help of the Open Source community and the widespread use across all sponsoring institutions has the SDMX standard achieved relatively large popularity. The standard is increasingly used by other - non-sponsoring- institutions and the private sector.Scope: International, Pan-European