Why interoperability testing?
The continuous growth of information exchanges between public administrations (across borders and sectors) gives a high need for interoperability in Europe. The ISA² interoperability test bed (Action 2016.25) provides generic testing facilities to initiatives and public administrations that create interoperability solutions in a cross-border context or linked to European Legislation. Achieving interoperability is a challenging task, considering the range of different technologies that are used, the diverse organisational structures they support and the varying semantics that may be applied to the exchanged information.
This testing challenge is in fact twofold, on the one hand on the system under development that needs to run interoperability tests as early as possible in its elaboration phase, and on the other hand on service providers or peer systems that need a means to expose IT services for testing without impacting their production system operations.
What is the ISA² interoperability test bed?
The ISA² interoperability test bed is a set of hardware and software resources offered as a central, standards-based reusable service, to facilitate interoperability and conformance testing. Examples of its supported use cases would be:
- Simulation of a web service for clients to test against.
- Validation of content sent through various channels.
- Conformance testing against a message exchange protocol.
- Testing of an entire message exchange choreography.
The test bed allows users and systems to connect for the execution of test cases against simulators or reference implementations of specifications that are transparently hosted on its infrastructure. In addition it offers a test registry and repository (TRR) to store test artefacts (assertions, test cases, validation schemas etc.) and federate test services (validation services, simulator services etc.). The centralised test bed service is cloud-based but separate instances can also be easily ran on premise if such a requirement exists.
What should it not be used for?
Given that the test bed’s focus is interoperability and conformance testing there are some cases that it is less suited for:
- Functional or regression testing: The test bed's focus is on interoperability, not finding internal bugs.
- Performance testing: The test bed aims for configurability and traceability of test sessions, not the automated generation of traffic for stress testing.
- Penetration testing: To enhance connectivity and ease of use, security is relaxed and exchanged data is exposed.
How do I use the test bed?
Using the testbed can be summarised in three steps:
- A user logs onto the test bed platform and selects one from a set of reusable test cases.
- The test case executes, testing the user’s system against simulators and validators. Tests can range from the verification of complex message exchanges as complete conversations, to validation of content, received through a variety of communication channels.
- During execution and upon completion, exchanged messages and validation results are displayed and compiled in a test report that is stored for later reference.
The picture below shows a screenshot of an executed test case. It visualizes the actions done by the user and the Test engine, and it gives for each step a report of the results.
The main users of the test bed are initiatives such as public administrations, projects or other, which have relevance to interoperability. They can make use of the test bed by having their reference implementation and test suites installed on it, so it is ready for their community to be used. Anyone who has a relation with this community can then test their software against these test suites.
Anyone, including someone not eligible to directly use the ISA² test bed service, is free to download the test bed software and install a local instance in their own premises.
Where to find the test bed
The following options are available to start using the test bed:
- The ISA² test bed instance is accessible at https://www.itb.ec.europa.eu/itb. This is a shared instance operated by ISA² that anyone from the interoperability testing community can use.
Install a self-hosted version of the test bed software. To do this it is recommended to follow the installation steps explained in the document “Introduction to the ISA test bed”.
Requests for more information and potential collaborations can be sent to DIGIT-ITB@ec.europa.eu.
Provide us your feedback
The ISA² Interoperability Test Bed Action is constantly trying to improve the Interoperability Test Bed and its supporting services to best match the needs of the interoperability testing community. If you have a few minutes to spare it would be highly appreciated if you would consider completing our feedback survey. Please don't hesitate if you are not already familiar with the Test Bed; all input helps and it could be that you also discover information that is useful to your project.
The survey is accessible at https://ec.europa.eu/eusurvey/runner/itb.
Presentations, leaflets and extra information:
Resources on the GITB Test Description Language (TDL), test services and online guides:
- The GITB TDL Controlled Vocabulary (SKOS).
- The online GITB TDL documentation.
- The online GITB test services documentation.
- The test bed's online guides and tutorials.
The Interoperability Test Bed user guides:
- For organisation users (i.e. basic test bed users).
- For organisation administrators (i.e. administrators of organisations registered for testing).
- For community administrators (i.e. administrators for sets of organisations corresponding to a project or user community).
- For test bed administrators (i.e. overall system administrators).
Resources on the GITB test bed software