Detailed information

Quick reference

Use the following links to jump to a specific section:

 

How does the Test Bed work?

The Test Bed is a complete platform consisting of both software and hardware components with the purpose of facilitating testing. The particular focus in this case is conformance and interoperability testing, ensuring that tested systems conform to a specification’s requirements and can interoperate consistently with conformant peer systems.

The following diagram illustrates the high-level operation of the Test Bed, in which the highlighted elements represent the system being tested and its tester:

Test Bed overview

Testing is organised in test suites, collections of one or more test cases, that define the scenarios to test for. Such test cases link to specifications and are assigned to systems under test once these are configured to claim conformance to them. The management of specifications and their test cases, organisations and their users, as well as the execution of tests and subsequent monitoring and reporting, take place through the Test Bed’s web user interface.

Test cases are authored in the GITB Test Description Language (TDL), an XML-based language that allows the definition of the steps to successfully complete and the involved actors, one of which is always realised by the system under test with others either being simulated or realised by other actual systems. Test steps can vary from validation and messaging to arbitrary processing, manipulating and evaluating the test session’s state, either per step or by checking the overall session’s context. A good example of the latter is ensuring conversational consistency by validating that a received message corresponds to an earlier request.

Each step’s operation takes place using the Test Bed’s built-in capabilities but, if needed can delegate to external components. This is where the Test Bed shines, in its ability to include custom and independent processing, messaging and validation extensions to address missing capabilities or domain-specific needs, by means of exchanges over a common web service API (the GITB service APIs). In using such extensions, the Test Bed acts as an orchestrator of built-in and externally provided capabilities that make it flexible enough to accommodate most conformance testing needs.

When to use (and not to use) the Test Bed

The Test Bed’s focus is on conformance testing, specifically testing IT systems against semantic and technical specifications. The following table summarises when the Test Bed can offer a good solution but also cases when other solutions would be better suited.

Use the Test Bed for: Do not use the Test Bed for:
  • Content validation.
  • Service simulation.
  • Testing of message exchange processes.
  • Testing of conversational consistency.
  • Enabling self-service testing by users.
  • Monitoring of conformance testing progress.
  • Performance testing.
  • Penetration testing.
  • Unit testing.
  • Functional or regression testing within a single application.

In short, the Test Bed is very flexible but nonetheless best suited to test interactions between IT systems. Other aspects such as performance could potentially be addressed, but this should be more as an addition to the core conformance testing setup and not the main goal.

Standalone validators

Not all projects require a full conformance testing platform to realise their testing needs. It if often the case that what is most needed is a means of validating content as a tool offered to the community or as a validation building block of a broader solution.

The Test Bed addresses this need by means of its standalone validation services. These are services operated independently from the Test Bed that focus specifically on receiving content via various channels and validating it against specifications to produce reports. Input can typically be via web user interface, SOAP and REST web services or even via command line for offline use. All provided input and resulting reports are never stored and access is fully anonymous. This allows validators to be used in any context and to be deployed and scaled according to a project’s needs.

Certain validators for commonly used content types are supported by the Test Bed by providing a generic reusable core that can be easily configured for use in specific instances. These allow users to:

  • Create new instances by means of simple configuration and customise them as needed.
  • Benefit from hosting on Test Bed resources and automatic service updates upon change.

The content types for which such extended support is available are:

  • XML: For validation of XML-based specifications using XML Schema and Schematron (see detailed guide).
  • RDF: For validation of RDF-based specifications using SHACL shapes (see detailed guide).
  • JSON: For validation of JSON-based specifications using JSON Schema (see detailed guide).
  • CSV: For validation of CSV data using Table Schema specifications (see detailed guide).

The following screenshot presents a validator for electronic invoices, reusing the Test Bed’s generic XML validation core via simple configuration:

Sample XML validator (electronic invoicing)

All validators, regardless of whether or not they benefit from the discussed extended support, implement the standard validation service API. This is a SOAP web service API that allows them to be used by the Test Bed during tests as the implementation to handle the validation of content. This is an important point as it allows validators to be offered both as standalone validation tools but also to be used as conformance testing building blocks as part of broader scenario-based conformance tests (e.g. validate messages received during exchanges).

Validator usage

The validators currently offered by the Test Bed are listed in the next section.

Technical capabilities

The following tables list the Test Bed’s technical capabilities in terms of specifications, that can be reused when implementing a new conformance testing setup. The listing is split in:

  • Domain-agnostic capabilities that could apply regardless of policy or business domain.
  • Domain-specific capabilities that would typically be meaningful within a given domain.

In each case services are further classified based on their purpose, specifically as messaging services, used to send and receive content to and from the Test Bed, and validation services, used for content validation.

Note that not finding a capability listed here is never a blocking point. It is very simple, often also preferred by users, to cover missing capabilities by defining new extension services using the Test Bed’s template services.

Domain-agnostic

  Specification Description Public API
Messaging services AS2 Sending and receiving AS2 messages. -
  AS4 Sending and receiving AS4 messages. -
  eDelivery eDelivery integration via a Domibus access point. -
  HTTP(S) Sending and receiving content over HTTP(S). -
  REST Consuming and serving REST services. -
  SOAP Consuming and serving SOAP web services. -
  TCP Sending and receiving of content as TCP streams. -
  UDP Sending and receiving of content as UDP packets. -
  UN/FLUX Integration with the UN/FLUX network. -
Validation services ASiC Validation of Associated Signature Containers (ASiC). SOAP
  CloudEvents Validation of JSON objects against the CloudEvents specification (supporting also custom schemas for project-specific rules). UI, SOAP
  CSV¹ Validation of CSV data using Table Schema. See the Test Bed’s CSV validation guide for detailed information. UI, SOAP, CLI, Docker
  GITB TDL Validation of GITB TDL test suites. UI, SOAP
  GraphQL Validation of GraphQL queries against GraphQL schemas. SOAP, REST
  JSON¹ Validation of JSON using JSON schema. See the Test Bed’s JSON validation guide for detailed information. UI, SOAP, CLI, Docker
  Message digests Validation of text digests using various algorithms. SOAP
  SBD Validation of Simple Business Documents. UI, SOAP
  SBDH Validation of Simple Business Document Headers. UI, SOAP
  SHACL Validation of SHACL shapes. UI, SOAP, REST, CLI
  RDF¹ Validation of RDF content using SHACL shapes. See the Test Bed’s RDF validation guide for detailed information. UI, SOAP, REST, CLI, Docker
  REM Validation of Registered Electronic Mail (REM) evidences. UI, SOAP
  Table Schema Validation of Table Schema instances (used to formalise CSV and other tabular data specifications). UI, SOAP
  XML¹ Validation of XML using XML Schema, Schematron, XPath, XMLUnit. See the Test Bed’s XML validation guide for detailed information. UI, SOAP, CLI, Docker
  XML digital signatures Validation of XML signatures in arbitrary XML content. SOAP

¹ Extended support available for specific instances via configuration-driven setup and automated service updates.

In addition to the above supported specifications, the Test Bed also includes numerous low-level built-in capabilities for the arbitrary manipulation and validation of content (e.g. pattern matching via regular expressions).

Domain-specific

  Specification Description Public API
Validation services BRegDCAT-AP Validation of RDF content against the BRegDCAT-AP (Specification of Registry of Registries) rules. UI, SOAP, REST
  CII Validation of electronic invoices as Cross Industry Invoice documents. UI, SOAP
  CPSV-AP Validation of RDF content against the CPSV-AP (Core Public Service Vocabulary – Application Profile) rules. UI, SOAP, REST
  DCAT-AP Validation of RDF content against DCAT-AP (Data Catalogue Vocabulary – Application Profile) rules. UI, SOAP, REST
  DCAT-AP.de Validation of RDF content against the German DCAT-AP extensions rules. UI, SOAP, REST, CLI
  EIRA Validation of ArchiMate models (solutions and Solution Architecture Templates – SATs) against the European Interoperability Reference Architecture and target SATs. UI, SOAP
  ELRD Validation of European Land Registry Document instances. UI, SOAP, CLI
  ESPD Validation of European Single Procurement Document instances. UI, SOAP
  Kohesio Validation of CSV data collection files for the Kohesio portal. UI, SOAP, CLI, Docker
  UBL 2.1 Credit Notes Validation of electronic credit notes as UBL 2.1 documents. UI, SOAP
  UBL 2.1 Invoices Validation of electronic invoices as UBL 2.1 documents. UI, SOAP
  UBL 2.2 eProcurement Validation of procurement process UBL documents as defined in the PEPPOL pre-award profiles (P001, P002 and P003). UI, SOAP

 

Note that the above listing of domain-specific validators is not exhaustive, but presents the cases where a public API is offered and there is good reuse potential.

Deployment models

The Test Bed offers a flexible deployment model to suit most project needs. Projects using the Test Bed can choose to:

  • Reuse the shared instance operated by DIGIT. This is a cloud-based environment shared with other users (i.e. multitenant) where each project is defined as a “community”. Communities can be set as public but are by default private to their members.
  • Install an on-premise instance. Installation and update of such Test Bed instances is fully streamlined and allows projects to have full control of their deployment and use.

The simplest approach is typically to use the shared instance as this is free and removes operational concerns. Opting for an on-premise instance could be preferable however for projects that have access restrictions, specific non-functional needs or that prefer to have full operational control.

The Test Bed also supports the simple on-premise set up of sandbox instances by a project’s users. Project administrators can define a complete testing setup and provide this to their users to set up test environments for local testing. An example where this could be interesting is when an EU-wide project offers National administrations such preconfigured sandboxes for local testing and experimentation before engaging in formal acceptance tests in a shared environment.

Test Bed sandbox usage

Exploring the available deployment options and defining the best approach for a specific project is one of the first points addressed as part its onboarding process.

Test Bed releases

The Test Bed team is committed to making frequent releases to continuously improve the capabilities of the core Test Bed software. Specifically:

  • Planned releases are made at approximately three-month intervals.
  • Unplanned minor updates can also be made if merited by specific blocking issues.

The planning of releases could also exceptionally be adapted to match planning constraints of Test Bed’s users (e.g. a project’s own release milestones). Once a new release is available, the environments operated directly by the Test Bed team are updated and users operating on-premise instances are notified. The update process for on-premise instances is streamlined, with new releases always guaranteed to be backwards compatible with existing data and test suites.

Test Bed release management

It is important to point out that the Test Bed team always prioritises user feedback for its change management. To propose a new issue or report a problem send an email to the Test Bed team at DIGIT-ITB@ec.europa.eu.

Being notified of new releases

Whenever a new release is made the Test Bed team sends its known users a brief email listing key highlights. Such emails take into account received feedback to focus on points that are of most interest. Apart from this however, you can also be notified automatically for new releases by:

  1. Clicking the "Subscribe to this solution link".
  2. Managing your profile's subscriptions to receive email notifications.

Test Bed operational hours

All Test Bed services hosted by DIGIT are available during the Test Bed's operational hours. These are defined as follows:

  • The GITB Test Bed platform is online from Monday through Friday, between 05h00 and 20h00 CET.
  • Hosted validators and documentation resources are online every day between 05h00 and 01h00 (of the next day) CET.

Note that published resources such as Docker images are not subject to this operational window and remain always available. In case your needs are not covered by the provided window and plan to make use of Test Bed services hosted by DIGIT please contact the Test Bed team at DIGIT-ITB@ec.europa.eu.