Open source motor for Basque…

Open source motor for Basque Country’s big data project

27/11/2014

The Government of the Basque Country in Spain is relying on open source for a large-scale distributed storage platform. The system manages most of the file needs of the autonomous region government, handling over one million requests per day.

The systems allows the secure and managed exchange of files between applications hosted in the computer network of the Basque government and in networks of other organisations.

The Basque Country big data implementation combines Apache HBase, a distributed database management system and the Hadoop distributed storage solution. Search functionality is provided by Elasticsearch.

“A few years ago, the exchange of large files between public administrations caused big headaches”, says Oscar Guadilla Jimenez, a software engineer working for the government-owned ICT service provider, EJIE (Eusko Jaurlaritzaren Informatika Elkartea). “IT administrators would say that their systems or servers would not permit it, or that such an exchange was just impossible. So we built our own solution - in just six months, using open source.”

The system now drives the Basque Country’s Platea Integración Services. Platea is the acronym for the country’s e-government platform. Several of its components are made available as open source, one example being Platea Integración Files, made available using the European Union’s open source licence, EUPL.

The solution is also used for the Basque Country’s open data project.

Easy to use

The five developers at EJIE put the emphasis on ease of use, he says. “The exchange works across all servers and desktops.”

The big data system’s web interface allows users to upload files and transfer them to other users. Alternatively, a cloud file service allows users to store, synchronise and share files that are too large to be sent by email, sharing them with others using URLs. One nifty feature is the file upload solution, says Guadilla: “It allows users to upload files and at the same time fill-in the required forms.”

“Our service bus handles more than a million file requests per day”, says Guadilla. The work flow requests are stored in Hbase, and is monitored by the developers at EJIE, using the reports to invent, build and test additional file exchanges.

Scalable and robust

“Our system shows other public administrations that open source is mature for big bata projects”, Guadilla says. “The platform is scalable and robust, and we can track and trace everything that happens to a file, in all detail, conforming to our privacy laws”, says engineer Guadilla.

The software developer presented the big data project at LibreCon, a conference on open ICT, which took place in Bilbao on 11 and 12 November.

 

More information:

Platea Integración Ficheros (PIF) (in Spanish)
LibreCon website

The content of this field is kept private and will not be shown publicly.