Risk aversion might be the easiest and safest way, it's also very expensive
The city of Ede, the Netherlands, currently has an annual total ICT budget of six million euros. According to the Dutch Berenschot benchmark for municipal ICT costs, that is 24 percent less than other municipalities of comparable size are spending. Drilling down shows that most of this reduction can be explained by Ede's extremely low spend on software licenses: only 56 euros per full-time equivalent employee (FTE) instead of 731 euros. That's a very impressive 92 percent less than average. Such a large reduction was achieved by moving from proprietary to open source software.
1 Vendor lock-in
Municipal software oligopoly
The Dutch ICT market for municipal applications is dominated by two players (all in Dutch): PinkRoccade Local Government and Centric. These integrators traditionally have been reluctant to support true open standards and open source, instead employing lock-in to "protect" their business interests.
Centric is one of the largest suppliers of municipal applications for allowances and taxes, Ede's Computerisation and Automation director Bart Lindeboom explains.
The other is Procura (in Dutch), and then there are some niche players for smaller modules. That's very different from the market for permits, for example, in which there are as many as five active suppliers.
Centric also sells Oracle licences as part of their business. In fact, in the market for municipal applications they are the main supplier. So they absolutely don't want you to switch to PostgreSQL, because they would lose both sales volume and margin. So they have a strong interest in maintaining the status quo, and in keeping us on the Oracle drip. That's how they make money.
Oracle on Linux and Itanium
Lindeboom provides various examples of how these companies use lock-in trickery to keep their customers in a position of dependency.
I fought a tough battle with Centric — and lost — for the use of Linux on Itanium servers. Despite the fact that the platform is supported by both HP and Oracle, Centric was not willing to support it. They didn't even want to touch it. One day, one of their support engineers was working on an application running on a Linux+Itanium system. He was not even aware of it until everything was running smoothly and he was almost done. Then he dropped everything he was working on and said he could not complete the job because the platform was not supported by Centric.
Centric was absolutely unwilling to support our configuration. The only reason they could give, however, is that they can't support everything. If I wanted to change things, they said, I should go to the user group and try to arrange it over there ourselves. They know, of course, that I can't go to the head of the taxes department over here with a story like that. I have to provide guarantees for my services to the internal customers. So we're running Centric on HP-UX on Itanium for now. As soon as we switch to Xeon processors we will replace HP-UX with Linux as well, because that configuration is supported by Centric.
No additional risk
Despite these problems, Lindeboom is more than satisfied with his Itanium servers.
The platform is very robust and the systems were no more expensive than Xeon-based servers. But when you are deploying Oracle databases, for example, it's so much cheaper. The prices of their licenses are calculated based on the number of cores and processors. So Itanium is your best choice, because this hardware platform provides the most performance per unit. For us, Linux on Itanium delivered the highest value for money. Open source might be a belief, but it is also a matter of calculating.
The city of Ede is probably one of a very few municipalities deploying Itanium systems. Lindeboom, however, does not see this as an additional risk.
There is a supplier that I can fall back on, he says.
Reducing risk using open source
The use of open source software can reduce risk too, Lindeboom continues:
Each time Apple changes its iOS (the iPhone and iPad operating system), for example, there are issues with Microsoft Exchange, especially for people who have more than one device. If you are using an iPad and some other mobile devices, and these are all communicating with Exchange, unpredictable things will happen, meaning for example that appointments and meetings go awry. The Microsoft website is filled with pages warning how you shouldn't use more than one device, and tons of instructions on how to make it work with Exchange.
But we don't use Exchange over here, so we don't have these problems. We have deployed Zarafa [a Linux-based open source mail and groupware replacement for Microsoft Exchange Server]. So when things go bad for the rest of the world after an iOS update, we have it fixed a lot sooner than those Exchange users. Zarafa is really fast with its patches. That is a huge advantage of open source: we are not depending on large suppliers using technical tweaks to force you into buying their whole ecosystem. Using open source software, we actually reduce our risks.
Outdated proprietary software model
Smartsite provides another example. It's a proprietary Microsoft-based content management system (CMS) used by many Dutch local and regional public agencies.
Years ago we had to decide which system to buy, Lindeboom recalls.
A lot of people wanted to use Smartsite, because at that moment it was the market leader. Only a smaller group of web-masters and ICT specialists said we should deploy TYPO3 [a CMS especially popular in Europe and Germany]. I asked Seneca, the supplier of Smartsite, to look into TYPO3, because they did have a lot of expertise. So TYPO3 might have been an opportunity for them — if the world stops buying submarines, you should start producing tin cans. They didn't, and now they are losing market share rapidly.
2 Moving to open source
No fat clients
The city of Ede is already running dozens of open source solutions: LibreOffice, Zarafa+webmail, Firefox, TYPO3, zaaksysteem.nl (in Dutch), MySQL, PostgreSQL, Nagios, and Asterisk, to name but a few.
Quite a lot of the people here are still using Outlook, says Lindeboom,
so it is still part of the VDI (virtual desktop infrastructure). Because of the licensing costs, however, they are not allowed to use Outlook from home. There they use Zarafa's webmail entrance. We don't offer Thunderbird [an open source e-mail client by Mozilla, the makers of the Firefox browser] either, because I don't want another fat client to maintain. And frankly, I think Zarafa offers a better webmail interface, facilitating plugins for video-conferencing and presence, for example.
Breaking lock-in using Firefox
Yet we did deploy Firefox as the standard web browser, Lindeboom continues.
We used it to force our software suppliers to remove any lock-in to Internet Explorer. Dependencies on this deliberately non-conforming browser allow companies in the Microsoft ecosystem to piggyback their lock-in strategy.
Luckily, I could explain it in platform terms as well: Firefox is also available for Linux and OS X. That gives me the freedom to choose the platform I prefer, and I have to train the users only once.
Vicrea's Neuron Stroomlijn (in Dutch) is an information query application used by over 70 Dutch municipalities.
Originally, it ran only on Internet Explorer. On several occasions I made clear to the developers that we would throw it out, unless they made it available for Firefox. It took them some time, but now it does work with Firefox too.
So Firefox has proven to be a good tool to break the Microsoft lock-in by our suppliers. Telling them that you will be standardising on Firefox offers you a way out. Apart from that, Firefox is a great browser from a security perspective as well. They are always the first to publish a patch.
Gentle push to open source software
According to Lindeboom, the user experience is no problem at all.
Open source or closed source, it doesn't really make a difference to them. Do we need to train our people to use Firefox? Of course not. We did train them on LibreOffice, however, but that's basically a matter of showing them where certain buttons and functions are placed. And we did an impact analysis where people could complain about everything that wasn't working for them, and we showed them how these issues could be solved. We also made a tool-kit available.
Of course, users can find out things for themselves too. Google is a great help if you want to know how to handle those proprietary .docx files. Our users are slowly turning into open source adepts as well.
Our strategy is to gently push our users in that direction. You can use Microsoft Office at your workplace, for example, but not from home because of the license costs involved. LibreOffice, however, is offered at both locations. So a lot of interface and conversion issues are solved if you start using LibreOffice at work too. We now see our users moving away from Microsoft Office. In the meantime, of course, we have to script for both application suites.
3 Open source savings
Open source software procurement
Our case management system was acquired using a call for tender which stated upfront that the price would have to be below 200,000 euros, says Lindeboom.
We were actually surprised that some proprietary solutions were being offered, sometimes even for less than half of their ordinary price. But we also got reactions saying that it was absurd and impossible.
This tender was won by zaaksysteem.nl, an open source solution. But if a closed source offer had been a better choice, we would have taken that. We officially prefer open source, and most of us have become open source enthusiasts, but it's not sacred. We don't (yet) put OSS-specific properties in the requirements of our tenders; currently, it's "hidden" in the budget.
At the same time, we made it possible for open source suppliers to participate in our public tenders. We used the NOiV action plan 'The Acquisition of (Open Source) Software' as a starting point, and we removed all this nonsense about "minimum turnover" and "guaranteed further development" from our calls for tender, to allow open source solutions to be offered as well. That way we have built up our whole software procurement policy from scratch.
Pressure from proprietary parties
To be honest, I expected a visit from Microsoft when we moved to LibreOffice or Zarafa, Lindeboom continues,
but they never showed up. If they had made me an offer I couldn't refuse, we might very well have been using Microsoft for the next few years. I cannot ignore them; it is tax money, and I have to look at the costs. More and more software products are being released as open source, because they are the least expensive and it is strategically the smartest model to use, providing transparency and sustainability for free. It could be that proprietary solution providers have already given up on us.
One organisation who did show up was the Software Alliance (BSA).
Two years ago they installed a sniffer in the network, because we weren't as "organised" as we should have been. So we had to buy additional Microsoft licenses.
Despite that, the savings we have managed to achieve are huge, Lindeboom assures us.
We participate in the Dutch Berenschot benchmark for municipal ICT costs (Benchmark ICT-kosten; in Dutch). In 2011, our ICT management costs were at exactly the average. And last year, after correction for the services we are providing to external customers, these costs were only slightly higher (5-10 percent) than the average. But we more than make up these costs in licenses. We pay only one tenth (!) of what other municipalities are paying for their software. No wonder if you look at all the open source applications we have deployed here.
ICT costs 2012 per FTE of the entire organisation. Source: Berenschot.
the city of Ede
average in comparable municipalities
depreciation on hardware and software
other computer facilities (i.e. data communications)
All these open source services run on Linux.
We have tens of these systems. We're not standardised on a specific distribution, but most of them are SUSE-based. All systems are virtualised, so it's very easy to create a new one. We can have a new Linux server up and running in an afternoon — and it's free — so we do it all the time. Furthermore, they are all independent of each other. I prefer separated environments for development, testing, acceptance, and production (DTAP), for example. Using Linux, MySQL and PostgreSQL, it's very easy to make a clone of an existing system. It would be very expensive to do the same with an Oracle environment.
Regional service centre
According to Lindeboom, the savings from open source are easily enough to employ another eight people, for example to accelerate innovation.
But that's not how it works over here, and that is a good thing. The savings are cut from our budgets, and that way they are given back to our citizens. It's not up to me to allocate municipal resources. Maybe these savings are badly needed to finance local care. That's the responsibility of the city's Board of Aldermen.
Still, I do want to innovate and expand, and I do run my department in a business-like way. I'm often able, for example, to offer ICT services to our regional partnership at less than half of what the most expensive municipal bidders offer. And that has nothing to do with volume, but everything to do with the choices we make. I'm really surprised that the secretaries at the other municipalities are not seriously considering restructuring their infrastructure like ours, or even have us run it for them. It's not up to me — I'm a public servant with the city of Ede — but there is gold in having us run a regional service centre. Of course, apart from the economies of scale, the city has no direct interest in insourcing ICT infrastructure from other municipalities.
To manage all these systems and software, the Ede ICT department employs about ten system managers, specialising in Windows and Unix/Linux.
They are also responsible for the network, Lindeboom explains.
Most of the applications are managed by suppliers. For others we hire expertise for the initial set-up. Campai (in Dutch) helped us with the installation of Zarafa, for example. For Nagios we flew in people from another service provider. We also use these companies to train our users and system managers.
OpenWAVE (in Dutch) provides another example. It's a case management system for environmental permits, and was released as open source by its original owner, REM Automation.
We challenged them to go that route, and it did bring benefits to their business, Lindeboom says. The company still takes responsibility for upgrades, further development, and services. Larger projects are initiated by the user community.
You can download and install it all by yourself, of course, if you want to. But just like all the other municipalities, we are unable to maintain over 400 applications ourselves. So we rely on REM for support.
Eighty percent of the configuration and operational tasks you can try to do yourself, says Lindeboom,
but you have to know which twenty percent to leave to others. For tasks that are too specific, it's impossible to build the required expertise. Running a single instance, you will never be able to match the level of a specialised supplier who is solving similar problems every day. Furthermore, it's a bad investment, because you are not able to sell your expertise to others.
Still, in Ede we try to do as much as we can ourselves. Sometimes we start projects that we cannot oversee completely at the inception. Most ICT departments avoid risks as much as possible. The traditional manager does not want to innovate; he wants to keep the status quo. Although risk aversion might be the easiest and safest way, it's also very expensive. So in Ede we try to do things in a different way.
Open source software introduction
Five or six years ago, most of our systems were running Windows, Lindeboom continues.
But we have always been running HP-UX over here, previously on HP 9000, the predecessor of the Itanium-based Integrity Servers. I love those machines; they are incredibly robust and reliable. So we kept them running long after their economic depreciation. As long as the maintenance costs are lower than the capital costs, you are saving money. Of course, you have to look at the risks, so we closely monitored the frequency and severity of outages, but that has never become a problem. Over here, we don't replace monitors because they are five years old. We only do that when they're broken or consume too much power.
For the introduction of Linux, we found the city of Rijswijk to be a good example. So we had one of their people come over here to talk about their experience with this open source operating system. And we started as far away from the users as we possibly could: in the services. Our HP-UX system managers had no trouble at all taking Linux into their care; there were so many companies already deploying Linux in large, mission-critical environments. The next step was the deployment of TYPO3, then Zarafa and Nagios. That's when proprietary packages were phased out, and that's how open source software slowly took over our infrastructure.
When we brought in Zarafa, the only notable difference to the users was that their mailbox space went from 50 Mbyte to unlimited. And the web interface of Outlook was terrible to use anyway, especially from devices without Internet Explorer. Now they suddenly could move the contents of their .pst files to their inboxes. So they were actually very happy with the transition.
From clickers to typers
Generally, Linux/UNIX systems are managed by people with a higher level of education than the people managing Windows. That's a difference in technology — Linux applications are managed using scripts and configuration files, while Windows uses a cascade of configuration windows — but also in culture: UNIX is famous for its orthogonal design and concepts, while Windows systems simply need to work.
This division can also be found within Ede's ICT department.
No one has left as a result of the transition. Some of the Windows clickers became typers, but most of them were already involved in MSI scripting. The introduction of Zarafa was certainly hard for our Exchange manager. This open source messaging system is based on Linux and Apache, so he had to familiarise himself with a whole new software stack and a new way of working.
Along the way, most clickers learned that typing is a very handy thing, says Lindeboom.
Configuration files offer you far more flexibility, and [shell] scripts are a very efficient way to automate repetitive tasks. When you have reorganised your storage, for example, you don't want to spend an afternoon dragging-and-dropping home directories. That's what people have been learning here over the last years.
Nevertheless, we still do quite a lot of clicking. We still have a fat client in our VDI (virtual desktop infrastructure), because even now most of these 400 municipal applications are Windows-based. It will take years to get rid of them. The software has an economic lifecycle of five years, so in theory that should do the job. But we've learned that it's very hard to break this lock-in, although it becomes easier along the way.
We also put pressure on ourselves by booking savings in advance, Lindeboom continues,
so the funds will simply not be available in the future. For example, the cost of the VDI was 1.2 million euros. We will not have that budget next time, so we have to phase it out. We are doing the same for Microsoft Office. When the budget is gone, you no longer have to think about whether to buy new licenses or not. You just have to find yourself an open source alternative.
There is no additional risk attached to this approach. We don't have Software Assurance [Microsoft's paid-for software upgrade program], so we can keep on using our current licenses for as long as we want.
5 A rapidly changing world
Although government agencies generally can afford to follow a long-term approach, Lindeboom emphasises that his world is changing rapidly.
There is a lot going on: cooperation at municipal and regional levels, and the set-up of centralised service centres. At the same time, cost considerations are forcing municipalities to outsource their ICT to the traditional oligopolistic parties. PinkRoccade is currently lobbying for public agencies and policies to steer in that direction. To keep the costs down, they are looking into open source software themselves now.
Centralisation of municipal ICT
Despite the fact that in the Netherlands more and more government tasks are currently being transferred to the municipalities, Lindeboom sees an undercurrent moving in the opposite direction when it comes to the underlying ICT systems.
Services are being nationalised. On the one hand, central government wants the municipalities to be the main entrance for their citizens. That's where all interactions with government should start. Currently, each municipality is running all these systems from suppliers like Centric, PinkRoccade and Procura. They all have their 'Gemeentelijke Basisadministratie Persoonsgegevens' [GBA, the Dutch population register], they have all the permits, and they know all about taxes.
On the other hand, there is a very slow and subtle movement of the supporting ICT systems to central government. The city is still the main entrance. At the same time, however, the website mijn.overheid.nl [in Dutch; i.e. my government] is increasingly the central portal where you can look into your personal affairs and find out what is going on in your local environment. This site is starting to function as a central service centre.
That means that there have to be central databases too, because Omgevingsloket Online [in Dutch; OLO, the central portal for environmental permits], for example, cannot be connected to all these local systems running at the municipalities. The same has been true for years for the population register. So in the future, municipalities will no longer have to think about what database system to place under their register; it will be based on a centralised platform.
Centralised local taxes
These days, a visionary government no longer deploys its applications at a local level but organises a centralised infrastructure. Why should every single municipality levy taxes for dogs, sewage and properties? Why isn't that being done by the national tax and revenue office? They already do motor vehicle taxes, healthcare benefits and income taxes. For them to do sewage taxes as well shouldn't be a problem at all.
Of course, the municipalities might be afraid that centralising their tax systems will be the prelude to losing control over these taxes. To overcome this reluctance, it should be clear from the beginning that the national tax and revenue office serves only as a centralised service provider for these levies.
That shouldn't be a problem, says Lindeboom.
For example, motor vehicle taxes also include a percentage for the provinces. They are levied by the national office and distributed over the Dutch regions. Currently, a structure like that does not exist for any of the municipal taxes. But this example shows that it can be done.
400 separate implementations
Lindeboom calls this situation ridiculous.
Each municipality spends a lot of money on tax systems. Across the Netherlands, tax systems have to be bought and implemented 400 times. The city of Ede spends about 200,000 euros on its tax systems every year, and that's without the services and the underlying databases. If you do all the adding and multiplying for the whole country, the total amount is huge. It can easily be done at a central level for a fraction of the money.
Fortunately, we are slowly but steadily moving towards a centralised infrastructure for local government. You can see it currently happening with OLO, mijn.overheid.nl, and GVOP [in Dutch; Gemeenschappelijke Voorziening Officiële Publicaties, a centralised regulations publication platform for local and regional governments]. So, following the databases, applications are being centralised as well, and delivered to citizens and public servants through a web interface.
Moving away from Oracle
If this trend continues — and I think it will — all of these Oracle databases will eventually be phased out of the municipal data centres, Lindeboom predicts.
You don't use an Oracle database to store data on 17 million Dutch citizens. The scale of an operation like that both requires and allows a different solution.
In a small municipality with 20,000 citizens, the ICT department is run by only a handful of people. It's very hard for them to manage all these different platforms, technologies and applications. They already have a couple of Microsoft and Oracle databases, and then some. It's impossible for them to develop Linux and PostgreSQL expertise on top of that. These municipalities are far too small to acquire and maintain this kind of knowledge and skills. That makes them dependent on suppliers which claim to act on the best interest of their customers.
In the meantime, however, the people at Oracle are laughing. In this transitional phase, municipalities are splitting off parts of their infrastructure and organising new collaborative partnerships. From Oracle's point of view, however, each one of these is a new entity, requiring separate licenses. So at this moment, consolidation brings additional costs. For the De Vallei region, an environmental partnership set up by the cities of Ede, Barneveld, Nijkerk, Scherpenzeel, and Wageningen, we had to pay 80,000 euros to Oracle, and an additional 20 percent every year for maintenance. That's why we are so eager to migrate to PostgreSQL, despite this huge municipal application landscape. We're currently working our way to freedom.