Migration XXL: The large-scale GNU/Linux roll-out of Lower Saxony\'s tax authority

The migration team opted for a rather restrictive configuration, minimising the possibilities for error or confusion on the users\' part: "We have pre-configured and frozen about two thirds of the KDE settings", states Uffel. "We can prevent the users from doing things they shouldn\'t, such as storing data in certain places, or establishing certain network connections. On the other hand, they have the necessary freedom to adjust things such as color or font settings."

These are things that were not possible with the previous CDE desktop. Especially visually impaired users benefit from increasing the font size, or choosing color settings with higher contrasts.

To configure the KDE desktop and its Kiosk mode, the migration team worked with key KDE developer Eva Brucherseifer and her company basyskom. During several workshops, Brucherseifer helped the team to precisely configure KDE\'s Kiosk mode. She also documented how the configuration process worked, so that the migration team could use this information for future upgrades. Uffel says: "We didn\'t need a finished configuration. What we needed was to learn how to configure KDE and make it do what we wanted to do".

Uffel says that he never even considered a migration to Microsoft Windows. The environment and infrastructure has been based on Unix systems since 1995. Staff have the corresponding knowledge and are experienced in handling this type of operating system. Deploying Microsoft Windows would not only have required the IT staff to re-learn most of their tasks; it would also have required new hardware, particularly on the server side. At the time when the decision to migrate was taken, the tax authority used PowerPC servers, an architecture for which Windows is not available.

On the new GNU/Linux system, Mozilla Firefox is used as the standard web browser. Email functionality is currently provided by Lotus Domino, which will soon be replaced by Mozilla Thunderbird.

A number of specialised tax applications which had been in use on the old Solaris systems were written in COBOL or Delta Prolog. Most of these were ported to or re-programmed in Java, in order to work on the new GNU/Linux system, but some of these applications proved too difficult to port. These programs are still in active use on Solaris gateways in each tax office, and can be accessed remotely by users.

 [top]

The migration did not have much of an effect on the hardware deployed: "We usually exchange the PCs in the offices about every two years, in order to keep up with the RAM requirements of our applications." In the fall of 2005, 6000 PCs were replaced with then-current machines (Pentium 4, 3 Ghz, 1GB RAM). These will probably remain in service until 2009 or 2010.

Currently, new hardware is being bought to replace the computers that have been in service since 2002 (Pentium 3, 1 Ghz, 512 RAM - non-expandable). Though these older machines run fine under the Suse GNU/Linux operating system, their performance is too limited for the memory-hungry Java-based applications. The new hardware will have a performance similar to those acquired in 2005.

[top]

The roll-out was conducted centrally from the Hannover IT centre. The tax authority\'s computers are equipped with a so-called BootPROM module, which makes it possible to install new operating systems over the network, remotely and unattended. IT staff can simply send the operating system image to the desktop computers in question over the network. But careful timing was necessary in order to avoid congesting the network, as huge amounts of data needed to be transferred. The migration team usually attempted to do this work during the night or over the weekend, when there was little other traffic.

Initially, the migration team mentioned their plans in several meetings with other tax authority management staff. They also put out a number of internal announcements. As the migration got delayed for a number of months, Ute Beckmann, the leader of the tax authority\'s IT training team, recalls that there was a certain degree of unrest. The workers waited for something to happen, but nothing did. Ironically, this meant that people were quite relieved when the migration went ahead smoothly, some asking "why did everybody make such a fuss, if the new system is this easy?"

Early on during the preparatory phase, Uffel sought contact with the tax authority\'s voluntary IT trainers. This is a group of about 30 volunteers from among the tax authority\'s employees, who use 30-50% of their working time to provide basic training to users. During three or four workshops, they tested and commented on the new system. Requested changes were then implemented in another workshop with Ms Brucherseifer. According to Beckmann, the IT trainers\' feedback greatly helped in making sure that the default configuration of the desktop was easy to use for the tax authority\'s employees.

After the migration schedule was fixed, the IT trainers were instructed in the use of the new system. In the tax offices where they were based, they were then given a computer on which they could work and experiment with the software from October 2005. Again, their feedback gave the migration team the chance to remove some more errors.

Local system handlers (Uffel: "our hands and eyes on-site"), about 140 people, were instructed for 2-3 days, a few weeks ahead of the migration. They learned how the roll-out would work, which problems might occur, and how they could solve some of these locally.

The tax office at Göttingen acted as a pilot for the roll-out. According to Beckmann, it was chosen because it has a number of very active IT trainers, making it a good proving ground.

[top]

As the roll-out began, the IT trainers went to each tax office to train contact persons there, making use of small IT training rooms called "learning islands", which exist at nearly every tax office. These contact persons, representing about 10% of each tax office\'s staff, received an intensive training. They then trained the rest of their colleagues, and helped them during the transitional phase. All employees also received a flyer explaining the migration process.

Right after (or sometimes even during) the training, the computers of those workers which were trained were migrated to the new system. All employees also received a flyer with cookbook-style instructions for the most important operations on the new KDE desktop.

This "layered" training concept had been developed by the tax authority for some time. Beckmann recalls that when computers were introduced in the 1990s, the IT department trained all users directly; this didn\'t work out too well. During the past two migrations, she and others refined the training concept. This time, the experiences were very positive, as reflected in the results of the user suryey (below). Beckmann says that she will continue to use this training structure for future software deployments. "If this worked well, why should we do it any differently?"

One challenge was to make sure that the local management would actually make available the time for people to attend trainings. When a tax office entered the migration process, its management was obliged to offer training to all employees within three months. Beckmann says: "We handled this pretty rigorously this time. We ordered the tax offices to offer training, and to report back to the IT department who had been trained, to make sure the trainings really happened."

[top]

According to Beckmann, who conducted an online survey among the tax authority\'s workers to evaluate their views on the migration and the new system, users\' views were "favourable, absolutely favourable".In a voluntary online survey four to eight weeks after the migration, Beckmann asked users about their views of the migration process. She also surveyed the contact persons to find out what their workload related to users\' questions about the new system was.The answer where overwhelmingly positive, with a large majority of users stating that they liked the new system and that they felt "very well" or "well" prepared for the new software.

"The response really couldn\'t be much better. Some people even called me up and said: \'This went really smoothly. Why was there so much fuss about the migration beforehand?\'", comments Beckmann. "The frictional losses we experienced were really small. We had expected it to be much worse."

[top]

Most of the work related to the migration was done within the usual range of responsibilities of the IT department. Although the migration required a substantial amount of effort, this was not related to the use of FLOSS (Free/Libre/Open Source Software); similar effort would have been required for any migration on this scale.

The IT department has ca. 180 total staff. Of these, 30-40 came into contact with the migration effort, but only Uffel\'s core team constantly worked on this task. This team consisted of 3-4 people adapting the KDE desktop and the operating system, another four working on office software, two IT staff taking care of the email client. Specialised tax applications were adapted by between three and fifteen people, depending on how deep the changes were. Three apprentices were also involved.

The IT department spent between 300.000 and 500.000 EUR on external services, mostly for adapting the distribution to certain special conditions of the tax authority\'s environment, and to simplify the roll-out process.

To fully account for the cost of the migration, Uffel also includes the time that employees spent in trainings instead of going about their work. Assuming that an average training session lasts three hours, these opportunity costs amounts to 750.000 EUR for the contact persons, and to 3.5 mio EUR for the rest of the 14.000 employees whom the contact persons trained.

For the previous Solaris systems, the tax authority used to have a service contract costing about 2 Mio German Marks (about one million Euro) for three years at the end of the 1990s; this included upgrades for new Solaris versions.

During the 1990s it was paying 300 German Marks (about 150 EUR) per seat in licencing fees. This grew to ca. 500 EUR for new versions of Solaris in the years after the turn of the century.

Today, there is no service contract with SUSE (or anyone else). Uffel says that if a problem were to arise that the IT department could not sort out by itself, they would strike up a short-term contract either with Novell or with another company with the necessary skills.

An additional advantage is that the choice of compatible hardware is greater for GNU/Linux than for Solaris, meaning that it is sometimes possible to buy cheaper equipment. There is also a much greater choice of software.

According to Uffel, there were no legal difficulties. Initially, Uffel was unsure whether he could, as he planned to do, simply buy one set of SUSE DVDs, and then use the programs on them on thousands of machines. By now, he is positive that there is no problem in doing so. "As to using the software on all these computers, the licences clearly permit that. And by now we know that they think so as well, or they would have reacted as soon as they heard about our project", says Uffel, who is in frequent contact with Novell/SUSE.

[top]

The training effort was key for the success of the migration. Beckmann is sure: "If we hadn\'t done [the training] in such an elaborate manner, things probably wouldn\'t have gone so smoothly. We have experienced that when we provide less training, people complain through every channel that\'s available to them. But this time, there was nothing, absolutely nothing." Routine certainly helps: "The effort for the organisation of the trainings shouldn\'t be underestimated. We\'ve been doing this sort of thing for some time, so we have quite a bit of experience with such roll-outs."

Making sure that everyone had the opportunity to receive training was as important as closely synchronising each worker\'s training with the migration of her computer. Thanks to the system of contact partners, everyone had someone they could ask when they ran into trouble, leaving few questions unanswered.

Remaining problems after the migration are minimal. They mainly occur when other public bodies send complex documents in Microsoft\'s proprietary .doc format. The StarOffice word processor that is currently in use does not handle some of these correctly. Uffel hopes to improve this with the deployment of OpenOffice 2.2 later this year, making it necessary to port the tax authority\'s macros. In the meantime, the IT department has set up a central conversion service, where employees can send problematic documents to be converted into formats that the present software can work with.

[top]

According to Uffel, today\'s systems are based on FLOSS "almost exclusively". An advantage of using FLOSS is the flexibility when looking for outside support. An instance of this were the workshops with Eva Brucherseifer during the preparatory phase. "We needed someone with good knowledge of the KDE source code, and with good contacts to the developers", says Uffel. "We would then hire her for five or ten days, and develop what we needed together with her."

Uffel says that thanks to the availability of source code, IT staff can quickly locate problems in the programs, and decide whether to patch it themselves or hire an outside developer with the necessary expertise. In a large administration, the approval of funds for such purposes takes some time. This means that the IT staff often resorts to community forums to fix a problem or at least find a temporary workaround: "External experts use these forums too", remarks Uffel.

Uffel points out that the tax authority\'s migrations was rather specific on a technical level, as they were moving to GNU/Linux from Solaris, rather than from Microsoft Windows. But regardless of the migration path, he recommends keeping the time between training and actual deployment of the new software as short as possible. Uffel also advises to choose a "soft" migration stragey, with small groups of users being migrated one after the other, rather than replacing all the administration\'s software in one go.

Communicating the migration is also important, says Uffel. If the migration team manages to get users interested in the migration and curious about the new software, they will be much more prepared to accept it. Given the users\' positive response, Beckmann concurs: "Linux now is an everyday thing here, and nobody is looking back."

[top]

Migration XXL: PDF version

[top]