When the tax authority Lower Saxony needed to upgrade the Solaris software on the 14,000 desktop computers in its 70 offices throughout the state, it chose Suse GNU/Linux as a replacement. Taking place between April and October 2006, roll-out and training were organised to interlock tightly. Cost was not as much of an issue as flexibility and a modern desktop environment. Today, users are very satisfied both with the new software and the training they received.
This case study for download
Migration XXL: PDF version
14 000 workers, seventy locations, one system
The German Land of Lower Saxony is the country\'s second largest state by area. Its population of nearly eight million people makes it the fourth largest in this category.
The tax authority\'s IT department in Hannover centrally administers the IT systems for all of the 70 tax offices in Lower Saxony. The offices process tax declarations filed by citizens and companies. For the most part, they are all dealing with the same kind of highly standardised data, making such a highly centralised system an efficient option. These tax offices are distributed around some 65 locations in the state, and employ roughly 14000 people. The size of each tax office varies between 90-100 and 300-350 employees, which mostly use one computer each. In addition to these employees, there are 2500 field workers using laptops, which currently run on Microsoft Windows 2000.
After the migration between April and October 2006, almost all employees\' desktop computers are running a SUSE 9.2 Professional GNU/Linux operating system. 180 machines continue to run on the previous Solaris 8 system in order to provide access to specialised applications that the IT department decided not to port to GNU/Linux.
Reasons to move forward
The tax authority had been using Sun\'s Solaris operating system ever since computers were introduced into its offices in the mid-1990s. Solaris 8 was deployed in 2002. It became clear that another upgrade would become necessary in 2005 or 2006.
In addition to the age of the system, which meant that the computers were running software far behind the state of the art, it became increasingly difficult to find suitable hardware. Andreas Uffel, the IT department\'s team leader for the Unix front-end, says that the options were to either migrate to a more recent version of Solaris, "so we could use hardware that you wouldn\'t have to buy from a museum or an antiques dealer"; or to migrate to a more common platform, with better hardware and driver support.
The graphical desktop environment was another problem. Solaris\' CDE desktop had become outdated, offering ergonomics that were below current standards. The tax authority\'s employees also needed a Java Runtime Environment, in order to run the specialised tax applications that are developed in cooperation with the rest of the German states.
Complete roll-out in seven months
During the last operating system upgrade in 2002, Uffel had already looked at GNU/Linux as an option. That time, he had decided against it because of hardware compatibility problems. But the next time around, in February 2005, the decision was taken to migrate to SUSE GNU/Linux 9.2.
This particular distribution was chosen because some of the IT staff had extensive experience with the system. Another reason was that the company behind SUSE was based in Germany (although it was being bought by the US outfit Novell at the time), supposedly making it easier to negotiate a service contract.
In the subsequent preparatory phase, Uffel\'s migration team checked that the existing hardware was compatible with the new system. From the SUSE repositories, they selected the packages that were going to form part of the new system, and put together a "gold master" operating system image that would be distributed to all computers.
After putting a lot of effort into designing the strategy for training and roll-out, the tax office in Göttingen was migrated as a pilot between February and April 2006. No major problems occured with the system, and the second tax office entered the migration process a few days later. By Oct 25, 2006, the migration of all tax offices in Lower Saxony was complete.
Preparing the KDE desktop: restrictions, defaults and user choice
"Our users have certain expectations as to continuity in their applications, which we have to meet as a minimum. This was the reason to choose the KDE desktop", says Andreas Uffel. The K Desktop Environment (KDE) does not only bear some resemblance to CDE; its "Kiosk" mode also gives the administrators the chance to determine which settings each user can modify, and which ones are fixed.
The KDE logo - © KDE e.V., licenced under GNU LGPL
The migration team opted for a rather restrictive configuration, minimising the possibilities for error or confusion on the users\' part: "We have pre-configured and frozen about two thirds of the KDE settings", states Uffel. "We can prevent the users from doing things they shouldn\'t, such as storing data in certain places, or establishing certain network connections. On the other hand, they have the necessary freedom to adjust things such as color or font settings."
These are things that were not possible with the previous CDE desktop. Especially visually impaired users benefit from increasing the font size, or choosing color settings with higher contrasts.
To configure the KDE desktop and its Kiosk mode, the migration team worked with key KDE developer Eva Brucherseifer and her company basyskom. During several workshops, Brucherseifer helped the team to precisely configure KDE\'s Kiosk mode. She also documented how the configuration process worked, so that the migration team could use this information for future upgrades. Uffel says: "We didn\'t need a finished configuration. What we needed was to learn how to configure KDE and make it do what we wanted to do".
Uffel says that he never even considered a migration to Microsoft Windows. The environment and infrastructure has been based on Unix systems since 1995. Staff have the corresponding knowledge and are experienced in handling this type of operating system. Deploying Microsoft Windows would not only have required the IT staff to re-learn most of their tasks; it would also have required new hardware, particularly on the server side. At the time when the decision to migrate was taken, the tax authority used PowerPC servers, an architecture for which Windows is not available.
On the new GNU/Linux system, Mozilla Firefox is used as the standard web browser. Email functionality is currently provided by Lotus Domino, which will soon be replaced by Mozilla Thunderbird.
A number of specialised tax applications which had been in use on the old Solaris systems were written in COBOL or Delta Prolog. Most of these were ported to or re-programmed in Java, in order to work on the new GNU/Linux system, but some of these applications proved too difficult to port. These programs are still in active use on Solaris gateways in each tax office, and can be accessed remotely by users.
Sticking with the hardware
The migration did not have much of an effect on the hardware deployed: "We usually exchange the PCs in the offices about every two years, in order to keep up with the RAM requirements of our applications." In the fall of 2005, 6000 PCs were replaced with then-current machines (Pentium 4, 3 Ghz, 1GB RAM). These will probably remain in service until 2009 or 2010.
Currently, new hardware is being bought to replace the computers that have been in service since 2002 (Pentium 3, 1 Ghz, 512 RAM - non-expandable). Though these older machines run fine under the Suse GNU/Linux operating system, their performance is too limited for the memory-hungry Java-based applications. The new hardware will have a performance similar to those acquired in 2005.
The roll-out was conducted centrally from the Hannover IT centre. The tax authority\'s computers are equipped with a so-called BootPROM module, which makes it possible to install new operating systems over the network, remotely and unattended. IT staff can simply send the operating system image to the desktop computers in question over the network. But careful timing was necessary in order to avoid congesting the network, as huge amounts of data needed to be transferred. The migration team usually attempted to do this work during the night or over the weekend, when there was little other traffic.
Training the trainers
The desktop\'s customised application menu - © Steuerverwaltung Niedersachsen. Used by permission.
Initially, the migration team mentioned their plans in several meetings with other tax authority management staff. They also put out a number of internal announcements. As the migration got delayed for a number of months, Ute Beckmann, the leader of the tax authority\'s IT training team, recalls that there was a certain degree of unrest. The workers waited for something to happen, but nothing did. Ironically, this meant that people were quite relieved when the migration went ahead smoothly, some asking "why did everybody make such a fuss, if the new system is this easy?"
Early on during the preparatory phase, Uffel sought contact with the tax authority\'s voluntary IT trainers. This is a group of about 30 volunteers from among the tax authority\'s employees, who use 30-50% of their working time to provide basic training to users. During three or four workshops, they tested and commented on the new system. Requested changes were then implemented in another workshop with Ms Brucherseifer. According to Beckmann, the IT trainers\' feedback greatly helped in making sure that the default configuration of the desktop was easy to use for the tax authority\'s employees.
After the migration schedule was fixed, the IT trainers were instructed in the use of the new system. In the tax offices where they were based, they were then given a computer on which they could work and experiment with the software from October 2005. Again, their feedback gave the migration team the chance to remove some more errors.
Local system handlers (Uffel: "our hands and eyes on-site"), about 140 people, were instructed for 2-3 days, a few weeks ahead of the migration. They learned how the roll-out would work, which problems might occur, and how they could solve some of these locally.
The tax office at Göttingen acted as a pilot for the roll-out. According to Beckmann, it was chosen because it has a number of very active IT trainers, making it a good proving ground.
Layers of instruction: training in small groups
As the roll-out began, the IT trainers went to each tax office to train contact persons there, making use of small IT training rooms called "learning islands", which exist at nearly every tax office. These contact persons, representing about 10% of each tax office\'s staff, received an intensive training. They then trained the rest of their colleagues, and helped them during the transitional phase. All employees also received a flyer explaining the migration process.
Right after (or sometimes even during) the training, the computers of those workers which were trained were migrated to the new system. All employees also received a flyer with cookbook-style instructions for the most important operations on the new KDE desktop.
This "layered" training concept had been developed by the tax authority for some time. Beckmann recalls that when computers were introduced in the 1990s, the IT department trained all users directly; this didn\'t work out too well. During the past two migrations, she and others refined the training concept. This time, the experiences were very positive, as reflected in the results of the user suryey (below). Beckmann says that she will continue to use this training structure for future software deployments. "If this worked well, why should we do it any differently?"
One challenge was to make sure that the local management would actually make available the time for people to attend trainings. When a tax office entered the migration process, its management was obliged to offer training to all employees within three months. Beckmann says: "We handled this pretty rigorously this time. We ordered the tax offices to offer training, and to report back to the IT department who had been trained, to make sure the trainings really happened."
The users speak
According to Beckmann, who conducted an online survey among the tax authority\'s workers to evaluate their views on the migration and the new system, users\' views were "favourable, absolutely favourable".In a voluntary online survey four to eight weeks after the migration, Beckmann asked users about their views of the migration process. She also surveyed the contact persons to find out what their workload related to users\' questions about the new system was.The answer where overwhelmingly positive, with a large majority of users stating that they liked the new system and that they felt "very well" or "well" prepared for the new software.
"The response really couldn\'t be much better. Some people even called me up and said: \'This went really smoothly. Why was there so much fuss about the migration beforehand?\'", comments Beckmann. "The frictional losses we experienced were really small. We had expected it to be much worse."
The costs of a large-scale migration
Most of the work related to the migration was done within the usual range of responsibilities of the IT department. Although the migration required a substantial amount of effort, this was not related to the use of FLOSS (Free/Libre/Open Source Software); similar effort would have been required for any migration on this scale.
The IT department has ca. 180 total staff. Of these, 30-40 came into contact with the migration effort, but only Uffel\'s core team constantly worked on this task. This team consisted of 3-4 people adapting the KDE desktop and the operating system, another four working on office software, two IT staff taking care of the email client. Specialised tax applications were adapted by between three and fifteen people, depending on how deep the changes were. Three apprentices were also involved.
The IT department spent between 300.000 and 500.000 EUR on external services, mostly for adapting the distribution to certain special conditions of the tax authority\'s environment, and to simplify the roll-out process.
To fully account for the cost of the migration, Uffel also includes the time that employees spent in trainings instead of going about their work. Assuming that an average training session lasts three hours, these opportunity costs amounts to 750.000 EUR for the contact persons, and to 3.5 mio EUR for the rest of the 14.000 employees whom the contact persons trained.
No need for a service contract
For the previous Solaris systems, the tax authority used to have a service contract costing about 2 Mio German Marks (about one million Euro) for three years at the end of the 1990s; this included upgrades for new Solaris versions.
During the 1990s it was paying 300 German Marks (about 150 EUR) per seat in licencing fees. This grew to ca. 500 EUR for new versions of Solaris in the years after the turn of the century.
Today, there is no service contract with SUSE (or anyone else). Uffel says that if a problem were to arise that the IT department could not sort out by itself, they would strike up a short-term contract either with Novell or with another company with the necessary skills.
An additional advantage is that the choice of compatible hardware is greater for GNU/Linux than for Solaris, meaning that it is sometimes possible to buy cheaper equipment. There is also a much greater choice of software.
Legal issues? What legal issues?
According to Uffel, there were no legal difficulties. Initially, Uffel was unsure whether he could, as he planned to do, simply buy one set of SUSE DVDs, and then use the programs on them on thousands of machines. By now, he is positive that there is no problem in doing so. "As to using the software on all these computers, the licences clearly permit that. And by now we know that they think so as well, or they would have reacted as soon as they heard about our project", says Uffel, who is in frequent contact with Novell/SUSE.
The secret formula: Interlocking training and migration
The tax authority\'s customised startup screen © Steuerverwaltung Niedersachsen. Used by permission.
The training effort was key for the success of the migration. Beckmann is sure: "If we hadn\'t done [the training] in such an elaborate manner, things probably wouldn\'t have gone so smoothly. We have experienced that when we provide less training, people complain through every channel that\'s available to them. But this time, there was nothing, absolutely nothing." Routine certainly helps: "The effort for the organisation of the trainings shouldn\'t be underestimated. We\'ve been doing this sort of thing for some time, so we have quite a bit of experience with such roll-outs."
Making sure that everyone had the opportunity to receive training was as important as closely synchronising each worker\'s training with the migration of her computer. Thanks to the system of contact partners, everyone had someone they could ask when they ran into trouble, leaving few questions unanswered.
Small nits to pick: remaining problems
Remaining problems after the migration are minimal. They mainly occur when other public bodies send complex documents in Microsoft\'s proprietary .doc format. The StarOffice word processor that is currently in use does not handle some of these correctly. Uffel hopes to improve this with the deployment of OpenOffice 2.2 later this year, making it necessary to port the tax authority\'s macros. In the meantime, the IT department has set up a central conversion service, where employees can send problematic documents to be converted into formats that the present software can work with.
Flexible with FLOSS
According to Uffel, today\'s systems are based on FLOSS "almost exclusively". An advantage of using FLOSS is the flexibility when looking for outside support. An instance of this were the workshops with Eva Brucherseifer during the preparatory phase. "We needed someone with good knowledge of the KDE source code, and with good contacts to the developers", says Uffel. "We would then hire her for five or ten days, and develop what we needed together with her."
Uffel says that thanks to the availability of source code, IT staff can quickly locate problems in the programs, and decide whether to patch it themselves or hire an outside developer with the necessary expertise. In a large administration, the approval of funds for such purposes takes some time. This means that the IT staff often resorts to community forums to fix a problem or at least find a temporary workaround: "External experts use these forums too", remarks Uffel.
Make it an everyday thing
Uffel points out that the tax authority\'s migrations was rather specific on a technical level, as they were moving to GNU/Linux from Solaris, rather than from Microsoft Windows. But regardless of the migration path, he recommends keeping the time between training and actual deployment of the new software as short as possible. Uffel also advises to choose a "soft" migration stragey, with small groups of users being migrated one after the other, rather than replacing all the administration\'s software in one go.
Communicating the migration is also important, says Uffel. If the migration team manages to get users interested in the migration and curious about the new software, they will be much more prepared to accept it. Given the users\' positive response, Beckmann concurs: "Linux now is an everyday thing here, and nobody is looking back."
This case study is brought to you by the Open Source Observatory and Repository (OSOR), a project of the European Commission\'s IDABC project.
Author: Karsten Gerloff, UNU-MERIT
This study is based on interviews with Ute Beckmann, leader of the IT training team, and Andreas Uffel, team leader Unix front-end, both in the IT department of Lower Saxony\'s tax authority.
This case study for download
Migration XXL: PDF version
Migration XXL: ODT version
© European Communities 2007
Reproduction is authorised provided the source is acknowledged.
The opinions expressed in this study are those of the authors and do not necessarily reflect the views of the European Commission.