Hordes of creepy critters are crawling through a forest of obstacles, as teams of children cheer them on. The creatures in the BAMZOOKi show are virtual, created with the latest computer technology. Now the equipment used to record this kids\' TV show is catching up. At the BBC, an Open Source suite of hard drive recording software is preparing to replace clumsy tapes.
This case study for download
ODT [542 Kb]
PDF [649 Kb]
So many tapes.....
The British Broadcasting Corporation (BBC) is the largest broadcaster in the world, with more than 20,000 employees. It produces a large number of programs for its various television channels.
These huge amounts of material usually stored on digital tapes. When introduced decades ago, these promised higher quality and easier editing than the analogue cassettes used before. Today, commodity computers can in theory do the same job faster than the old and expensive special equipment.
A group of developers in the BBC\'s Research & Innovation (R&I) division has built a tapeless recording system called Ingex to take advantage of the cheap storage and processing power offered by commodity PCs. This software makes it possible to record a TV show entirely on normal computer hard disks. This is not only more cost-effective, but makes handling and editing the material a lot easier, too. The BBC is making the software available as Open Source under the GNU General Public License (GPL).
Recording take by take
To understand why Ingex is such a step forward, it is instructive to look at how recording at the BBC is usually done. For multi-camera productions, each scene of a show is recorded with several cameras at once, with each camera sending its data to a different tape. Usually, several repetitions of the same scene — called takes — are necessary to get it just right.
The production control room during recording
To let the production team keep on top of this mass of material, an assistant writes down the tapes\' time codes whenever a recording is started or stopped. This list is then passed along to the editor, together with the stack of tapes.
For editing, the tapes are then copied ("ingested", in the jargon) to a computer hard drive. This step is quite time-consuming: For a one-hour show, there might be three hours of recorded material from each of four cameras. As ingesting is done in real time, importing the filmed material into the computer will take twelve hours in total, before the editing work can even start. Additionally, someone needs to type the hand-written list of time codes into a computer.
John Fletcher, Senior Technologist at BBC New Media & Technology, says that Ingex started out from the idea "to provide a PC-based means of logging the timecodes of the start and end of a take". A list of timecodes would be generated that made it much easier to bring the recorded material from tape into post-production. After some initial experiments in 2003 with recording broadcast-quality video on a PC, the group tried out the electronic logging of time codes in 2004, along with automating the ingest from tape.
Moving on from there, "we thought we could link this up to an electronic recording system", doing away with tapes altogether, says Fletcher. Today, Ingex comprises a whole suite of software applications for tapeless production, from recording up to getting the material ready for editing.
On the set, the cameras and microphones feed their data directly into a PC running Ingex. The software records this footage. The timecodes for the beginning and end of each take are automatically logged and written into the metadata of the recorded files. Instead of writing these codes down by hand, the production assistant only needs to press a button to mark each take as "good" or "no good". She can also enter comments for each scene. Ingex also contains a player for media files that lets the production team review its work.
Getting rid of tapes and paper
In 2005, the group first used a prototype combination of tapeless recording and electronic logging to record a series of the BAMZOOKi childrens\' show. The process is documented in a video (8 min, .avi) on the BBC\'s R&I website. 15 of the 20 episodes were broadcast using recordings made on Ingex. At that time, hardware limitations meant that Ingex created uncompressed video files, from which the compressed video files were generated by processing overnight.
In ongoing trials that started in late 2006 on the set of the popular soap opera EastEnders, the team explores networked recorder/server operation, and tries to understand how a tapeless workflow can be made to fit within the established TV production process. Advances in processing power since 2005 mean that video can now be compressed in real-time during recording.
In June 2007, Ingex was used to record the Dragons\' Den show over four weeks. Seven cameras, feeding two Ingex recorders running as a pair, were used to record the series over four weeks. Here, the time savings that are possible with Ingex became apparent: Had the show been recorded on tape in the conventional manner, then the material would have had to be transferred to computer disks for editing. This would have taken between 30 and 35 hours for each day of filming, says David Kirby, Lead Research Engineer at BBC R&I, who is in charge of the Ingex project.
Fourty Betacam tapes are needed for the data that fits on a 1TB hard drive
In August 2007, Ingex was used to record a concert by the band Foo Fighters for broadcast on the BBC Interactive service. Since the concert had to be broadcast soon after it was recorded, the time savings provided by Ingex were very important. The recorded material was put on a USB drive, where it was immediately available for editing.
A further advantage of recording to disk is that once the material is available as a digital file, it can be used in a variety of ways. "You can put it on an iPod, or upload it to a web page — you can do so much more than with tape recordings", says John Fletcher.
Ingex as a research project
Ingex is developed by the BBC\'s Research and Innovation (R&I) division. The BBC sees Ingex as a research project, using it to investigate how tapeless production works, what can be done with low-cost equipment, and how this technology affects the production workflow. David Kirby says that the roll-out of tapeless production would be handled by another BBC project, the Digital Media Initiative. "We also give production teams a chance to try that technology and see the possible benefits, but also the pitfalls." Whether Ingex will be used on a larger scale across the BBC has yet to be decided.
"It\'s really a research project that\'s being used for trials in the studio. We\'re approaching it from the research point of view. We don\'t want to get too overloaded being involved with production teams." Kirby says that productions which have used Ingex are very keen on using the system for their next recordings as well, but the group prefers to focus on research and development: "We\'re just holding back a little bit."
Open Source and Open Standards
One goal of Ingex is to see what can be done with low-cost, commodity PCs. "On things like processing time and bandwidth to disk, we\'re really pushing the limits of what you can do with current hardware", says Fletcher. Kirby adds: "The improvements over the past two years have been significant. What were doing two years ago, when it was borderline, we can now much more easily do with off-the-shelf hardware."
Ingex encodes the video input into a variety of formats for use in post-production. Especially for high-definition material, this demands large amounts of processing power. "To optimise the software so this can be done in real time has been quite challenging", says Fletcher.
Ingex uses numerous programs that are available under Open Source licenses. This makes the development significantly easier than it would be starting from zero: "We\'ve put one tenth of the work in that we would otherwise have needed to get the whole package working", says Kirby. The recording section of Ingex runs on computers using the GNU/Linux operating system. The R&I team has developed the video capturing from scratch, along with the necessary control system, and written some glue code to make the various packages work together. Ingex uses Open Standards throughout. The Material Exchange Format (MXF) file format, which the BBC\'s research department helped to develop, is in use along with the Advanced Authoring Format (AAF), "which we also had a big part in helping to standardise", says Kirby. "The intention is that everything around Ingex is open: Open Source and Open Standards."
Ingex runs best with two Intel Quad-Cor
The nuts and bolts of Ingex
e processors, 4 GB of RAM and 4TB of hard drive capacity. The hard drives are formatted with the XFS file system, which was found to have the best performance for storing high bit-rate video to the disk.
The computers are equipped with an SDI (Serial Digital Interface) card. SDI is a digital interface used to transmit broadcast-grade video.
The prototype Ingex system, put together from off-the-shelf components
Ingex encodes data with ffmpeg and writes to MXF files. The BBC contracted an Open Source developer to build a DVCPro-HD codec for ffmpeg. According to Stuart Cunningham, one of the team\'s software developers quoted in a computerworld.uk article on the system, this resulted in better performance than most hardware-based decoders can deliver.
The recorded data is either stored on USB drives, or on a NAS server, The files are then sent handed over to post-production, where they are edited on Avid workstations.
Making it work outside the lab
An interesting feature of Ingex is that it is able to copy material off tapes from an archive and store it on hard drives. Like other broadcasters, the BBC has huge archives of old shows on tape, which it wants to make available digitally.
To move out of the research lab, the system also needs to be as rugged as possible to stand up to the conditions of working in a studio. Kirby explains: "If you\'re recording feeds from five cameras, and one of the cameramen momentarily disconnects his camera to get a better shot, you can\'t have the system giving you error messages. It just has to carry on recording."
"It\'s those aspects that take if from just capturing video in the lab to making it rugged and reliable for use in the studio", says Kirby. "That\'s another challenge for us." To guard against hard drive failures, the next version of Ingex will also feature RAID5 storage.
Taking Ingex for a spin
"There\'s generally recording on tape in parallel", adds Fletcher. "We try to have a minimal impact on their production process." The tape serves as a backup if anything goes wrong with the Ingex system. Kirby says that problems at a trial last summer would have resulted in some data loss, due to problems with the capture card software. But he points out that such mishaps occur with the tape-based production system as well. "A normal production is not immune to problems anyway."
Lights, cameras, action: Set for the Dragons\' Den show
Saving costs with generic hardware
According to the BBC website, the hardware required for an Ingex recorder capable of recording four SDI inputs each with four audio channels and including 80 track-hours of storage currently costs € 10 000. A high-definition (HD) version can be configured by using HD capable SDI capture cards. Since the amount of data per second is greater due to the higher resolution, each PC only records the feeds from two cameras instead of four, using two HD-capable SDI cards. The BBC estimates that the cost of such a system would run to € 15,000.
A server running the MediaHarmony software, which lets several editors access the recorded material at the same time, equipped with 20 Terabytes of storage (which is sufficient for about 800 hours of content), currently also costs in the region of € 15 000. The BBC makes the required software for both systems available as Open Source.
There are proprietary hardware/software combinations that allow live recording. "But I don\'t think any of them offers all of the capabilities that we have in Ingex in one unit at a reasonable price", says Kirby. "If you had a large budget to spend, you could achieve what we\'re doing with Ingex. But Ingex has the advantage of tying it all together in one unit." Commissioning such a system from a manufacturer would probably lead to vendor lock-in: "They\'d obviously make up a system that works with their components, but obviously not with other peoples\'", adds Fletcher. "The flexibility of Ingex is not something you can get off the shelf."
Almost all of the development work on Ingex has been done in-house by the R&I team. The only parts that were contracted out were the developments of the DVCPro-HD and SMPTE VC3 (DNxHD) codecs for ffmpeg, which according to Kirby cost "a few tens of thousands of Euro".
According to Kirby, it is too early to say how much cost Ingex saves exactly. He says that based on the experience from the trials, Ingex will lead to a noticeable reduction in the overall costs of TV production.
Four to six people have been involved in developing Ingex. The cost of the development itself is hard to separate from other work in the BBC\'s R&I department. This includes the effort that resulted in the international standardisation of the MXF and AAF formats.
Good reasons for going Open Source
Why did the group choose to distribute the system as Open Source? David Kirby: "Considering the amount of software that was out there already that we wanted to use, that certainly would sway the decision into making the whole project available as Open Source."
Other R&D projects at the BBC such as Dirac have used Open Source licenses before. "We were quite happy to make use of Open Source software, and have been using it beforehand anyway. So making our own software Open Source was a continuation of that", says Fletcher.
Support is another consideration, says Fletcher. If Ingex comes into more widespread use, then people will need help in deploying and using the system. The research group doesn\'t have the capacity to provide such assistance. "Bearing that in mind, it seemed best to make the whole package available as Open Source. If somebody else wants to use it, they can make their own arrangements for support. Our users inside the BBC could contract support from outside if they wished." This strategy also fits the BBC\'s public status, which bars it from favouring any particular manufacturer.
The software and its source code are freely available, and the system uses off-the-shelf hardware. But setup and use are far from trivial. "You might typically seek some technology partner company to deal with the technical side of it, and then provide that service to you as a broadcaster", says Fletcher. According to him, a few companies have expressed their interest in doing just that, which Fletcher thinks is good news: "We\'re a research group, and we don\'t really want to get into regular day-to-day support of the system. If it were going to be widely used in the BBC, then we\'d want to find somebody else who can support the system."
Review and editing: faster and more efficient with tapeless production
Another reason for going Open Source is to seek input from other users, who can try out the software and report on their experiences. According to Fletcher, the BBC would be happy to work together with others to develop Ingex further. "Part of the reason for making it an Open Source project is to invite cooperation from others", he says. On the other hand, "we didn\'t seek to involve anybody else in the early stages". He says that outside interest has been limited so far. Some broadcasters might simply not know about Ingex, while others prefer to work on their own solutions. "You\'ve got to be fairly technically aware to download our software and start using it."
An outside view of Ingex
At UBF, a Dutch company which supplies equipment and operators for audio/video recording and post-production, Technical Project Manager Rob Verhoef has been experimenting with Ingex since early 2007. "A colleague of mine came across it on the Internet", he recalls. Then, "it was fairly basic". The software could record a maximum of four channels at the same time. Also, it only used a limited number of file formats for the recorded material, none of which Verhoef found really useful. "It turned out to be quite an experimental thing", Verhoef says. "It was difficult to set up and run."
Verhoef and his colleagues experimented with Ingex, and got in touch with John Fletcher at the BBC, who helped them to get the system to run. But given the limitations of that version of Ingex and the technical difficulties, UBF put its experiments on hold after half a year.
Verhoef started looking into Ingex again when the BBC\'s R&I team released a substantial update to the system in late 2007 . Ingex now supports more file types and compression formats, but "it\'s still some hassle setting it up", says Verhoef. As UBF mostly does recordings in the field rather than in the studio, stability and ease of use are most important to him.
Potential under development
Like the BBC team, Verhoef sees Ingex as more of a research project than a system for real-life use, at least at present. But he agrees that there is currently no affordable proprietary setup to do the things that Ingex does.
Verhoef sees the main advantage of Ingex in the fact that the files can go directly into the post-production, without having to be transferred from tape to disk again. Another big plus is the system\'s ability to deal with several channels at once. This makes it possible to record several cameras simultaneously on the same device, rather than having to use a recorder for each camera.
Controlling and monitoring the studio recordings. The Ingex screen is on the left.
Cost is also an advantage of Ingex, Verhoef says; there is no licence fee for the software, and the necessary commodity PCs are cheap compared to the specialised hardware that is normally used in TV production. But he points out that personnel costs are higher: "You have to have a very good IT person around the production", whose time costs more than that of the usual recording staff. Tapes and disk recorders, on the other hand, are so simple to operate that it is not necessary to have specialised technicians around all the time. This reduces Ingex\' cost advantages.
Another big plus is that the source code for Ingex is available. "This way, we can build our own solutions." Verhoef says that he is currently working on replacing one component of the system, a recording client, with a program of his own, in order to add some features. As Ingex uses Open Standards throughout, "it\'s a fairly easy task" to get the new component to work with the existing ones.
Verhoef is also looking into using Ingex simply to digitise existing tapes. "There\'s a lot of material already on tapes that needs to be digitised. That\'s really a business for us."
Make it easier to use
Verhoef says that the main challenge for Ingex is to become easier to install and use. "Getting it to run takes about three or four hours", says Verhoef. He hopes that in the coming years, the system will become "more operator-friendly". For example, some of the components currently do not have graphical user interfaces, relying on the command line instead.
Currently, Verhoef finds Ingex too complicated for use in the field, especially compared to the tapes that are currently in use. "Tapes are really simple. Everybody knows how to use them", he says. "When we\'re recording in the field, everybody is very busy with getting the cameras, the sound and the rest of the equipment working. They can\'t worry about the recording itself too. It just has to be a button that says \'Record\'".
"It\'s also new to the technicians", says Verhoef. They are used to working with tapes. "There, you can quickly rewind the tape and see what you recorded five minutes ago. With a [system like Ingex], they don\'t know how to do that so easily."
Working with the system means that various things have to be done differently from a conventional tape-based production. "You have to build an efficient workflow around Ingex", says Verhoef. The people involved need to learn how Ingex works. In comparison, "tape is simple".
"Contributing is difficult"
Verhoef is developing a program that makes setting up Ingex in the field easier. This currently requires a series of steps, which he hopes to automate. He has also published installation instructions for the software, which he says go beyond the documentation provided by the BBC. In addition, he says that he has made a number of suggestions to the development team. But he has found it difficult to contribute directly to Ingex: "You have to realise that though the project is Open Source, the development is not set up in an Open Source way." Indeed, the Ingex team is currently not actively aiming at building a community around the software suite. All the work is done by the team at BBC R&I division. He also says that the software availabe on the project\'s website is not always the latest version, as the BBC is continuously working on the code.
Another hurdle, as Verhoef points out, is that any contributor would have to spend a lot of time familiarising herself with the complex source code.
Despite the difficulties, Verhoef is very interested in Ingex. Though UBF is currently not using the system in production, he says he will continue to experiment with it: "It\'s really an R&D thing for us."
Close to the users
Looking back over the past years, Kirby thinks that the development has gone quite well: "We might think of other solutions now, but just in the light of the experience of doing it." The software developers who contribute to Ingex are familiar with handling media as well as the GNU/Linux operating system.
Kirby thinks that producers have agreed to work with Ingex during trials because the team has offered good and reliable support. "It\'s a different way of working from having tape machines. We need people who are familiar with how a studio operates, but also have a good knowledge of IT systems." Usually, there is a person on the set during trials who can administer GNU/Linux systems and is familiar with the way Ingex operates.
Editing the childrens programme Bamzooki from recordings made with Ingex
Kirby adds that an important success factor for Ingex is that it is developed with a lot of feedback from its users. "Understanding what users need is important. Having our colleagues who are involved in production work showing interest, coming along and talking to us, giving us some direct input so that we know how to develop the software to meet their needs — that\'s been crucial." The BBC has an advantage here, he says, "because we have people that we can phone up and talk to about these problems."
"From the outset we have been working closely with the end users [within the BBC]. That has been very valuable", agrees Fletcher.
Next steps: consolidate, automate, document
Though products with capabilities similar to Ingex might appear on the market in the near future, the mere fact that an Open Source alternative exists might lead to lower prices on the proprietary offers.
The next step, Kirby says, is to consolidate the system into a full release, making it easier to install and improving the documentation "so that others can use what we\'re using in the studios, without having to email us and ask questions about it."
The current work is aimed at making Ingex more robust. Next comes the addition of new features exploiting the fact that the material is available in digital files, such as automation of parts of the editing process: "There are various things we\'d like to explore once we\'ve got this testbed working reliably."
More work will go into automating the workflow, from capture and recording to post-production. "The longer term work is bringing automation to the process, to improve efficiency", Kirby explains.
Vision for the future: exploiting the digital advantage
At some point in the future, Ingex should be able to produce a rough edit of the program automatically. This would work by structuring the recorded material along the lines of the script of the production in question.
The content should also be made available in different ways. During recordings, instead sending each camera\'s feed to a monitor, Ingex could make the content available on a network in real time. This would allow anyone on the network to view the content using a PC, eliminating the need to run large numbers of cables around the set. It would also allow the producer to monitor the recording without actually being on the set, instead of receiving it on video tape a couple of days later.
"We\'re experimenting with copying the content onto a portable device", says Kirby. "The director or producer could go away from the studio with a lower quality version on the device, and maybe then review the material on the train home."
"It\'s all about making the material more accessible to everyone, and automating the process as far as possible", says Kirby.
Links
Ingex on sourceforge
BBC video explaining Ingex
BBC tapeless production website
Other BBC R&I projects
Dirac
Computerworld article on Ingex
The British Broadcasting Corporation (BBC)
The BAMZOOKi childrens\' show
EastEnders
Dragons\' Den
The Material Exchange Format (MFX) file format
United Broadcasting Facilities (UBF)
Ingex installation instructions by Ron Verhoef
The Advanced Authoring Format (AAF)
The XFS file system
The ffmpeg encoder/decoder
DVCPro-HD standard
Network-attached Storage (NAS)
Avid
RAID5
Serial Digital Interface (SDI)
This study is based on interviews with John Fletcher, Senior Technologist at BBC New Media & Technology; David Kirby, Lead Research Engineer at BBC Research and Innovation, who has the overall responsibility for the Ingex project; and Rob Verhoef, Technical Project Manager with United Broadcasting Facilities (UBF) in the Netherlands.
Copyright
© European Communities 2007
Reproduction is authorised provided the source is acknowledged.
Disclaimer
