How two volunteers built the Raspberry Pi’s operating system

When you buy a Raspberry Pi, the $35 computer doesn't come with an operating system. Loading your operating system of choice onto an SD card and then booting the Pi turns out to be pretty easy. But where do Pi-compatible operating systems come from?

With the Raspberry Pi having just turned one year old, we decided to find out how Raspbian—the officially recommended Pi operating system—came into being. The project required 60-hour work weeks, a home-built cluster of ARM computers, and the rebuilding of 19,000 Linux software packages. And it was all accomplished by two volunteers.

Like the Raspberry Pi itself, an unexpected success story

Although there are numerous operating systems for the Pi, the Raspberry Pi Foundation recommends one for the general populace. When the Pi was born a year ago, the recommended operating system was a version of Red Hat's Fedora tailored to the computer's ARM processor. But within a few months, Fedora fell out of favor on the Pi and was replaced by Raspbian. It's a version of Debian painstakingly rebuilt for the Raspberry Pi by two volunteers named Mike Thompson and Peter Green.

It began because Thompson was ready for the next big thing. Thompson had been CTO and co-founder of Atomz, a hosted search service acquired by WebSideStory in 2005. Thompson got a "not unhealthy slice" of the $45 million or so that company sold for, and he eventually stopped working to spend a few years hanging out with his wife and kids.

A year ago, he was finally ready to get back into the technology field. Robotics is one of Thompson's primary interests, and the Pi looked like a great platform for that. It's cheap and small enough to be easily embedded into a variety of systems. But there wasn't an operating system fully optimized for the Pi's floating point unit, which is important in robotics projects and various other types of math-intensive applications.

"When I first learned of the Raspberry Pi, I was disappointed that none of the Linux distributions available for it would make use of the very fast floating point hardware that was present on the Pi," Thompson told me. "As a long-time Debian user, I was like, 'I'd rather see Debian [than Fedora]' and I wanted to see the floating point enabled because I have this longer-term interest in getting robotics working with these inexpensive boxes."

Debian had added floating point support for the ARMv7 processor, but not the ARMv6 processor used in the Pi. Debian "didn't see a product like the Raspberry Pi coming on the horizon. Even though ARMv6 in Pi has a pretty capable floating point unit, they didn't support it," Thompson said. Thus, "all the thousands or tens of thousands of software packages they built wouldn't support the Raspberry Pi."

Just as a graphics processing unit handles graphics operations very quickly, a "floating point unit performs all the math very quickly," Thompson said. "It's a peripheral that not every computer has, but when it does you really want to take advantage of it." The floating point unit is part of the PI's Broadcom BCM2835 system-on-a-chip.

If you don't take advantage of floating point capability in the hardware, a lot of mathematical operations must be performed in software, lengthening the amount of time it takes for the Pi to do useful work. This is important for robots, where complicated math operations are used for processing data from cameras and sensors, and for controlling motors with precision and speed, Thompson said. It's also important for multimedia processing, encoding music, physics simulations, or "pretty much anything that is numerically intensive."

Raspbian project founder Mike Thompson.

A fruitful partnership

The path forward was clear to Thompson: rebuild Debian to run on the Raspberry Pi. This required shifting 19,000 packages from Debian to Raspbian—a monumental task.

Thompson didn't go it alone, though. He started a Raspberry Pi forum thread to talk to other people interested in bringing Debian to the Raspberry Pi. The thread caught the interest of Peter Green, a Debian developer and PhD student in the UK who went by the handle "plugwash" in the forums.

Green was a rare mix. He not only had the expertise to co-lead the project with Thompson, but he was also crazy enough to do it.

"I felt I was the only one of the people who were talking in the threads about the project that became Raspbian with enough Debian knowledge to make the rebuild a success," Green told me. "I'm sure there are other people within the Debian project who could have done it if they were interested and crazy enough, and there were many people in the Debian project who provided us with bits of help along the way."

When Thompson and Green got started, the Pi wasn't actually available yet. Even if it had been, it wouldn't be capable of rebuilding Debian in a reasonable amount of time. Thus, Thompson set up a cluster of eight Freescale iMX53 Quick Start Boards, each with 1GB of memory, a 1GHz ARMv7 processor, and (most importantly) SATA hard drives. One of the main reasons the Pi would be unsuitable for this type of work is the bottleneck introduced by USB storage, Thompson said. The Freescale boards were able to build Raspbian anywhere from four to 12 times faster than the Raspberry Pi would have.

Thompson spent nearly $3,000 on the cluster but recouped those costs through donations to the project. In addition to the single-board ARM systems, there is a separate Linux PC to serve as the repository for Raspbian builds. The repository server retrieves source packages from Debian repositories, schedules jobs for the Freescale systems, and collects the binary packages once the builds are complete. "The built packages are all staged in the repository and then synced with the external repository we maintain at www.raspbian.org where users pull their packages from," Thompson explained.

At first, Thompson used an ARM-based HP Media Vault MV5150 for the repository, but later upgraded to an Intel-based system when more horsepower was needed. Although each Freescale board has its own hard drive used for building packages, the main storage duties are handled by the repository server's 500GB drive, which is about two-thirds full now. Here's what the cluster looked like when he was first setting it up:

Five Freescale systems and the ARM-based HP server.

Credit: Mike Thompson

Credit: Mike Thompson

The cluster would eventually include eight single-board ARM computers.

Credit: Mike Thompson

Thompson and Green weren't starting from scratch. Debian is already one of the most widely used Linux-based operating systems, obviously. Its ARMv7 port provided a solid foundation.

"We greatly leveraged the work done previously by the Debian Project to support floating point hardware on ARMv7 devices," Thompson said. "Other than the effort to actually build [19,000] software packages, 95 percent of the work done to port the software to support the Raspberry Pi was already done by Debian."

That's not to say the Thompson/Green work in bringing Debian to ARMv6 and its floating point unit was trivial, however. Green explained:

In Debian the compilers have default settings built into them. These default settings set things like the CPU family, the minimum CPU requirements, and the ABI [application binary interface] in use. Most packages leave those settings alone. We modified the compiler packages to reduce the default settings to ARMv6

For the majority of source packages rebuilding them with a compiler that uses our new defaults is enough to make them build ARMv6 code.

A lot of the original work was done manually, but Green eventually created auto-builder software to automate much of the process. Those auto-builders (which run in a chroot environment on top of Debian) are still running to this day, pulling updated packages from Debian repositories and automatically re-compiling them for Raspbian.

Those auto-builders also detect when there's a problem that prevents the package from being automatically rebuilt. Green explained in an e-mail:

There are several things that could cause a package to come out still containing ARMv7 code.

We hacked together a script (using readelf) to check for ARMv7 code in packages so we could tell which packages... needed further attention to eliminate it. ARMv7 contamination can come from several sources.

Static libraries [are one example]: these are taken from the build environment and incorporated into binaries. So if the static library contains ARMv7 code then it will go into the resulting binary. This was a big pain during the early days of the project as we had to use packages from Debian armhf [Debian's ARM floating point port] to break dependency cycles, but those packages could contain static libraries with ARMv7 code in and sometimes it took several attempts at rebuilding and installing different packages to figure out which library the contamination was coming from, and replace it with an uncontaminated version. However, this isn't really a problem anymore since having got past the bootstrapping problem we can nearly always build packages in a clean Raspbian environment.

Raspbian became available for download on April 20, 2012 in limited form, containing only about five percent of the Debian packages. "Enough to build a root filesystem that would boot to a command line," Thompson said.

Rebuilding 19,000 packages

Debian ARM included about 36,000 packages that had to be ported over. Luckily, 17,000 of those did not contain any executable code and could be moved into Raspbian without delay. Those included "non-executable items such as help files, man pages, package documentation, fonts, etc." Thompson said.

The other 19,000 that did include executable code formed the bulk of the work, and it took until early June to rebuild all those packages.

For Thompson, it was essentially a full-time job for three months. "Some weeks I put 60 or 80 hours into it," he said.

Explaining some of the early challenges, Thompson said:

The initial problem was that Raspberry Pi hardware wasn't available until several months after Raspbian was started. However, some folks at the Raspberry Pi foundation helped with testing binaries that we were producing on the few beta units that they had which helped verify that all the software we were building would indeed work on the Raspberry Pi.

The second major problem was the initial steep learning curve required by me to learn how to build a Linux distribution. Fortunately, the Linux community in general, and the Debian community in particular, is very friendly and helpful, so advice on how I should do things was generally available when I ran into walls. Once Peter joined the efforts, the ball really started to get rolling and with a cluster of eight ARM build servers, we were able to build nearly a thousand packages a day.

Some were slower than others. Green noted that WebKit, which powers the Midori browser in Raspbian, "spends days in swap hell" but completes eventually.

Most packages make it through the process just fine, though. A command shell like bash is one of many "examples of a package that we would expect to port without any problem," Thompson said. "We just download it, rebuild it, and send it to the repository."

Thompson estimates that about five percent of Debian packages don't successfully make it through the auto-building process. This means he and Green had to manually alter any code targeted specifically for ARMv7 and rebuild it so it would work with ARMv6.

Compilers, Java interpreters, "anything that is sensitive to the architecture that it's running under... may give us that type of problem," he said. "In most cases we can make a fix to the package and get it up and send it on its way for people to use, but there are a handful of packages that aren't going to be available for the Raspberry Pi, for certain reasons."

Only about 50 to 100 packages wouldn't port at all. "Those were packages like debuggers and things specific to the ARMv7 processor, and a handful of them didn't even make sense to ever run on the Raspberry Pi," Thompson said.

Raspbian can be run with or without a graphical user interface. Since the Pi is targeted toward teaching programming, the default interface includes a Python development environment.

Raspbian's default graphical user interface.

Credit: Adafruit

Raspbian supplants Fedora—and will keep getting better

The end result: a version of Debian for the Raspberry Pi that's both friendly to beginners and familiar to experienced users of Debian, which has spurred mass amounts of experimentation (See: "10 Raspberry Pi creations that show how amazing the tiny PC can be"). "That's what Raspbian was about, rebuilding the Debian infrastructure completely so that a person who's familiar with Debian could just pick up the Raspberry Pi and download completely optimized binaries," Thompson said.

As mentioned earlier, an ARM-based remix of Fedora was originally the preferred operating system for the Pi. Fedora, however, was constrained by the amount of memory available on the system, especially when the Pi launched with just 256MB of RAM (a newer version has 512MB).

Eben Upton, the UK man who created the Raspberry Pi and leads the Raspberry Pi Foundation, told me that the organization switched to recommending Debian around May 2012, and then to "Raspbian when it became available (and we'd had a chance to build images from it) in August."

A smaller memory footprint, ease of use, and support for floating point hardware are the main reasons Raspbian got the nod. Fedora for the Pi is still under development, but the only operating systems on the foundation's download page today are Raspbian, Arch Linux ARM, and RISC OS.

"That was really gratifying to see that happen even though that was not my intention when I started off," Thompson said of Raspbian becoming the recommended OS. "It was just self-interest, something I wanted to learn and can now say I've done."

The work isn't quite over, though, at least for Green. The cluster is still running in Thompson's house, with Green logging in from afar to keep Raspbian up to date. Thompson is now working for a small robotics consulting company named OLogic, and he has ceded most of the project's operations to Green. "I've been practically running the project these past few months," Green said.

Raspbian project co-leader Peter Green.

Right now there's some manual work every few days in making changes to packages that don't automatically port over from Debian to Raspbian. The work will pick up again at some point, however, because Green doesn't intend to let Raspbian get out of date.

Raspbian is based on Debian's "Wheezy" build, the current test build. Wheezy was chosen because of its hardware floating point support, despite not yet being the stable build of Debian. Wheezy will eventually become the stable version of Debian, and the new test build is to be named Jessie (named after characters from Toy Story.)

The Raspberry Pi itself will not be getting a major hardware update any time soon. Upton said the Foundation and volunteers are working on further optimizing Linux for the Pi's ARMv6 hardware. "We think there's a lot of low-hanging fruit on the chip that we've got," Upton said.

Thus, Green would like to bring Debian Jessie to the Pi when possible. "The current model Pi is not going to be superseded any time soon, so people are going to want the next version of Debian for it," Green said. "I'm going to need to look at how I automate stuff to the point that we can deliver that without me driving myself insane."

Green is also looking to move on from the build cluster in Thompson's house.

"Right now Raspbian relies on a build cluster in Mike's basement, and also the auto-builder software we've got is very hacked together due to mainly a lack of documentation on Debian auto-building infrastructure at the time [it was built]," Green said. "One thing we want to do for Raspbian Jessie is move to a new auto-builder cluster with more powerful auto-builder hardware, up-to-date versions of the Debian auto-building software, and hosted in a proper hosting facility rather than Mike's basement."