By the Spring of this year, about 20 million Linux-based mobile phone handsets had been delivered to the market worldwide. This striking achievement occurred despite many experts all around constantly dismissing the idea that embedded Linux could play such a role in the market. “Linux is too big, it’s too slow and it certainly isn’t real-time, so it can’t possibly be a realistic solution,” they all said. But the fact is, through the efforts of the open source Linux community and commercial Linux suppliers such as MontaVista, Linux took the market by storm.

This should be welcome news to developers working in the appliance market, even though a handset and a typical appliance, at the surface don’t look to have much in common. But a closer look beneath the surface reveals a lot of common requirements and the challenges of meeting them. The benefits of embedded Linux have become relevant quite quickly to the appliance application area, and are no longer restricted to the more traditional or obvious areas of use of Linux in communications or industrial process equipment. There are many good reasons why product design engineers should consider learning about embedded Linux and its impact on system design and development.

The software explosion

Enlarge this picture Fig. 1. The software content of devices is growing rapidly.

Fig. 1 illustrates the heart of the challenge facing appliance developers. The software content of a device in today’s interconnected world grows not at a linear rate, which might be challenging enough, but at an accelerated rate. For example, the first generation of a new appliance might be very dedicated or have a static function. The device would have a simple, non-graphical user interface, be isolated and not connect to any network. But marketing departments and the competition never sleep, and the next generation calls for a browser with some connection to a network. Virtually all of this functionality is derived from the application software content of the device. And the story continues. Java is added because of its ability to securely support new applications downloaded to the device.

To further enhance the user experience, multimedia capability gets added, perhaps voice recognition and other advanced features that make for a compelling user experience with the device. This progression means that a company previously capable of developing simple microcontroller software with a small team of one or two engineers is now is faced with developing software content that rivals what came with a PC only a few years ago.

But behind the reality of ever increasing amounts of software content required to keep competitive, the complexity of the software itself, as well as the complexity and cost of managing the growing size of the software development team, present real and daunting challenges.  And the development of highly complex software can’t be accomplished by a two-person team anymore. Small team software development approaches that had been successful in the past simply won’t scale to meet the challenges of the new, interconnected device world. The appliance world is headed for a software crisis.

The software crisis

Enlarge this picture Fig. 2. Most software defects are discovered after the prototype phase.

Many industries have faced (or will face) the explosion of software content in their products. The term “software crisis” is often used to describe this situation. The good news for appliance developers is that they can learn from the successes of those industries that hit the software crisis before them. For example, as far back as the early 1980s, the U.S. Department of Defense recognized that they could no longer afford the high cost of custom software development for their major programs.

They initiated numerous efforts including the development of a standard programming language, Ada, as part of a comprehensive program to lower software development and maintenance costs, as well as increase reliability. Interestingly enough, the typical size of the software content of systems that started the DOD on its attack on the software development crisis was pretty much what is illustrated in Fig. 1. The software content of an F16 then was about the same as today’s set-top box or high-end handset.

There are a couple of constant themes found when looking at how the DOD and other industries adapted to the rapid expansion of software content: the development of standards and a move to commercial off the shelf (COTS) components.

Three key software development elements are essential to effective standardization:  programming languages, operating systems, and software development environments. As noted previously, the DOD went so far as to specify an entirely new programming language to help solve its problems. Similarly, the embedded software industry effectively standardized on a few existing programming languages, mainly C, C++ and Java, as a means of lowering cost and speeding development. However, in both the operating system and development environment areas, the overall embedded world has, until recently, not managed to find a way to gain the benefits of standardization. But that situation has changed dramatically and quickly from unexpected developments in the software world.

Open source

A new process for development software has emerged over the last 20 years, sometimes called the Free Software approach, other times called Open Source. But no matter which name is used, the process itself is clear. Teams of like-minded developers, sometimes individuals acting on their own, more often now developers on the payroll of commercial companies, join together and develop software governed by a very special license.

Perhaps the best known, but by no means the only such license, is the general public license “gpl” from the Free Software Foundation. The gpl and other similar licenses provide the basis that govern the development and distribution of software licensed under the gpl. Although the gpl is quite commonly misunderstood to require that the software be “economically” free, its real intent is to ensure that recipients of software based on the gpl receive all the rights under the gpl, including the rights to the source code underlying the software received. It is perfectly acceptable and within the license of the gpl to charge as much as the market will bear for the software.

Under this Open Source development process, a large body of software has emerged that is directly relevant to appliance developers. For example, the most widely used compiler and associated tool chains used in the embedded world are gcc and g++, which are not only developed under the gpl, but the development itself is also the major software output of the FSF. It turns out that a “perfect storm” has developed, fueled by the Open Source process, that is fundamentally changing for the better the way in which embedded software can be developed. This perfect storm involves both the operating system for embedded devices as well as the associated development environment.

Embedded Linux emerges

But by far the most well known example of the success and power of this approach is the Linux operating system. To be fair, Linux relies heavily upon critical software components developed by the FSF, and there are those who argue that the correct appellation for Linux is “gnu-Linux” where gnu represents the FSF software content.

Linux is a large-scale operating system and is closely related to Unix, an operating system widely used in workstations and servers. It should be no surprise, therefore, that the initial success of Linux was on server platforms, and moved into the enterprise computing area as well. However, early after the emergence of Linux, efforts began to adapt it to the embedded computing market.

At first the efforts were made by individuals, but by the later 1990s, several commercial companies emerged with the goal of transforming Linux into a suitable embedded operating system, and subsequently provided the industry the standard platform that had eluded development for many years. Linux has the sophistication to support the advanced software content required of today’s modern connected devices. Three observations serve as evidence:

• Products have increasingly complex system software requirements, including Internet connectivity, a need to use the most competitive leading-edge microprocessor technology, and a need to support rapidly evolving and extremely complex I/O technologies. Linux has well-established “gold standard” networking capabilities, and is often the first (and sometimes only) operating system ported to new silicon with a wide selection of device drivers to support the complex on-chip I/O.
  
   
• The royalty component of current off-the-shelf system software in the cost of goods is large and impacts corporate margins. Linux is typically sold in a royalty-free fashion.
  
  
• The selection of a common, strategic system software platform will help a company avoid getting mired in a multiplicity of solutions, each with a high cost structure and no overall leverage. Industry experience shows that disparate platforms inflate costs and increase the length of product development cycles. Linux is widely available across virtually all microprocessor architectures, allowing companies to use Linux as a standard OS platform.

All of these factors have combined to make a Linux-based OS an effective solution to help solve the growing software crisis facing device makers. That there’s no real debate about this is evident from the millions of electronic devices, such as mobile phones, set top boxes, HDTV and more, that have already shipped with Linux as their OS.

Emergence of Eclipse

The other essential piece that serves as a foundation for helping solve the software crisis is a standard development environment. As with operating systems, the embedded market was littered with incompatible software development environments that drove up software development costs. Engineers had to be trained on the nuances of each environment, learning a new editor or new debugger interface as they moved from project to project. Commercial suppliers of development environments in turn had to not only develop specific tool capabilities of value to a developer, but also the basic infrastructure to support sophisticated development environments. Consequently, a lot of development effort went in to duplicating these low-level capabilities among the suppliers, leaving less money for truly innovative features.

Around 2001, under a Open Source process and license (although not based on the gpl), IBM single-handedly altered the development landscape of not only embedded developers, but mainstream software developers, as well, when it announced the availability of an open-source-based integrated development framework, named Eclipse.

Eclipse is designed to provide the basic infrastructure required for a highly capable and extensible integrated software development environment. In short order, hundreds of software tools suppliers adapted or developed software tools that would use the Eclipse framework as their foundation.

Every major embedded software tools supplier has made their environments available with the Eclipse. MontaVista, for example, supplies a comprehensive suite of development tools that “plug-in” to Eclipse for memory utilization analysis, memory leak detection, performance monitoring and event tracing. Thus, another component of helping solve the software crisis has been put in place: the standard integrated development environment.

The Role of COTS

One sure fire way to reduce software development costs and tame the Software Crisis is to write as little software as possible. Given that a typical project has more software to develop than resources typically allow, one obvious strategy is to concentrate on one’s core value-add and obtain the rest of the software elsewhere. That’s precisely what a COTS approach entails. More commonly associated with hardware, the COTS approach can equally apply to software. One strong advantage of Linux, in fact, is the availability of COTS Linux (e.g., the MontaVista Linux distribution) and the very healthy ecosystem of COTS software available that is ready to run on COTS Linux, all of which dramatically reduces the burden of development from scratch.

Ironically, just as many industries are entering the Software Crisis and should be looking for ways to reduce the amount of software they own, a curious phenomenon has emerged: Do It Yourself (DIY) Linux. Because of the way Linux is developed in a very visible process on the Internet, and because of its licensing, a huge body of source code for Linux and its associated components are readily available on the Web. Notwithstanding that what is on the Web is essentially non-commercial, prototype software, some engineering teams elect to go it alone, or DIY. Fig. 2 shows the reality surrounding DIY Linux and the very real risks associated with the approach.

Although Linux has a deserved reputation for stability and quality, that’s typically based upon an experience with a Linux distribution. A distribution is a packaging of Linux, whether by a commercial or non-commercial organization, and there are many distributions by the way. Building a distribution involves integration and testing many disparate components, and employing an extensive shake-down period where many defects are found and corrected.

The core Linux kernel found at www.kernel.org should most properly be considered alpha or prototype software and is by no means commercial quality. What Fig. 2 illustrates is simply that most of the defects in a software product are found during the phases of development after the prototype phase. Consequently, if one mistakenly embeds “raw” kernel.org Linux in a product, there is a risk that significant defects are present in the software. It is well known that the cost of defect removal gets extremely high if the defects are found after products are deployed. One cell phone recall reportedly cost its maker on the order of $10 million to $20 million and impacted the company’s quarterly profitability.

Having built and delivered embedded Linux distributions for over seven years, MontaVista is qualified to understand the process and its costs. Here are some relevant statistics:

 

• MontaVista Linux 5.xx is composed of 30+ million lines of source code.
• 19 different unsynchronized and non-integrated code repositories provided the code base for the distribution. The code base often changes daily.
• The MontaVista Linux 5.xx distribution targets over 24 microprocessor architectures and variants and 100+ hardware platforms. (Many large companies use a wide range of embedded processor architectures.)
• It supports multiple different host computing environments (e.g. Windows, Linux and Solaris of various revisions).

It took 12 calendar months and approximately 40 plus engineer-years to construct, test and deliver the distribution. The costs of that, plus the cost of infrastructure development for supporting the development process, easily hits the high seven-figure range, which is why applying the COTS approach to Linux makes strong economic sense and is one of the essential components of solving the Software Crisis.

Summary

There’s an old saying that you can tell a pioneer by all the arrows in his back. Appliance developers can avail themselves of the lessons software developers have learned over the last years and embrace the processes known to help tame the software crisis: standards such as Linux and Eclipse; and the COTS approach, namely finding commercial suppliers for Linux and tools whenever possible. Or they can get their own set of arrows.

For more information, enter email: jready@mvista.com