Embedded Linux Programming

Presented by Robert J. Sandilands

What is Embedded Systems?

Embedded systems can be described as systems that normally operate with limited resources to do a very specific and dedicated task. With limited resources we talk about limited RAM, storage capacity and interaction with the outside world.

Examples of embedded systems include cell-phones, printers, routers, and embedded firewalls. There may be some debate whether to include Personal Organisers and Games Consoles into this category. Obviously it is quite different to program a printer than it is to program for a Games Console and the resources available is quite different. The main reason for this is the requirements people have for these devices.

When we talk about limited resources it means different things for different environments. For a cell-phone it means very limited power usage, a few kilobytes of RAM and several hundreds of bytes of permanent storage. For a game console there may be more than sufficient amounts of RAM, although it would be a fixed number, no permanent storage capacity and limited user input requirements but very high user output requirements.

An easy way to identify and embedded system is whether it is something you can take out of the box, put down, configure and use without any additional resources or interaction. No operating systems to load, software to install and configure.

Why would people want that? Canít a normal PC do all the same things? Yes, you can program a computer to act like a cell-phone, but what would you rather have in your pocket: a 300g cell-phone or a 4 kg PC? Which of the two would be the more stable, reliable and the easiest to use?

If a system is dedicated to do one task and one task alone, you can control nearly all the operating parameters and therefore have a measure of control over the environment that makes it easier to guarantee reliability, availability and security. Due to the fact that the complete embedded system is geared towards one specific task you can optimise the user interface to ensure maximal ease of use for that one application.

That is most likely why most people learn to use a cell-phone with more ease than they do a new computer. I do have to mention that VCR manufacturers seem to go out of their way to prove this assumption a fallacy.

The other term that appears when people talk about embedded systems is real-time. This is a term used to define the responsiveness and/or time-accuracy of a computing system. Real-time from a user perspective is whether the ATM takes less than 3 seconds to respond to a button push or not. This time may vary somewhat but should be below some fuzzy threshold. This is called soft real-time. If a computer has to do something inside a very specific time-limit and even more specifically at the same time every second/minute/hour with very little margin for drift/change, this is called hard real-time. An example would be a control system for a welding machine that has to do measurements and control decisions of the position of the machine at fixed intervals or it would not be able to fulfill its requirements for accuracy.

Outside of the control systems or manufacturing environment it is highly unlikely that you will find hard real-time systems.

Where do I fit?

I build embedded firewalls. Why do we make embedded firewalls?

We think there is a market for an easy to install, easy to manage, low-cost firewall that is reliable and secure. Embedded type technology allows us to ensure the ease of use and security. Due to the low hardware requirements implied by the very specific implementation and control over the environment the hardware is also minimal which helps to insure a low cost.

Why Linux?

When building an embedded device you have several questions to answer. These are:

Do I need a fully-fledged operating system?

If your device has 32 KB of RAM the answer to this is most likely no. If however you need access to device drivers, multitasking or other advanced features the answer is yes and then you have to select the most suitable operating system and make it fit within your hardware constraints.

What are the real-time requirements for this device?

Does this device need to do hard or soft real-time processing? For hard real-time processing you most likely have to either license a real-time OS, try to convert an OS to do hard real-time or write your own OS. Except if you are braver than I am the first choice make the most sense. For the soft real-time case this can be a consideration in choosing the OS but not the major deciding factor.

What hardware will be used?

This is a critical decision as it controls most of the other choices you make. With a cell-phone this choice would be governed by physical space, weight and power usage. With a games console sound and video processing ability would govern it. This makes each embedded device an unique opportunity with each own challenges and rewards.

What are the driver requirements for the environment to be used in the end?

This question forces you to measure the compatibility of your selected hardware against the possible Operating Systems that you are evaluating. In our environment we had to abandon PicoBSD as a choice due to the lack of driver support for the network cards and flash disks we were using at that time.

Control over the environment to be used.

Due to your hardware constraints you are limited in many of your design choices. You want your operating environment to not further restrict your choices. This basically ends up with the Open Source versus Propriety Technology debate.

The advantages of Propriety Technology for the embedded space would be:

  1. Specialised and tested environment
  2. Bigger User base within the environment you are working in.
  3. Somebody to blame if something does not work.
  4. Lower level of skill needed for implementation.

The advantages of Open Source for the embedded space would be:

  1. Full control over the operating system, its resources and requirements.
  2. If something does not work, you can fix it yourself.
  3. If a feature or ability is missing, the ability to add it yourself.

The disadvantages of using Propriety Technology

  1. If you donít have huge volumes the manufacturer may not be bothered to add features you need or fix bugs that stop you from doing your work.
  2. The technology may not work as advertised or as you understood it.
  3. Even if you have huge volumes of sales the manufacturer may not be interested in helping you anyway.
  4. All or some of your assumption about the advantages may not be true or not required for your implementation.

The disadvantages of using Open Source

  1. Higher level of skills needed.
  2. If something is wrong or missing you most likely have to fix it yourself.
  3. Itís Open Source, there must be other catches :-)

One important point that is obvious in its omission is royalties and license fees. Royalties and license fees influence the final price of the product but does not influence any of the technical processes except when you end up with some interesting licensing schemes. The issue of these costs should be evaluated as part of the total cost of implementation when making your decisions. It is important to remember that although something may be Open Source there is an indirect cost to utilising it. Cost is not the reason to use Open Source products, freedom is.

The reasons for our choice

We wanted full control over the complete environment due to our space restrictions and security requirements. That left us with the Open Source Operating Systems. We evaluated Linux, FreeBSD and PicoBSD. Unfortunately the BSD environment had to be abandoned due to the lack of drivers for the embedded network card and the specific flash disk we used at that stage.

In the end we think we made the right choice to go with Linux. We could implement our embedded device quickly without having to write our own device drivers. We are very glad we took the choice of an Open Source operating system due to the control we have over the features and size of the OS and the one bug we had to fix inside the device driver for the network device. With a Propriety OS we would most likely have been stuck or been forced to wait for the manufacturer to fix the problem.

Tricks and Catches

What is difficult about developing software for an embedded device and especially for an embedded firewall? You install Linux, enable iptables, set up some rules and you have a firewall.

Now let us remember that we only have 16 MB of disk space whilst a standard Linux installation takes 200 MB or more. If you install the complete Redhat Linux 7.0 you end up around 1.2 GB. So let us cut the operating system to just the bare necessities. So you modify the boot-up process, remove unnecessary programs and libraries and include just the kernel and some small supporting programs like iptables. If you are lucky you have cut enough fat of Linux to be able to fit it in. But you will most likely see that it is not enough. You will then have to get alternative utilities with a smaller disk footprint and even go as far as compressed file systems.

Something else then occurs, you donít have a standard keyboard or a standard screen so the fact that you had to delete Gnome and KDE does not bother you that much. Now you either have to create a serial terminal interface, easy, or some network interface that configures itself, not so easy. Then you have to either use a LCD or a set of LEDís to provide visual feedback to the poor programmer and later to the user.

When you develop software for an embedded device you have several budgets to look at and they are in order of importance:

  1. Disk space: This is a very limited resource that can not be exceeded by even the smallest margin.
  2. RAM: This is a very limited resource that can not be exceeded by even the smallest margin.
  3. People: Very rare resource that can only do one thing at a time and needs to sleep for at least a quarter of the day.
  4. Money: The tooth fairy also exists :-)

This adds several new dimensions to project planning and implementation and makes for some interesting lessons if you are not careful.

The basic features that I would want on a firewall are the firewall software and a way to configure it. Unfortunately the clients also want several advanced features like SNMP, DHCP and DNS on a firewall. Fortunately Open Source provides us with excellent tools to implement these requirements.

For SNMP the standard implementation is UCD-SNMP. The first time we compiled it, it turned out to be approximately 30 MB in size. This exceeded the disk space budged by about 29.5 MB. We had several choices:

  1. Look for an alternative package.
  2. Write our own SNMP package.
  3. Try to cut UCD-SNMP to within budget.
  4. Donít implement SNMP.

In this case there did not seem to be alternative packages available that were either Open Source or Propriety. We than looked at the RFCís for the SNMP specifications and decided that writing our own version should be the very last option. We had a definite requirement stated by clients and potential clients for SNMP functionality so the lack of this functionality will most likely have a negative influence on the marketability of our products.

We then proceeded to try and limit the size of the existing package. After some hacking of the configuration files and some liberal removal of features from the UCD-SNMP we ended up with a version that exceeded our 500 KB budget by 200 KB.

Fortunately we found Open Source DHCP and DNS implementations that were 400 KB under budget in total so we could afford to go over budget with the SNMP implementation. Unfortunately the DNS implementation can not do DNS Zone transfers. This is the cost we had to pay to be able to get the small disk footprint and allow for the larger size of the SNMP implementation.

Open Source and Embedded Programming

The previous section explains the advantage of using Open Source in an embedded space.

The sadness however of embedded software development is the focus you have and the limitations you have when developing in this environment. The software is targeted in such a way that it is difficult to find ways of sharing the development effort in a meaningful way.

Even if we gave away all our source code the only use will be for people implementing precisely the same environment as we do, and due to the huge diversity of choices you have in embedded hardware this would be difficult to achieve.

This problem seems to be universal in the embedded software development field. That cancels many of the advantages of using Open Source.

An idea we are playing with now is to try and generalize building blocks of our software and make that available as re-usable tools for other people. With the high level of integration and specialization inherent in embedded programming it is not an easy task and may even be impossible.

We will continue to try and solve this problem and it seems like other companies have the same hope. Iím sure that even with these obstacles we will find a solution to this problem in the near future.