Unit Tests for Embedded Code

• Software Development
• Tutorials
• Debugging

I originate from an electrical engineering background and my first industry experience was in a large, staid defense contractor. Both of these experiences contributed to a significant lack of knowledge with regards to software development best practices. Electrical engineers often have a backwards view of software in general; large defense contractors have similar views of software and couple it with a general disdain for any sort of automation or ‘immature’ practices.  While there is significant value in the more traditional methods of the more established engineering disciplines the software industry is pioneering many useful practices that should be adopted as much as possible by other disciplines.  One very useful tool I’ve used in developing embedded software is the unit test.

What are Unit Tests?

Unit tests are functional tests of individual program elements. That’s a lot of high-minded words. A more straightforward way of putting it is to say that unit tests are to verify that individual functions work the way they’re supposed to. They fit into an overall test strategy and work best when used with a variety of testing methods:

• Unit tests verify that individual functions work the way they’re supposed to. They work best when they test only one function at a time - not the entirety of your codebase.
• High-level tests verify that the entirety of your code (the sum of the functions) does what it’s supposed to do - typically this is evaluated at the boundaries of your devices (i.e., for most microcontrollers, at the device pins).
• Integration tests ensure that your device works properly within the context of the system as a whole: does it play nice with the other components of the system? Does it respond to serial messages as it’s supposed to, when it’s supposed to?

My experience of larger defense-focused engineering firms has been that everything other than integration testing is generally ignored. My frustration with those practices, subsequent experience with testing and personal research into good software development practices has convinced me that while it may be time-consuming to implement all of these levels of testing it’s rarely a waste of time.

Why Unit Test?

Unit testing helps to identify bugs that may be hidden or difficult to find when all of your code is run at once. One of the most frustrating aspects of debugging software is determining where exactly a bug is within your code. If you’re doing high-level testing only (i.e., testing your whole program at once) you’ll be reduced to single-stepping through a debugger, toggling port pins or using a plethora of debug printf statements to track down where an error may be. Unit tests give you the certainty that individual functions are operating as they should be, freeing up your debugging time to examine the less-certain areas of your code.

Unit testing also acts as an insurance policy against future code changes. Every project goes through multiple iterations, hardware changes, coding standards changes, feature creep, etc. Even if your requirements don’t change you’ll always be refactoring code for greater readability, lower memory usage or greater speed. Do you need to implement a quick sort instead of a bubble sort in a function that returns the maximum value in an array? No matter which sorting algorithm you use the function should still return the maximum value from the array - your unit tests don’t change when you change algorithms. No matter what is under the hood in your functions, unit tests verify that they still operate as they’re supposed to.

Unit testing is also important because it allows you test your code in non-ideal situations. Defensive coding practices dictate that preconditions should always be checked and error conditions should be handled. In the real world these situations may never arise: a parsing function may never encounter a malformed data stream if it’s tucked deep within your program, guarded by multiple levels of functions performing their own checks. However, if you reuse that code in a different application with fewer safeguards you may start exercising your error handling functionality and find that your functions are doing something foolish like indexing an array out of bounds, dividing by zero, or silently ignoring errors rather than reporting them up the chain. Unit testing lets you exercise these portions of your code without having to find ways to produce complex error conditions in the real world.

When to Unit Test

Ideally you would generate unit tests for every function in your program. Generally though this gets tedious - especially if you’re the only developer on the project. While I can’t tell give you an exact formula for determining which functions you should test and shouldn’t there are a few good rules of thumb that I can give you:

• Your main loop - main() shouldn’t have a significant amount of logic or processing in it and for embedded applications it will involve an infinite loop. Generally infinite loops don’t make for good, bounded unit tests.
• Functions involving hardware - Any function that relies on hardware to operate is in general going to be difficult to generate a unit test for: either you’d have to test with the hardware or find a way to fake out the hardware. Either way, most of the difficulties you’re going to have with these sorts of functions generally involve the hardware itself, not the logic inside of them. Unit tests often don’t tell you anything useful about these sorts of functions, so you can profitably avoid testing them.
• Functions with little or no logic - Not all functions are created equal. For example, in object-oriented languages like C++ there exist get and set functions whose sole purpose is to read and write to a variable hidden within an object. You may have similar functions in C whose only purpose is increase code clarity or service some particularly difficult design decision. These functions are not important to test - if anything do them last!

Generally this is going to leave several classes of functions:

• Functions that implement complex mathematics - It’s very easy make mistakes when implementing these sorts of functions which makes it critical to verify they work before integrating them into a larger program.
• Functions that implement complex logic - It’s important to unit test function that make a lot of decisions for several reasons: to ensure that the decisions operate correctly, and to ensure that all of the code paths are exercised and tested.
• Functions with significant failure modes - This includes anything that will process raw data or signal faults within your system. This functionality is typically very important to get right: you always want your system to fail safe.

Approaches to Implementing Unit Tests

Although the idea of unit tests is fairly straightforward there are often many difficulties in actually implementing them. I’ll discuss two general approaches.

On-Target, No (or minimal) Framework

The simplest way to test your functions individually would be to create special programs that run on your microcontroller (often called the target) that implement your tests without utilizing any special frameworks, libraries or other software packages. There are many benefits to this approach. One of the main ones is that your code is running on the target itself so if your test passes you can be pretty certain your code will work once it’s integrated into the rest of the program. It also doesn’t require you to learn or buy any additional software to perform testing - you can get started right away with only the tools you need to develop code. And finally, you have the added benefit of being able to test hardware at the same time as software if you want.   At first glance there doesn’t seem to be anything that would preclude this approach and it can be effective but there are several wrinkles that introduce difficulties:

• Reporting results can be ungainly - Several approaches are possible: use a serial port to report results to a desktop PC, or use a debugger to halt on errors and inspect the state of the program. The serial port has the issue that it can be less than straightforward report anything more complex than a pass or fail result. Debuggers are always nice, but the hardware can be expensive and debuggers can sometimes modify the way that code executes. This can lead to questions as to whether a failure is real or simply induced by the test environment.
• If any aspect of the test compromises the integrity of the overall program (accidentally wiping the processor state, overwriting important parts of global memory, inadvertent infinite loop, etc.) you may not get a result that’s more informative than ‘it failed somehow’.
• Running on the target might allow error conditions that aren’t related to the code under test. If your device initialization routines don’t configure the clock correctly, or you fail to clear the watchdog timer it will cause spurious failures in your tests. You could spend a long time trying to track down these ‘bugs’ only to find the issue is in another part of your code.

Hosted with Framework

Some of the issues associated with running on the target without a framework can be remedied by moving the test environment to a desktop PC (a hosted environment) and utilizing a unit test framework. Unit test frameworks are very nice pieces of software that smooth out the entire process of writing tests, running them and generating reports. They offer a lot more options than rolling your own framework. Some of these include:

• Report generation - Printf may be your friend, but there’s a lot to be said for an HTML file that tells you percentage pass/fail, which tests failed, execution times, etc.
• Overflow and timeout checking - One of the problems with not using a framework on the target processor was the inability to diagnose whether memory was overwritten or an infinite loop occurred. Unit test frameworks in a hosted environment can typically tell you if the code is trying to access memory it shouldn’t, or cause a test failure if it doesn’t finish within a certain amount of time. Catching these sorts of mistakes on your desktop PC (where you can debug them more easily) will save you a lot of time
• Coverage tools and reports - One issue that I haven’t talked about yet is generating coverage results. When writing a test you’ll want to make sure you’ve exercised all of the execution paths in the function. Coverage tools tell you which code has run and which wasn’t. While there exist coverage tools for some microcontrollers you can never be sure that yours will and you may not want to pay for it when it does exist. For your desktop PC there are free coverage tools available such as the GNU coverage utility - GCov.
• Smoother and faster testing - Desktop PCs are fast nowadays and will have no problem running your tests in the blink of an eye; microcontrollers are much more limited. On a desktop all you have to do is compile, run and get your results. For a microcontroller you have the added step of programming the device (which can be a frustrating addition to the process when you’re trying to debug). You also have more automation options on your desktop than you do on a target device.

Some people may cry foul: it’s unreasonable and difficult to test code meant for an embedded device on a desktop PC. Generally, there are two main complaints.  

The first is that embedded code won’t easily compile for a desktop PC; the code is too dependent on the target’s compiler, libraries and overall environment to allow compiling on a desktop PC. This complaint is generally true, but it can be mitigated through good code design and practices. If a majority of your functions depend explicitly on the hardware or software libraries present only on your target it means that very likely your code isn’t organized properly. For the sake of testing and reuse, functions which are limited in their use to a very specific set of circumstances (compiler, libraries, hardware, etc.) should be distinct from functions which have duties separate from them. If you follow this paradigm a majority of your functionality will require only minor stubbing to operate on the desktop PC or on the target itself. For example, if the compiler you’re using for your target is a GCC-derived compiler you probably have the option of using standard integer types (uint8_t, int32_t, etc.) rather than the built-in types unsigned char, int, long, etc. which may be slightly different for your target architecture. Library-specific functions can be overridden

The second is that even if you can compile and test your code on a desktop PC it doesn’t tell you enough about how the code is going to act on the embedded target itself. There is some truth to this statement - testing on a desktop PC is definitely not the same as testing on the actual device. However, this is not the goal. The vast majority of bugs you’ll find in your code have nothing to do with the hardware it’s running on. More likely you’ll find your typical off-by-one issues, minor gaps in logic and other mundane errors in your code. True, there’s nothing saying with 100% certainty that if your tests pass on the desktop PC that your code will function correctly on your target. That’s not the point of these tests: the point is to debug the simple, foolish mistakes that we always make in a more capable debugging environment than we often find on embedded devices.

Other Approaches

There are other potential twists on both of these approaches. I’ve worked with large software packages which combine unit tests frameworks which run the tests on the target. While these packages are expensive and sometimes painful to use there’s no denying the benefits of combining the features of a unit test framework with the assurance of testing done on the target itself. Generally this isn’t going to be an option for hobbyists or small companies due to the complexity and cost.

There are lightweight unit test frameworks that don’t rely on heavy external libraries or facilities available only on a desktop PC. Generally these frameworks will either be included in your project as a single header file or a pair of header and source files with few or no dependencies. Because of this they can easily be compiled for your target framework so your tests can run in the native environment of your target. Despite this, these frameworks are generally less feature-filled than the more extensive hosted frameworks, but they definitely have their advantages.

If you like the convenience of testing on a desktop PC but want the assurance of testing on the actual hardware, you might look into whether a simulator is available for your target architecture. A simulator allows you to compile your tests for your target architecture and run them as if they were on the actual hardware, but with the benefits of your desktop PC: namely, increased speed and a much more user-friendly debug interface. Simulators are generally not universally available, nor are they universally free - but there are exceptions. AVR Studio is free and comes with a simulator for all AVR chips. Cursory investigation seems to indicate that there might also be solutions for the MSP430, ARM and maybe others.

A Simple Unit Test

My preferred method of implementing unit tests is to use a hosted approach with my favorite unit test framework: Check

I won’t go into the full spiel about it (that’s what the website is for) but I will mention that it’s a unit test framework for C that is simple to use, has some nice features I appreciate and is targeted towards Unix-like operating systems (which means you’ll need Cygwin and/or MinGW to run it on Windows). It’s certainly not the only unit test framework available for C, nor perhaps is it the most appropriate framework to use for embedded code. You can examine a large list of unit test frameworks on Wikipedia and do some more research for yourself, or find a framework for an alternative language.

Due to the relative complexity of installing and configuring Check on Windows I won’t discuss how that is accomplished, and instead focus on what you’ll want to do with it to test your code when you finally have it installed. This is the function we’re going to test:

uint8_t test_function(uint8_t a, uint8_t b)
{
    if(a>b)
    {
        return a-b;
    }
    else
    {
        return b-a;
    }
}

What does this function do? Nothing really special, but it has some logic, has some math, and is just complex enough that my sleep-deprived brain has difficulty with it. That means it’s perfect for writing a quick set of unit tests against.

Before you start worrying about how the unit test framework works or how you’re going to compile and build your tests you should spend a second thinking about what sort of inputs you’re going to test your function with. I could write a whole article on selection of inputs for unit tests to maximize test validity, but for this specific situation in which all of the inputs and outputs are integers and there’s one decision point there are a few quick guidelines I can share:

• For integer inputs always test 0 and 1 - these are considered special cases for integers and often produce interesting behavior
• Always test the maximum and minimum values for the integers - in this case since we have 0 (the minimum value) taken care of it means we should make sure to test 255 (the max for an 8-bit unsigned integer)
• Test around your decision points - test a slightly greater than b, a=b, and a slightly less than b. This ensures that your logic is correctly implemented at the decision point.
• Test a uniform distribution of values within the range - you always want to make sure that your function works correctly for a wide range of values within your normal operating range. In this case the range is 0-255 and I’ve decided to test five uniformly distributed sets of values (they’re not actually uniformly distributed - I’m winging it). Generally, the more sets of inputs you test the merrier (with exhaustive testing of every value in the range being the most merry) you’ll spend lots of time writing and running the test, so don’t go overboard.

The paradigm that Check uses to organize all of its testing is as follows:

• A test case contains multiple unit tests for an individual function. In this example, each unit test is a set of inputs passed to the function and the expected result.
• A suite contains multiple test cases - typically all of the cases associated with a set of functionality (called a module in the Check parlance)
• A suite runner executes a suite of test cases

This is a fair amount overhead for a unit test framework, but it’s not completely over-bearing. A fully-implemented unit test for the above function is seen below:

#include <check.h>
#include <stdint.h>
//Top-level: a suite
//Suites contain a test case
//Test cases contain unit tests
//Unit tests are equivalent to test cases discussed above
//Function under test
uint8_t test_function(uint8_t a, uint8_t b)
{
    if(a>b)
    {
        return a-b;
    }
    else
    {
        return b-a;
    }
}
//One unit test
START_TEST(basic_test)
{
    fail_unless(test_function(0,0)==0,"Case 0,0");
    fail_unless(test_function(1,0)==1,"Case 1,0");
    fail_unless(test_function(0,1)==1,"Case 0,1");
    fail_unless(test_function(1,1)==0,"Case 1,1");
    fail_unless(test_function(255,0)==255,"Case 255,0");
    fail_unless(test_function(0,255)==255,"Case 0,255");
    fail_unless(test_function(255,255)==0,"Case 255,255");
    fail_unless(test_function(4,3)==1,"Case 4,3");
    fail_unless(test_function(3,4)==1,"Case 3,4");
    fail_unless(test_function(100,54)==46,"Case 100,54");
    fail_unless(test_function(54,100)==46,"Case 54,100");
    fail_unless(test_function(27,36)==9,"Case 27,36");
    fail_unless(test_function(15,4)==0,"Case 15,4");
}
END_TEST
int main (void)
{
    int number_failed;
    //Create the test suite for the test function 
    Suite *s = suite_create("Test");
    TCase *tc_core = tcase_create("Core");
    tcase_add_test(tc_core,basic_test);
    suite_add_tcase(s,tc_core);
    SRunner *sr = srunner_create (s);
    srunner_run_all (sr, CK_NORMAL);
        number_failed = srunner_ntests_failed (sr);
        srunner_free (sr);
        return (number_failed == 0) ? 0 : -1;
}

Needless to say this isn’t the exact structure you would use when you unit test your code - the function under test would be in its own source file for one. This is just an easy way of showing you all of the code necessary to get your unit test framework up and running. The structure of the code is as follows:

The test_function is the function under test. There are no special modifications you’ll need to make to it in order to unit test it.

The basic_test test is the unit test for the test_function. There are many tests incorporated into this one unit test but that may not always be how your unit tests are organized. With Check, every test runs in its own memory space under its own process, so that will figure in to how you break up your tests. For simple functions like this it doesn’t really hurt anything to group them like this. Each specific input/output combination is tested via the fail_unless statement. Check offers a variety of possible ways to implement tests: fail_unless, fail_if, ck_assert_int, ck_assert_str, etc. You have a lot of options. I like fail_unless because it offers the ability to write a string describing the test that failed. I chose the specific tests for the function based on the criteria I discussed earlier and you can see many of the items I discussed (i.e., testing 0,1, a range of nominal values, limits of the type, etc.). Each specific case is labeled with a string that is reported in case of any failures - this helps you track down any issues (especially since you can print variable data from these calls just like printf).

Inside of main all of the test overhead is initialized: The suite is created via the suite_create call (obvious, I know). The “Core” test case is initialized via the tcase_create call and the unit test basic_test is added to the test case by the tcase_add_test call. To execute the suite requires the calls to create the runner (srunner_create) which then runs the suite (srunner_run_all) and the number of failed tests is retrieved (srunner_ntest_failed) before destroying the runner (srunner_free).  If any tests failed, the program returns a status of -1, otherwise 0 if everything passed. It’s all a bit complex for a simple application, but once you’ve addressed all of the overhead you probably won’t have to deal with it too much afterwards.

This file can be compiled (on Cygwin or any Unix-like operating system) by using the command ‘gcc -lcheck -o

’. When run, it produces the following output:

Running suite(s): Test

0%: Checks: 1, Failures: 1, Errors: 0

basic_unit_test.c:38:F:Core:basic_test:0: Case 15,4

You may be wondering why there are any failures. In fact, I inserted a failure into one of my unit tests just to show you what it would look like. The failure is reported with the string passed to the fail_unless function and is identified explicitly with the line in the source file that produced the error. That’s really slick and represent one of the benefits of using a hosted unit test framework as opposed to rolling your own. This isn’t the only reporting option though - Check offers a variety of output formats from which you can choose.

I hope I’ve swayed you to start writing unit tests for your code and that my brief tutorial has been helpful in giving you direction to implement the tests. Over a long period of time practices such as this can be very beneficial and save you a large number of headaches, so I hope you take some of these lessons to hear.

Sign in