Forums

Test Driven Development (TDD) Framework for Embedded Systems

Started by Unknown April 23, 2014
Hi,
We currently do most (all) our embedeed FW testing on either real HW (if available), using FPGA, or using HW simulator and Verilog models.
We are looking at performing more testing on the "Host" rather than the target, and since this is new to our group, I am looking for suggestions and comments.
BTW, we use C, currently we don't use an RTOS, but going forward, this is something that we will be looking at
Specifically: Do you use a Unit Test Framework, and does it have any features specific to embedded systems?
Thanks,
Steven
On Wed, 23 Apr 2014 11:19:41 -0700, moogyd wrote:

> Hi, > We currently do most (all) our embedeed FW testing on either real HW (if > available), using FPGA, or using HW simulator and Verilog models. > We are looking at performing more testing on the "Host" rather than the > target, and since this is new to our group, I am looking for suggestions > and comments. > BTW, we use C, currently we don't use an RTOS, but going forward, this > is something that we will be looking at Specifically: Do you use a Unit > Test Framework, and does it have any features specific to embedded > systems? > Thanks, > Steven
I'm surprised you haven't gotten any answers yet. I use a unit test framework (cppunit), but it gets used on all of the code that doesn't depend on hardware, or hardware behavior. So there's lots of code that gets "unit tested" by carefully going through and making sure the system works correctly. I've never tried it, but I've heard good things about c-unit (cunit?). The challenging if you're going to test anything that depends on the hardware is making a good behavioral model of that hardware: if your code needs to wiggle pin A, see a number appear in register B, then wiggle pin C, you're going to have to do a lot of work in the hardware abstraction -- and then your unit tests are only as valid as your hardware simulation. That, in a nutshell, is why I don't do more testing on the hardware. -- Tim Wescott Control system and signal processing consulting www.wescottdesign.com
moogyd@yahoo.co.uk writes:
> BTW, we use C,... > Specifically: Do you use a Unit Test Framework, and does it have any > features specific to embedded systems?
I've used different frameworks for different languages. For C, I've mostly used DejaGNU which is more of a regression test system than about unit testing. Some approaches to unit testing tie in with OOP and therefore don't really fit C that well. I think test-driven development isn't a good match for C if that's what you are considering. TDD emerged from languages like Smalltalk which had no static type system but lots of runtime error checks. So most coding errors would cause a runtime exception that would crash the program, and you'd write enough unit tests to exercise all parts of the code, which would usually trigger most of the errors. C has slightly more compile time checks but no runtime checks, so errors that get through the compiler also have a good chance of not getting noticed by simple testing (another aspect of TDD philosophy is to write lots of small tests that are very simple). If you're doing something serious in C, you have to rely more heavily on clean design, careful code reviews, and (lately) fancy static analysis.
On 24/04/14 18:06, Paul Rubin wrote:
>>Some approaches to unit testing tie in with OOP and > therefore don't really fit C that well.
I don't see why you claim that; the problem isn't in the way the program is structured, it is in the way the execution environment allows silent but deadly errors to go unnoticed. And that's just as possible with C++ (ugh) or Objective-C (i.e. a respectable combination of C and Smalltalk)
> I think test-driven development isn't a good match for C if that's what > you are considering. TDD emerged from languages like Smalltalk which > had no static type system
... but which is strongly typed, unlike C. In Smalltalk and Java you simply can't pretend that a Car is a Person (i.e cast a Car into a Person). TDD became really practical with languages like Java, which has both strong run-time typing and a decent type system.
> but lots of runtime error checks. So most > coding errors would cause a runtime exception that would crash the > program,
I don't know where you get that idea. For a start, in Smalltalk the concept if a "program" is a bit vague - you have your code being edited and compiled in an "image", and executed by a VM. Unless there's a VM error, the nearest to a "crash" is either - a "does not understand" message being caught and handled by the code in the image being executed by the VM, or - your code simply not working as expected In other words, the VM+image continues operating normally, without error - completely unlike C, of course.
> and you'd write enough unit tests to exercise all parts of the > code, which would usually trigger most of the errors.
Smalltalk is neither better nor worse than any other language in that respect, since it is dictated by budget and skill.
> C has slightly > more compile time checks but no runtime checks, so errors that get > through the compiler also have a good chance of not getting noticed by > simple testing (another aspect of TDD philosophy is to write lots of > small tests that are very simple).
While both are true, the parentheses aren't related to the preceding sentence, of course.
> If you're doing something serious in C, you have to rely more heavily on > clean design, careful code reviews, and (lately) fancy static analysis.
And most especially a heavily constrained subset of the language and library that don't prevent static analysis. Same is true for ADA.
Tom Gardner <spamjunk@blueyonder.co.uk> writes:
>>>Some approaches to unit testing tie in with OOP and therefore don't >>>really fit C that well. > > I don't see why you claim that; the problem isn't in the way the> > program is structured,
I'm talking about the pattern where you want to test some code that talks to the external world, but instead of making i/o calls directly, it interacts with an "external world" object that you pass into the code (Java applets always work that way). So you can implement both a live object that does the real i/o, and a test or mock object that supplies canned "external" data and checks the responses. Yes of course you can use this approach in C with some contortions, but it's not natural C style.
> TDD became really practical with languages like Java, which has both > strong run-time typing and a decent type system.
Nah, it originated in dynamically typed languages and works quite well in them (Python, Ruby, etc. have popular TDD frameworks).
>> a runtime exception that would crash the program, > Unless there's a VM error, the nearest to a "crash" is either > - a "does not understand" message being caught and handled
Yes, that would count, normal execution of the function under test is stopped.
>> C has slightly more compile time checks but no runtime checks, so >> errors that get through the compiler also have a good chance of not >> getting noticed by simple testing (another aspect of TDD philosophy >> is to write lots of small tests that are very simple). > > While both are true, the parentheses aren't related to the preceding > sentence, of course.
The relation is that traditional TDD doesn't attempt to be exhaustive. The programmer writes tests to exercises typical cases and also the corner cases that come to mind. I don't think JUnit does any automatic fuzz testing. Haskell's QuickCheck does that (I don't know if it was the first) and it has propagated to some other languages like Erlang.
>> If you're doing something serious in C, you have to rely more heavily on >> clean design, careful code reviews, and (lately) fancy static analysis. > > And most especially a heavily constrained subset of the language and > library that don't prevent static analysis. Same is true for ADA.
True, especially in critical systems that must always work (give right answers) for every input, which is probably a more significant issue in embedded systems than in (say) typical web applications or compilers. More usual programs must never give a wrong answer, but if some weird unexpected input ends up making the program fail by printing "error, please contact support", that's just an annoyance rather than a disaster. Simply using memory-safe languages transforms a lot of "potentially disastrous wrong answer" cases into "annoying error message" cases, which is a big help with such programs. Constraining C is of some help with static analysis but it only goes so far. Here is an interesting blog post about why static analysis missed the Heartbleed bug, and another about an improvement that the Coverity guys are trying as a result: http://blog.regehr.org/archives/1125 http://blog.regehr.org/archives/1128
On 24/04/14 20:05, Paul Rubin wrote:
> Tom Gardner <spamjunk@blueyonder.co.uk> writes: >>>> Some approaches to unit testing tie in with OOP and therefore don't >>>> really fit C that well. >> >> I don't see why you claim that; the problem isn't in the way the> >> program is structured, > > I'm talking about the pattern where you want to test some code that > talks to the external world, but instead of making i/o calls directly, > it interacts with an "external world" object that you pass into the code > (Java applets always work that way). So you can implement both a live > object that does the real i/o, and a test or mock object that supplies > canned "external" data and checks the responses. Yes of course you can > use this approach in C with some contortions, but it's not natural C > style.
I first did that, in C, in (gulp) 1982. It made the embedded system very easy to debug by running it in a test harness on a PDP11 instead of the embedded Z80. It is no more natural or unnatural than any other model (either implicit or explicit) in C or any other language. All programs contain many different models; the better programs make the models explicit.
>> TDD became really practical with languages like Java, which has both >> strong run-time typing and a decent type system. > > Nah, it originated in dynamically typed languages and works quite well > in them (Python, Ruby, etc. have popular TDD frameworks).
TDD was popularised there, but it didn't originate there. I've come across many ad-hoc TDD-style frameworks, starting with those for pure hardware.
>>> a runtime exception that would crash the program, >> Unless there's a VM error, the nearest to a "crash" is either >> - a "does not understand" message being caught and handled > > Yes, that would count, normal execution of the function under > test is stopped.
There are significant qualitative differences. In particular you can easily see, inspect and debug the path and objects that lead up to the well-contained explicit message. Contrariwise, in C you are lucky if you detect that something happened at some time in the past, but all useful evidence has not been retained.
>>> C has slightly more compile time checks but no runtime checks, so >>> errors that get through the compiler also have a good chance of not >>> getting noticed by simple testing (another aspect of TDD philosophy >>> is to write lots of small tests that are very simple). >> >> While both are true, the parentheses aren't related to the preceding >> sentence, of course. > > The relation is that traditional TDD doesn't attempt to be exhaustive. > The programmer writes tests to exercises typical cases and also the > corner cases that come to mind. I don't think JUnit does any automatic > fuzz testing. Haskell's QuickCheck does that (I don't know if it was > the first) and it has propagated to some other languages like Erlang. > >>> If you're doing something serious in C, you have to rely more heavily on >>> clean design, careful code reviews, and (lately) fancy static analysis. >> >> And most especially a heavily constrained subset of the language and >> library that don't prevent static analysis. Same is true for ADA. > > True, especially in critical systems that must always work (give right > answers) for every input, which is probably a more significant issue in > embedded systems than in (say) typical web applications or compilers.
Of course. Embedded environments are second-rate by comparison.
> Constraining C is of some help with static analysis but it only goes so > far. Here is an interesting blog post about why static analysis missed > the Heartbleed bug, and another about an improvement that the Coverity > guys are trying as a result: > > http://blog.regehr.org/archives/1125 > http://blog.regehr.org/archives/1128
Sure. There are no silver bullets.
On Wednesday, April 23, 2014 2:19:41 PM UTC-4, moo...@yahoo.co.uk wrote:
> Hi, > > We currently do most (all) our embedeed FW testing on either real HW (if available), using FPGA, or using HW simulator and Verilog models. > > We are looking at performing more testing on the "Host" rather than the target, and since this is new to our group, I am looking for suggestions and comments. > > BTW, we use C, currently we don't use an RTOS, but going forward, this is something that we will be looking at > > Specifically: Do you use a Unit Test Framework, and does it have any features specific to embedded systems?
Hi Steven, I would suggest the first step for you is changing you software architecture slightly. In any embedded system there is the software that deal with the bit twiddling of the hardware and the application logic that makes the product what it really is. Separating those two aspects is important. So I suggest you design with a hardware abstraction layer and application layer. All the application layer code can be tested on a host. The hardware layer code can also be unit tested if the hardware can be separately simulated (e.g., the communications hardware in the device does not share bits in registers with the motor control). You mentioned that you do't yet use an RTOS. Are your programs written in the Grand DO_ALL loop? Long term, I think you will find an RTOS can really simplify your development. ed
On 26/04/14 13:55, Ed Prochak wrote:
> On Wednesday, April 23, 2014 2:19:41 PM UTC-4, moo...@yahoo.co.uk wrote: >> Hi, >> >> We currently do most (all) our embedeed FW testing on either real HW (if available), using FPGA, or using HW simulator and Verilog models. >> >> We are looking at performing more testing on the "Host" rather than the target, and since this is new to our group, I am looking for suggestions and comments. >> >> BTW, we use C, currently we don't use an RTOS, but going forward, this is something that we will be looking at >> >> Specifically: Do you use a Unit Test Framework, and does it have any features specific to embedded systems? > > Hi Steven, > > I would suggest the first step for you is changing you software architecture slightly. > In any embedded system there is the software that deal with the bit twiddling of the hardware > and the application logic that makes the product what it really is. Separating those two
> aspects is important. I wholeheartedly agree with that.
> So I suggest you design with a hardware abstraction layer and application > layer.
For many embedded applications a "full blown" HAL would be overkill. Usually all that is needed is a clear distinction between the objective/intent (e.g. indicate "error") and mechanism (e.g. set bit 5 high to turn on LED). If there is a single lightweight mechanism coupling all the mechanisms and objectives, then it will help when replacing the test harness on the host with the real logic, and replacing the test harness on the target with the application. > All the application layer code can be tested on a host. The hardware layer code can also
> be unit tested if the hardware can be separately simulated (e.g., the communications hardware > in the device does not share bits in registers with the motor control). > > You mentioned that you do't yet use an RTOS. Are your programs
> written in the Grand DO_ALL loop? Long term, I think you will
> find an RTOS can really simplify your development.
That's a classic build-vs-buy dilemma, and whether it is worth it largely depends on what protocol stacks come with the RTOS. If you can cast your application into the form of FSMs that receive[1] and generate[2] events, then an RTOS is very likely to be helpful -- and it will greatly ease your debugging and integration. But you knew that, I expect. [1] e.g. motor too hot, button pressed etc [2] e.g. turn led on, warn operator, switch fan on
Am 23.04.2014 20:19, schrieb moogyd@yahoo.co.uk:
> Specifically: Do you use a Unit Test Framework, and does it have any features specific to embedded systems?
Hello Steven, maybe the Unity test framework is what you are looking for. It is a tiny test framework for embedded systems and it is completely written in C. There was an article on Dr. Dobb's last year: http://www.drdobbs.com/testing/unit-testing-in-c-tools-and-conventions/240156344 Best regards, Olaf
I use Ceedling for TDD in Embedded, it's easy to configure and use. It uses CMock to automatically mock modules by using the Header file.

http://throwtheswitch.org/white-papers/ceedling-intro.html