Test framework for automated FW testing

Hi!

I'm planning to automatically test a reasonably simple board in an 
embedded system. The tests cover functional requirements of the 
firmware. Unit testing is not needed (it's already done).

The module under test talks to a 3 other boards via SPI and serial. It 
has a few sensors and relays connected. All external boards and sensors 
will be simulated by a software test bench running on a PC.

I'll create small programs that simulate external components using some 
language that can access DLL-s or Active X components (SPI and I/O 
adapters provide those). The details largely depend on the test 
framework which must control those simulations.

This brings me to the actual problem - which testing framework to use? 
There are shiny expensive commercial solutions (such as TestComplete and 
NI TestStand). There are FOSS programs (dejagnu). There is a pile of 
unit testing frameworks, 2-3 per every language ever conceived.

Most of these emphasize PC testing - pushing buttons, looking for text 
responses etc. I assume testing an embedded system is a bit different - 
lots of stuff going on at any given time, it is less determined, there 
are real-time considerations etc. Then again, maybe I assume wrong.

I suspect experience can sort out this mess. What frameworks have you 
used for testing firmware and how did it work out?

--
Kind regards,
Tarmo Kuuse

Tarmo Kuuse wrote:
> Hi!
> 
> I'm planning to automatically test a reasonably simple board in an 
> embedded system. The tests cover functional requirements of the 
> firmware.

[...]

> This brings me to the actual problem - which testing framework to use? 
> There are shiny expensive commercial solutions (such as TestComplete and 
> NI TestStand). There are FOSS programs (dejagnu). There is a pile of 
> unit testing frameworks, 2-3 per every language ever conceived.

[...]

> I suspect experience can sort out this mess. What frameworks have you 
> used for testing firmware and how did it work out?

This is a well known problem. No one knows your system better then you 
are. Hence, unless your board belongs to a category that they already 
have in their framework, you will do better with the test framework of 
your own, created in house from the scratch. Otherwise you will have to 
fight with the bugs and constraints of somebody else's instead of 
working on your target system.

Since there can be a wild variety of the different conditions and 
situations, the test system should target only the relevant case or the 
specific set of cases. Making the universal test system === bloat, bugs 
and limitations. The right way to do it is developing the test suite as 
the common program in the general purpose programming language such as 
C++.


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com

Vladimir Vassilevsky wrote:
>> I'm planning to automatically test a reasonably simple board in an 
>> embedded system. The tests cover functional requirements of the firmware.

[snip]

> This is a well known problem. No one knows your system better then you 
> are. Hence, unless your board belongs to a category that they already 
> have in their framework, you will do better with the test framework of 
> your own, created in house from the scratch. Otherwise you will have to 
> fight with the bugs and constraints of somebody else's instead of 
> working on your target system.

You describe something slightly different. I probably used the term 
"test framework" too loosely. I'll just call it a "test tool". I 
certainly don't expect it to know my system. I implement all required 
knowledge of the system myself - it is embedded in test cases and HW 
simulation.

What I am looking for is a tool that allows me to define individual test 
cases in a high-level language. The language can access our I/O devices 
(DLL-s or Active X components). It is real-time aware (can use delays, 
timeouts). Test cases can be grouped, run, paused, stopped. Tool outputs 
a formatted test report (usable in a review with managers).

> Since there can be a wild variety of the different conditions and 
> situations, the test system should target only the relevant case or the 
> specific set of cases. 

I did not get the point. It's the test engineers job to design tests so 
relevant stuff is covered. Test system just executes them.

> Making the universal test system === bloat, bugs 
> and limitations. The right way to do it is developing the test suite as 
> the common program in the general purpose programming language such as C++.

A team in my company has been using NI TestStand and LabView for years. 
It is cumbersome and an overkill for our project. We have looked at 
TestComplete. That fills the requirements, but it's still too 
complicated. So I'm looking for viable alternatives.

We have considered creating a custom test suite in Python or C++. It 
seems to be a good idea, but our development resources are quite 
limited. I would very much like to minimize the amount of work and not 
write it from scratch. Some elements - managing tests, running them and 
reporting results - are quite universal. Is there any effort to make a 
reusable library or framework for this purpose?

--
Kind regards,
Tarmo Kuuse

On Dec 29, 6:04=A0am, Tarmo Kuuse <tarmo.ku...@mail.ee> wrote:
> Vladimir Vassilevsky wrote:
> >> I'm planning to automatically test a reasonably simple board in an
> >> embedded system. The tests cover functional requirements of the firmwa=
re.
>
> [snip]
>
> > This is a well known problem. No one knows your system better then you
> > are. Hence, unless your board belongs to a category that they already
> > have in their framework, you will do better with the test framework of
> > your own, created in house from the scratch. Otherwise you will have to
> > fight with the bugs and constraints of somebody else's instead of
> > working on your target system.
>
> You describe something slightly different. I probably used the term
> "test framework" too loosely. I'll just call it a "test tool". I
> certainly don't expect it to know my system. I implement all required
> knowledge of the system myself - it is embedded in test cases and HW
> simulation.
>
> What I am looking for is a tool that allows me to define individual test
> cases in a high-level language. The language can access our I/O devices
> (DLL-s or Active X components). It is real-time aware (can use delays,
> timeouts). Test cases can be grouped, run, paused, stopped. Tool outputs
> a formatted test report (usable in a review with managers).
>
> > Since there can be a wild variety of the different conditions and
> > situations, the test system should target only the relevant case or the
> > specific set of cases.
>
> I did not get the point. It's the test engineers job to design tests so
> relevant stuff is covered. Test system just executes them.
>
> > Making the universal test system =3D=3D=3D bloat, bugs
> > and limitations. The right way to do it is developing the test suite as
> > the common program in the general purpose programming language such as =
C++.
>
> A team in my company has been using NI TestStand and LabView for years.
> It is cumbersome and an overkill for our project. We have looked at
> TestComplete. That fills the requirements, but it's still too
> complicated. So I'm looking for viable alternatives.
>
> We have considered creating a custom test suite in Python or C++. It
> seems to be a good idea, but our development resources are quite
> limited. I would very much like to minimize the amount of work and not
> write it from scratch. Some elements - managing tests, running them and
> reporting results - are quite universal. Is there any effort to make a
> reusable library or framework for this purpose?

http://www.python.org/workshops/2002-02/papers/01/index.htm
http://www.codesourcery.com/qmtest

 I have never used it but it is free.
 Keep us posted if you try it out.

On Dec 29, 6:04=A0am, Tarmo Kuuse <tarmo.ku...@mail.ee> wrote:

also:

Functional Testing Tools, the Open-Source Wave
http://www.sdtimes.com/content/article.aspx?ArticleID=3D32583

http://www.opensourcetesting.org/

Marco wrote:
>> We have considered creating a custom test suite in Python or C++. It
>> seems to be a good idea, but our development resources are quite
>> limited. I would very much like to minimize the amount of work and not
>> write it from scratch. Some elements - managing tests, running them and
>> reporting results - are quite universal. Is there any effort to make a
>> reusable library or framework for this purpose?
> 
> http://www.python.org/workshops/2002-02/papers/01/index.htm
> http://www.codesourcery.com/qmtest
> 
>  I have never used it but it is free.
>  Keep us posted if you try it out.

Thank you for the link. At first glance it seems just what is needed. A 
closer look will have to wait for the time being. I'll peek at the other 
links you provided also.

Winter holidays are a quiet, peaceful time - good for dreaming and 
researching. As soon as they are over, "normal" life returns.

-- 
Kind regards,
Tarmo Kuuse