Help devising software tests

In article <1121374174.402532.73020@g44g2000cwa.googlegroups.com>,
	"Lanarcam" <lanarcam1@yahoo.fr> writes:
> 
> Black box testing is not testing individual C functions
> one by one. You must test functionalities.

.. and functionalities should be stated as requirements in the
specification. I think I'm beginning to understand now. The problem
I was facing is that my specification has some requirements stated
along with the functions they correspond to, but there are also
requirements which do not correspond directly to any single function.
So, the obvious approach of using the list of functions as the main
structure for the test document won't work here. I'll have to start
from the list of requirements.

> If your system exhibits a state, you must test all
> C functions in every accessible state.

OK. I could treat the state affecting a function's behavior as
if it were another "virtual" parameter to the function. Sounds good.

> This seems a lot of work but you can't avoid it if you
> need to prove the functionnal correctness.

Right (except that (please excuse my nitpicking...) functional
correctness cannot be *proven* by testing)

Thanks for your help

Arno

"Arno Nuehm" <arno@localhost> wrote in message 
news:42d77d27_2@news.arcor-ip.de...
> In article <1121374174.402532.73020@g44g2000cwa.googlegroups.com>,
> "Lanarcam" <lanarcam1@yahoo.fr> writes:
>>
<snip>
> Right (except that (please excuse my nitpicking...) functional
> correctness cannot be *proven* by testing)
>
> Thanks for your help
>
> Arno

You are correct.  You cannot prove correctness, but you must try to prove 
incorrectness.

We use this adage, "A developers job is to make their software work.  A 
testers job is to try to prove that it doesn't!"

Scott 

"Arno Nuehm" <arno@localhost> wrote in message 
news:db62to$cue$00$1@news.t-online.com...
> Hi Scott,
>
> In article <3jnbdmFquvd9U1@individual.net>,
> "Not Really Me" <scott@exoXYZtech.com> writes:
>>
<snip>
> If you could shed some light on the process that was used to define those
> tests for MicroC/OS-II, it would probably help me a lot. But I can
> see pretty well that you can't do that as it's probably the basis of
> your company's business.
>
> Thanks
>
> Arno

The response by Lanarcam says it pretty well.  Regardless that it is not a 
safety-critical app, I highly recommend that you get copies of the RTCA 
specs DO-178B and DO-248.  While not aimed directly at security, they do 
identify the steps that you will need to follow.

Simply the rule of thumb is "have a plan and test everything".
If your company doesn't have them, generate configuration management 
procedures and plans, software QA procs and plans, specs for requirements, 
designs and tests, test plans, and keep everything under source/document 
control.
Test the requirements - are they complete and correct.
Test the design - does it match the requirements.
Test the code - does it match the design.
Test the white box tests - do they do an adequate job, do they test all the 
low-level (design) requirments.
Test the build and test procedures - Can you repeat everything if necessary?
Test the black box tests - do they test every requirement?
Argh! Generate a tracability matrix.  This is a matrix that correlates the 
requirments with the design with the code with tests.
Oh, and good luck.  (Contact us if you need help/advise on any of the above)

Scott

Not Really Me wrote:
> "Arno Nuehm" <arno@localhost> wrote in message
> news:42d77d27_2@news.arcor-ip.de...
> > In article <1121374174.402532.73020@g44g2000cwa.googlegroups.com>,
> > "Lanarcam" <lanarcam1@yahoo.fr> writes:
> >>
> <snip>
> > Right (except that (please excuse my nitpicking...) functional
> > correctness cannot be *proven* by testing)
> >
> > Thanks for your help
> >
> > Arno
>
> You are correct.  You cannot prove correctness, but you must try to prove
> incorrectness.
>
> We use this adage, "A developers job is to make their software work.  A
> testers job is to try to prove that it doesn't!"

This is right, my use of the verb prove was sloppy;)

This leads to the nightmare of the conscientious programmer
who can never get a good sleep while his creation is
in the wild.

You stop testing sometimes because of the project schedule or
because you become short of funds. In this case you can
never be sure that all bugs have been removed.

When you write code that will be part of a certified
system you have at least the approval of the official
certification body.

A colleague said that a software is free of bugs only
when the customer has got rid of it;)

Not Really Me wrote:
> 
... snip ...
> 
> You are correct.  You cannot prove correctness, but you must try
> to prove incorrectness.
> 
> We use this adage, "A developers job is to make their software
> work.  A testers job is to try to prove that it doesn't!"

I consider that fundamentally flawed.  A developer should write
code that is obviously correct.  That means short routines that can
be described in a few sentences, with close type checking on
parameters.  Brinch Hansen was a master at this.  

This leaves the tester checking boundary conditions, and for proper
usage of those routines.

-- 
"If you want to post a followup via groups.google.com, don't use
 the broken "Reply" link at the bottom of the article.  Click on 
 "show options" at the top of the article, then click on the 
 "Reply" at the bottom of the article headers." - Keith Thompson

CBFalconer wrote:
> Not Really Me wrote:
> >
> ... snip ...
> >
> > You are correct.  You cannot prove correctness, but you must try
> > to prove incorrectness.
> >
> > We use this adage, "A developers job is to make their software
> > work.  A testers job is to try to prove that it doesn't!"
>
> I consider that fundamentally flawed.  A developer should write
> code that is obviously correct.  That means short routines that can
> be described in a few sentences, with close type checking on
> parameters.  Brinch Hansen was a master at this.

I tend to agree about the idea that testing is looking
for errors, not verifying the correctness. But this is
more of a philosophical debate and I should perhaps
stick to facts.

I agree that what you suggest is sound advice and if
programmers respected that there would be fewer bugs.

But even then you can never guarantee that you have
checked every conditions except in functions that are
trivial. You could do that but you would have to use
formal methods and they are not pratical in general.
I know some who used them in rail transport systems
but they were gurus.

Even if you have proved the correctness of functions,
when you assemble the whole thing you can find unexpected
errors. If the design is simple they should be caught
easily. But if you have several interrelated tasks
of some complexity, you can't assume you have no
bugs simply because you have written your code
carefully.

There are what are called real time bugs caused by
unexpected events that no one had suspected. And
the problem is when do you know you have corrected
the last one?

> This leaves the tester checking boundary conditions, and for proper
> usage of those routines.

This is true for linear calls of functions, but when
you have asynchronous execution this is not sufficient.
For instance when you have interrupts or preemptive tasks.

It is true however that what you suggest should be used
for safety critical systems. For these systems you have
generally a main loop that executes functions cyclically
and no interrupts. It is much easier to prove the
correctness of such designs.

But for other types of systems this is not always 
practical.

Lanarcam wrote:
> 
... snip ...
> 
> This is true for linear calls of functions, but when
> you have asynchronous execution this is not sufficient.
> For instance when you have interrupts or preemptive tasks.

Those can generally be characterized with loop invariants, or just
plain invariants.  A producer/consumer relationship is typical. 
The real bear is when you have to guarantee worst case delays.

> 
> It is true however that what you suggest should be used
> for safety critical systems. For these systems you have
> generally a main loop that executes functions cyclically
> and no interrupts. It is much easier to prove the
> correctness of such designs.

You may note I cited Per Brinch Hansen earlier.  He and his staff
could write a complete multiprocessing operating system and be
virtually convinced of correctness out of the gate.  He had (has) a
genius for this sort of organization and simplification.  Yet the
overall system can be highly complex and deal with many interrupts
and processes.

-- 
"If you want to post a followup via groups.google.com, don't use
 the broken "Reply" link at the bottom of the article.  Click on 
 "show options" at the top of the article, then click on the 
 "Reply" at the bottom of the article headers." - Keith Thompson