Reply by Szabolcs Ferenczi●July 10, 20072007-07-10
On Jul 9, 3:35 pm, Colin Paul Gloster <Colin_Paul_Glos...@ACM.org>
wrote:
> How do you test production code?
I have shown an example for production code test in that
demonstration. Although, I have written the test before writing the
production code. (Note that it is not finished yet.)
> Also, what penalties can be expected in the Globally Asysnchronous
> Locally Synchronous (GALS) paradigm?
I never heard about that paradigm but I have looked it up. It looks to
me that it applies to hardware blocks that have different clock
sources. So, I would say the idea is the same: As far as I could
conclude from the stories of the hardware guys, in the GALS paradigm,
the clocks are also synchronised during the test. Just as I have
suggested: make the system deterministic for the test.
Best Regards,
Szabolcs
Reply by Colin Paul Gloster●July 9, 20072007-07-09
In news:1183807930.850704.72520@n60g2000hse.googlegroups.com
timestamped Sat, 07 Jul 2007 11:32:10 -0000, Szabolcs Ferenczi
<szabolcs.ferenczi@gmail.com> posted:
"On Jul 7, 11:53 am, cosmocam...@yahoo.com wrote:
[..]
> The second question is: what features would you like to see in a
> testing tool for concurrent software?
Reproducible behaviour, i.e. eliminating non-determinacy for the test.
You can find some initial effort in this from me on another discussion
forum. It is not completed yet, in fact there is a known bug left
behind, but in turn I am going to continue it by applying some form of
mocking at the testing of the scheduling.
TDD for multi-threaded programs
http://tech.groups.yahoo.com/group/testdrivendevelopment/message/24115
(You might check a cached version where the indentation is preserved
for the code fragments, e.g.
http://www.nabble.com/TDD-for-multi-threaded-programs-t3627394.html)
Best Regards,
Szabolcs"
Szabolcs,
This seems to be good. I do have a reservation however.
From
HTTP://WWW.Nabble.com/TDD-for-multi-threaded-programs-t3627394.html
:"[..]
Step 1: A thread should be suspended on an empty buffer
@Test
public void parSuspendOnEmptyBuffer() throws InterruptedException {
Thread t1 = new Thread() {
public void run() {
q.get();
}};
t1.start();
Thread.sleep(DELAY);
assertEquals("Buffer is empty and thread is waiting on get()",
Thread.State.valueOf("WAITING"), t1.getState());
t1.interrupt();
t1.join();
}
Note that the delay is inserted for the test to allow some time for
the tester thread to stabilize. This way we can make the test
deterministic.
[..]
3. I don't like to see `Thread.sleep(DELAY);' in production code
[..]
His idea was to allow enough time for the processes to stabilise by
using a virtual clock on which the test processes could synchronise.
So we know that at each tick of the virtual clock which state we are
at. I used explicit delays for simplicity. [..]
[..]"
How do you test production code?
Also, what penalties can be expected in the Globally Asysnchronous
Locally Synchronous (GALS) paradigm?
Regards,
Colin Paul Gloster
Reply by Colin Paul Gloster●July 9, 20072007-07-09
In a post timestamped Sat, 07 Jul 2007 09:53:44 -0000,
cosmocampus@yahoo.com posted:
"[..]
The second question is: what features would you like to see in a
testing tool for concurrent software? My understanding is that
concurrency testing in the real-world setting is really stress
testing, and that there are no good notions of coverage. [..]"
Apparently some properties of Plan9 were verified with Spin, but a
post timestamped 22 May 2007 08:56:24 -0700 in
news:comp.specification.misc ( Message-ID:
<1179849383.348588.301680@o5g2000hsb.googlegroups.com>
) is not encouraging:
"Hello.
A colleague and I are trying to use SPIN to find
all the potential error paths in a model. As near as
we can tell, this is hard to do without getting lots
and lots of redundant results. We have the following
never claim:
never { /* <> p */
T0_init:
if
:: (1) -> goto T0_init
:: (p) -> goto accept_all
fi;
accept_all:
skip
}
We use "spin -a foo.pml" followed by
"gcc -o pan -DSAFETY pan.c", and then
run "pan -c -e". A simple example that
should have two unique error paths gives
tens of thousands of trails. I understand
that we will see some redundancy because, e.g.,
some process will change state after the
condition p has been violated, but before this
is detected, but this seems completely out
of hand. Are we missing something? Better
yet, is there a switch that tells spin/pan to
do what we want?
Thanks.
--
Durward McDonell
durward.mcdonell@jhuapl.edu
"
Regards,
Colin Paul Gloster
P.S. Good luck. You will need it. You may also wish to try the
comp.parallel.* newsgroups.
Reply by Tom Lucas●July 9, 20072007-07-09
<cosmocampus@yahoo.com> wrote in message
news:1183802024.092179.181690@n2g2000hse.googlegroups.com...
> Dear all,
>
> I am an academic researcher interested of late in testing, debugging
> and static analysis of concurrent software. Two of the directions I am
> interested in are: (1) developing static techniques to prove that a
> certain piece of software is free of, say, deadlocks and data races,
> and (2) developing better testing tools for concurrent code, e.g,
> understanding what the right coverage criteria should be. It is easy
> to lose sight of the forest and get too occupied with trees while
> defining research problems, so I thought I would talk to some people
> who are really on the frontlines of concurrent code and seek some
> advice about what are the real problems that you would like to see
> solved.
>
> The first question is rather specific: can you tell me of a piece of
> software that is:
> a) written by fairly competent people--- i.e., not just a case of bad
> programming;
> b) highly concurrent and uses complicated concurrency constructs. this
> is the most important requirement;
> c) buggy, or makes non-standard assumptions about the platform (of
> course, there may be bugs that are unknown);
> d) not very large in size, as I would like to start by trying to prove
> properties of the code by hand.
> e) ideally but not necessarily, has recursive control flow in some
> parts.
>
> I have looked at some embedded software and some web servers, but have
> not yet found anything that somes even close to fitting these
> criteria. Any thoughts?
>
> The second question is: what features would you like to see in a
> testing tool for concurrent software? My understanding is that
> concurrency testing in the real-world setting is really stress
> testing, and that there are no good notions of coverage. Would you
> agree with this statement and that things need to change? Regarding
> bugs or security flaws that you have actually found, what categories
> would you say are the hardest or need tool support? Timing errors?
> Data races? Deadlocks? Is there a repository of concurrency error
> reports somewhere that I can look at to understand the issues?
>
> Best regards,
> Swar
I would check out what PolySpace have been doing. They have pretty
advanced static and run-time analysis tools and are likely to have
looked at the sort of things you are asking about.
Whilst their tools are too expensive for the size of my projects, they
did give away the best freebies at the Embedded Systems Show and so I am
always happy to give them a plug!
Reply by Paul E. Bennett●July 8, 20072007-07-08
cosmocampus@yahoo.com wrote:
> Dear all,
>
> I am an academic researcher interested of late in testing, debugging
> and static analysis of concurrent software. Two of the directions I am
> interested in are: (1) developing static techniques to prove that a
> certain piece of software is free of, say, deadlocks and data races,
> and (2) developing better testing tools for concurrent code, e.g,
> understanding what the right coverage criteria should be. It is easy
> to lose sight of the forest and get too occupied with trees while
> defining research problems, so I thought I would talk to some people
> who are really on the frontlines of concurrent code and seek some
> advice about what are the real problems that you would like to see
> solved.
>
> The first question is rather specific: can you tell me of a piece of
> software that is:
> a) written by fairly competent people--- i.e., not just a case of bad
> programming;
> b) highly concurrent and uses complicated concurrency constructs. this
> is the most important requirement;
> c) buggy, or makes non-standard assumptions about the platform (of
> course, there may be bugs that are unknown);
> d) not very large in size, as I would like to start by trying to prove
> properties of the code by hand.
> e) ideally but not necessarily, has recursive control flow in some
> parts.
>
> I have looked at some embedded software and some web servers, but have
> not yet found anything that somes even close to fitting these
> criteria. Any thoughts?
>
> The second question is: what features would you like to see in a
> testing tool for concurrent software? My understanding is that
> concurrency testing in the real-world setting is really stress
> testing, and that there are no good notions of coverage. Would you
> agree with this statement and that things need to change? Regarding
> bugs or security flaws that you have actually found, what categories
> would you say are the hardest or need tool support? Timing errors?
> Data races? Deadlocks? Is there a repository of concurrency error
> reports somewhere that I can look at to understand the issues?
>
> Best regards,
> Swar
Are you considering concurrency issues on a single processor or concurrency
issues across multiple processors?
For the former, there would be plenty to look at in some of the embedded
control projects that deal with lots of individual interface.
For the latter, there may not be much about, but the shared computing
resource projects may be best to look at in this category.
--
********************************************************************
Paul E. Bennett ....................<email://peb@amleth.demon.co.uk>
Forth based HIDECS Consultancy .....<http://www.amleth.demon.co.uk/>
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-811095
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************
Reply by CBFalconer●July 7, 20072007-07-07
Marcel M�ller wrote:
>
... snip ...
> I do not know any /common/ language the really supports threads
> beyond the library level. Of course the programmer still will have
> the job to put semantically dependant objects into one logical
> object. But this is close to OOP concepts anyway.
> The first question is rather specific: can you tell me of a piece of
> software that is:
> a) written by fairly competent people--- i.e., not just a case of bad
> programming;
> b) highly concurrent and uses complicated concurrency constructs. this
> is the most important requirement;
> c) buggy, or makes non-standard assumptions about the platform (of
> course, there may be bugs that are unknown);
> d) not very large in size, as I would like to start by trying to prove
> properties of the code by hand.
> e) ideally but not necessarily, has recursive control flow in some
> parts.
>
> I have looked at some embedded software and some web servers, but have
> not yet found anything that somes even close to fitting these
> criteria. Any thoughts?
Maybe you can look for process controlling or measurement controlling
software. If the setup to control is complicated enough then rather
tricky concurrency issues are not that uncommon. There are many
asynchronuous events if component protection, user interaction and maybe
unpredictable measurement results affect the control flow. Real time
software for nuclear physics experiments are sometimes in this range.
Of course, it is unlikely the you will find a real world application
which will also fit to your condition d).
> The second question is: what features would you like to see in a
> testing tool for concurrent software? My understanding is that
> concurrency testing in the real-world setting is really stress
> testing,
mostly true.
> and that there are no good notions of coverage.
Well, in theory there are ways to describe the coverage, but nobody
wants to know the number since it will be close to zero. In fact yo have
to run the code with any possible thred swichtching which is of course
not possible. And if it would be possible than many of the the testcases
are that unlikely that they likely will never happen while our universe
exists.
I think the coverage in terms it is used for convetional unit testing is
not the significant value.
> Would you
> agree with this statement and that things need to change?
I would wonder if this changes this way. In fact the times where it was
possible to prove that a software works is gone a long time ago. We
should better think on how we deal with software that is /not/ free of
faults. This is much closer to the reality and not that susceptible to
systematic errors of all kind.
Of course, this will not remove the need for well written software, but
it may give another point of view to bugs in general. Not their
existance is a problem but the probability of their occurence.
> Regarding
> bugs or security flaws that you have actually found, what categories
> would you say are the hardest or need tool support? Timing errors?
> Data races? Deadlocks?
From my experience especially including code written from other people
data races are the most common threading bugs. Maybe because deadlocks
are very obvious, once they occur. In contrast wrong results are often
taken as they are without thinking twice. "If the pocket calculator say
2+2=5 then this must be true."
Other sources of bugs are that the programmers somtimes do not realize
the locality of their objects. Things like thread local data, session
data and so on are intermixed if the dependencies ares not that obvious.
More helpful than (complicated) tests or at leat an important part would
be a threading support at language level. If the association of an
object to a thread or a mutex is (implicit?) part of the declaration,
compiletime checks would be possible that could check for some of the
most common threading issues. Debug versions of the runtime environment
could verify that the objects that are read or written to are not
potentially accessable by other readers or writers in the meanwhile in
an unallowed way. At least for high level languages like java this
should be possible.
I do not know any /common/ language the really supports threads beyond
the library level. Of course the programmer still will have the job to
put semantically dependant objects into one logical object. But this is
close to OOP concepts anyway.
Marcel
Reply by Szabolcs Ferenczi●July 7, 20072007-07-07
On Jul 7, 11:53 am, cosmocam...@yahoo.com wrote:
> The first question is rather specific: can you tell me of a piece of
> software that is:
> a) written by fairly competent people--- i.e., not just a case of bad
> programming;
See the book: The Architecture of Concurrent Programs by Per Brinch
Hansen (1977). He presents there three concurrent programming example.
It is written and documented by a fairly competent person.
> The second question is: what features would you like to see in a
> testing tool for concurrent software?
Reproducible behaviour, i.e. eliminating non-determinacy for the test.
You can find some initial effort in this from me on another discussion
forum. It is not completed yet, in fact there is a known bug left
behind, but in turn I am going to continue it by applying some form of
mocking at the testing of the scheduling.
TDD for multi-threaded programs
http://tech.groups.yahoo.com/group/testdrivendevelopment/message/24115
(You might check a cached version where the indentation is preserved
for the code fragments, e.g.
http://www.nabble.com/TDD-for-multi-threaded-programs-t3627394.html)
Best Regards,
Szabolcs
Reply by ●July 7, 20072007-07-07
Dear all,
I am an academic researcher interested of late in testing, debugging
and static analysis of concurrent software. Two of the directions I am
interested in are: (1) developing static techniques to prove that a
certain piece of software is free of, say, deadlocks and data races,
and (2) developing better testing tools for concurrent code, e.g,
understanding what the right coverage criteria should be. It is easy
to lose sight of the forest and get too occupied with trees while
defining research problems, so I thought I would talk to some people
who are really on the frontlines of concurrent code and seek some
advice about what are the real problems that you would like to see
solved.
The first question is rather specific: can you tell me of a piece of
software that is:
a) written by fairly competent people--- i.e., not just a case of bad
programming;
b) highly concurrent and uses complicated concurrency constructs. this
is the most important requirement;
c) buggy, or makes non-standard assumptions about the platform (of
course, there may be bugs that are unknown);
d) not very large in size, as I would like to start by trying to prove
properties of the code by hand.
e) ideally but not necessarily, has recursive control flow in some
parts.
I have looked at some embedded software and some web servers, but have
not yet found anything that somes even close to fitting these
criteria. Any thoughts?
The second question is: what features would you like to see in a
testing tool for concurrent software? My understanding is that
concurrency testing in the real-world setting is really stress
testing, and that there are no good notions of coverage. Would you
agree with this statement and that things need to change? Regarding
bugs or security flaws that you have actually found, what categories
would you say are the hardest or need tool support? Timing errors?
Data races? Deadlocks? Is there a repository of concurrency error
reports somewhere that I can look at to understand the issues?
Best regards,
Swar