books for embedded software development

Hi everyone,

I just started to (re)design the software for an embedded application on
a very old DSP (ADSP21020 32bit floating point) and I was trying to look
for some good books on embedded software development since I wanted to
start it right from the beginning rather than chase it later on.

The application is a Star Tracker which is using a CCD aiming at stars
while the DSP processes and compresses the images for transmission to
ground.

Thanks,

Al

-- 
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

On Monday, December 12, 2011 3:01:12 PM UTC-5, Alessandro Basili wrote:
> ... I was trying to look
> for some good books on embedded software development ...

"Better Embedded System Software" - Philip Koopman
"Test-Driven Development for Embedded C" - Grenning

Two recent keepers.
Enjoy,
Best Regards, Dave

On 12/13/2011 12:06 AM, Anony Mous wrote:
> On 12/12/2011 3:01 PM, Alessandro Basili wrote:
>> I just started to (re)design the software for an embedded application on
>> a very old DSP (ADSP21020 32bit floating point) and I was trying to look
>> for some good books on embedded software development since I wanted to
>> start it right from the beginning rather than chase it later on.
[...]

> 
> Not specifically on the ADSP21020, but the best embedded software book
> that I've read is Practical UML Statecharts in C/C++ by Miro Samek.
> http://www.state-machine.com/psicc2/index.php
> 

That sounds good, I actually wanted to handle the whole code through
hierarchical state machines. My biggest concern is the Time Analysis, or
a strategy to keep the execution time under control. The application
should not be very demanding in terms of computation but I still have to
serve serial port commands and I wanted to avoid the usage of the
interrupts.

> Specific to the ADSP21020, you may be intersted in the following:
> http://www.bound-t.com/targets/sharc/index.html
> http://www.analog.com/en/processors-dsp/sharc/adsp-21020/products/code-examples/21xxx_applications_handbook_file_area/resources/fca.html
> 

Thanks a lot for these as well, indeed I was aware about those code
examples but I haven't had the time to have a look at them yet.

On 12/13/2011 12:06 AM, Anony Mous wrote:
> On 12/12/2011 3:01 PM, Alessandro Basili wrote:
>> I just started to (re)design the software for an embedded application on
>> a very old DSP (ADSP21020 32bit floating point) and I was trying to look
>> for some good books on embedded software development since I wanted to
>> start it right from the beginning rather than chase it later on.
[...]

> 
> Not specifically on the ADSP21020, but the best embedded software book
> that I've read is Practical UML Statecharts in C/C++ by Miro Samek.
> http://www.state-machine.com/psicc2/index.php
> 

That sounds good, I actually wanted to handle the whole code through
hierarchical state machines. My biggest concern is the Time Analysis, or
a strategy to keep the execution time under control. The application
should not be very demanding in terms of computation but I still have to
serve serial port commands and I wanted to avoid the usage of the
interrupts.

> Specific to the ADSP21020, you may be intersted in the following:
> http://www.bound-t.com/targets/sharc/index.html
> http://www.analog.com/en/processors-dsp/sharc/adsp-21020/products/code-examples/21xxx_applications_handbook_file_area/resources/fca.html
> 

Thanks a lot for these as well, indeed I was aware about those code
examples but I haven't had the time to have a look at them yet.

Dave Nadler wrote:

> On Monday, December 12, 2011 3:01:12 PM UTC-5, Alessandro Basili wrote:
>> ... I was trying to look
>> for some good books on embedded software development ...
> 
> "Better Embedded System Software" - Philip Koopman

Phil's book is an excellent suggestion. Really good for any development 
situation.

I would also add into the pile:
The Art of Designing Embedded Systems Jack Ganssle ISBN 0-7506-9869-1

Handbook of Walkthroughs, Inspections and Technical Reviews (Evaluating 
Programs, Projects, and Products. Daniel P. Freedman and Gerald M Weinberg 
ISBN 0-932633-19-6

I included the latter because the review aspect can become a project saviour 
if performed properly throughout the development.

-- 
********************************************************************
Paul E. Bennett...............<email://Paul_E.Bennett@topmail.co.uk>
Forth based HIDECS Consultancy
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-510979
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************

Hi Alessandro,

On 12/12/2011 1:01 PM, Alessandro Basili wrote:

> I just started to (re)design the software for an embedded application on
> a very old DSP (ADSP21020 32bit floating point) and I was trying to look
> for some good books on embedded software development since I wanted to
> start it right from the beginning rather than chase it later on.

Perhaps the most important and effective thing you can do
is to enumeate the reasons *why* you are redesigning  the
device (assuming at east one iteration exists already and has
been deployed, etc.).  E.g., component availability, feature
creep, etc.

Then, add to that list any "issues" that have been uncovered but
not yet (satisfactorily) addressed in the conceptual design of
the device.  E.g., any "mysterious/anomalous" behaviors that
may have been noticed (and possibly "resolved themselves" *or*
were handled by resetting the device).

IMO, you'll get more *practical* "return for this time invested
than rethinking an implementation methodology (which you might
do wrong)

> The application is a Star Tracker which is using a CCD aiming at stars
> while the DSP processes and compresses the images for transmission to
> ground.

Is the device space-based?  E.g., can you make significant improvements
in power consumption (or other resource limitations)?

On 12/13/2011 11:05 PM, Don Y wrote:
> Hi Alessandro,
> 
> On 12/12/2011 1:01 PM, Alessandro Basili wrote:
> 
>> I just started to (re)design the software for an embedded application on
>> a very old DSP (ADSP21020 32bit floating point) and I was trying to look
>> for some good books on embedded software development since I wanted to
>> start it right from the beginning rather than chase it later on.
> 
> Perhaps the most important and effective thing you can do
> is to enumeate the reasons *why* you are redesigning  the
> device (assuming at east one iteration exists already and has
> been deployed, etc.).  E.g., component availability, feature
> creep, etc.

the current software is:
1. not fulfilling the specification; the software is supposed to provide
compressed data for stellar fields and it simply does not.

2. "unstable"; after few hours of operation in non-compressed mode the
software hangs and a hardware reset is needed.

3. structureless; after a code review it is clear that debugging it
would be more costly rather then redesigning it from scratch.

4. full of logical flaws; synchronization problems are the most common
mistakes, but interrupt service routines are excessively and needlessly
long, essentially preventing any time analysis.

5. overly complicated in the commanding interface; the software tries to
handle a command queue with no control over the queue itself (queue
reset, queue status).

6. lacking control over the CCD; there's a register implemented in the
hardware which gives control over the CCD functionalities, but has been
ignored in the current implementation.

7. reinventing the wheel with basic functions; the C-runtime library
provided by AD is completely ignored and a long list of functions have
been implemented apparently for no reasons.

8. not utilizing the available bandwidth; there's a serial port through
which the DSP can write its data to an output buffer, but the bandwidth
available is reduced to essentially 256/1920 due to the handshake
protocol implemented (where on the other side no one is really
performing any hand shake). This limit poses a very hard constraint on
the science data, to the point where the information is not enough to
reconstruct pointing direction with the accuracy needed.

9. not maintained anymore; the previous software developer left and a
new team took over. We tried to recover from the ashes all that we could
from the previous work, but apparently a huge management flaw lost
control over the schedule and deliverables that we are now facing a
touch choice: where to start from?

10. not designed for testing; there are a lot of functions that are not
observable and there's no logging mechanism in the code for tracing
either. It's what I usually call "plug and pray" system.

I think there are other items that would call for a redesign, as lack of
documentation, lack of revision control system, lack of test campaigns,
lack of tools to work with the software, ...

> 
> Then, add to that list any "issues" that have been uncovered but
> not yet (satisfactorily) addressed in the conceptual design of
> the device.  E.g., any "mysterious/anomalous" behaviors that
> may have been noticed (and possibly "resolved themselves" *or*
> were handled by resetting the device).
> 
> IMO, you'll get more *practical* "return for this time invested
> than rethinking an implementation methodology (which you might
> do wrong)

I agree that redesign is not the solution, if you want is just a mean to
get it working. We normally follow an "iterative and incremental "
model, in order not to invest too much time in the "wrong design" and
leave room for adjustment along the way. Unfortunately the previous team
didn't have any guideline to follow and spent 99% in development and %1
in "hope it works". To their credit they didn't have any support from
any external test team and they inevitably fell short of feedback as well.

> 
>> The application is a Star Tracker which is using a CCD aiming at stars
>> while the DSP processes and compresses the images for transmission to
>> ground.
> 
> Is the device space-based?  E.g., can you make significant improvements
> in power consumption (or other resource limitations)?

It is space-based and it is currently flying. Actually the goal here is
to *have* a design, which seems to me the current software is lacking of
completely. Well I can imagine it's hard to explain how did we get to
this point and I also believe it would be interesting to analyze what
happened to got us to this point in order not to repeat the same
mistakes, but unfortunately the situation is what it is and something
has to be done if we ever want to correlate our photon reconstruction
capabilities with pointing information.

Actually I tried to get the chance to have some good reference to foster
discussion in the team as well. I believe that following some simple and
well described approach can enable us to bend it later on to our
particular needs (which might not be clear at this point).

Alessandro Basili wrote:

> On 12/13/2011 11:05 PM, Don Y wrote:
>> Hi Alessandro,
>> 
>> On 12/12/2011 1:01 PM, Alessandro Basili wrote:
>> 
>>> I just started to (re)design the software for an embedded application on
>>> a very old DSP (ADSP21020 32bit floating point) and I was trying to look
>>> for some good books on embedded software development since I wanted to
>>> start it right from the beginning rather than chase it later on.
>> 
>> Perhaps the most important and effective thing you can do
>> is to enumeate the reasons *why* you are redesigning  the
>> device (assuming at east one iteration exists already and has
>> been deployed, etc.).  E.g., component availability, feature
>> creep, etc.

Whichever route Alessandro takes (the start from scratch route or the 
improve current design route as Don suggested) the books will be good 
reminders of the proper conduct of the effort involved.
 
> the current software is:
> 1. not fulfilling the specification; the software is supposed to provide
> compressed data for stellar fields and it simply does not.

Start with a good and thorough review of the specification document. Take it 
apart and test every statement of requirement it contains. Vagueness or 
assumed facets should be transformed into clear, concise and testable 
requirements specifications. Do not start any design until this review is 
complete and you have a fully tested specification document. Up to 80% of 
the bugs for a system can be eliminated at this point by robust reviewing.
 
The point about writing requirements specifications is that they have to be 
clear, concise and testable statements of what is required in the system by 
way of functionality, interfaces, performance and maintainability. The 
specification document should be free of assumptions but if assumptions are 
necessary these should be very clearly identified as such and the basis for 
the assumptions clearly described. However, it would be better to eliminate 
them altogether.

> Actually I tried to get the chance to have some good reference to foster
> discussion in the team as well. I believe that following some simple and
> well described approach can enable us to bend it later on to our
> particular needs (which might not be clear at this point).

Phil Koopman's book and the Freidman and Weinberg book on reviews will stand 
you in good stead for a lot of what you have to resolve. There are a number 
of others I could reccommend but you have to get some work done. Make sure 
that your management read Phils book. In my review of this book I stated 
that it should be "impossibly open at all chapters at once on the developers 
desk". You will find much in it that will help you through your current 
problems.

In addition to the books, you should also spend a bit of time to visit the 
design of your development process. Know what documentation you will need to 
build as evidence that the development was performed correctly. Do not 
forget to build in a mechanism for management of the innevitable changes.

Finally, you spoke of scrapping the current software and starting again. 
More often than not this can be a good thing to do. However, be sure not to 
make the same mistakes the second time around.

-- 
********************************************************************
Paul E. Bennett...............<email://Paul_E.Bennett@topmail.co.uk>
Forth based HIDECS Consultancy
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-510979
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************

On 12/14/2011 05:47 PM, Alessandro Basili wrote:

>>> The application is a Star Tracker which is using a CCD aiming at stars
>>> while the DSP processes and compresses the images for transmission to
>>> ground.
>>
>> Is the device space-based?  E.g., can you make significant improvements
>> in power consumption (or other resource limitations)?
>
> It is space-based and it is currently flying. Actually the goal here is
> to *have* a design, which seems to me the current software is lacking of
> completely. Well I can imagine it's hard to explain how did we get to
> this point and I also believe it would be interesting to analyze what
> happened to got us to this point in order not to repeat the same
> mistakes, but unfortunately the situation is what it is and something
> has to be done if we ever want to correlate our photon reconstruction
> capabilities with pointing information.
>

I'm a little curious about this. Are you planning to uplink new software 
to the already flying hardware, or redesigning for the next attempt?
If it's the latter, I'd maybe reconsider the hardware as well, though 
you may not have time for that.

Was any ground testing done previously?
I'm just wondering if the radiation environment might be a contributing 
factor to your failure.
I've seen/heard lots of success stories of COTS parts working just fine 
for the most part on cube sats and such, even without much in the way of 
rad tolerance measures in the design, but you probably have to do 
frequent full resets.

Hi Allessandro,

On 12/14/2011 9:47 AM, Alessandro Basili wrote:

>>> I just started to (re)design the software for an embedded application on
>>> a very old DSP (ADSP21020 32bit floating point) and I was trying to look
>>> for some good books on embedded software development since I wanted to
>>> start it right from the beginning rather than chase it later on.
>>
>> Perhaps the most important and effective thing you can do
>> is to enumeate the reasons *why* you are redesigning  the
>> device (assuming at east one iteration exists already and has
>> been deployed, etc.).  E.g., component availability, feature
>> creep, etc.
>
> the current software is:

Excellent! You know where/why you're starting!  (Unfortunately,
it looks like you've inherited a real *mess*  :< )

> 1. not fulfilling the specification; the software is supposed to provide
> compressed data for stellar fields and it simply does not.

Is there a *real* specification?  Or, just a general "goal"?
I.e., is there anything to test against?

> 2. "unstable"; after few hours of operation in non-compressed mode the
> software hangs and a hardware reset is needed.

(speaking with zero knowledge of the application) This is suggestive
of memory (management) problems, counter overflows or "deadly embrace".
(I mention these simply to get a feel for what "services" your
application can benefit from)

> 3. structureless; after a code review it is clear that debugging it
> would be more costly rather then redesigning it from scratch.

(sigh)  That is probably the case.  Even (just a) disciplined developer
would impart *some* structure to his/her code.  And, from your other
comments, it seems like this was an "ad hoc" team effort so any
attempt at structure was lost in the noise.

> 4. full of logical flaws; synchronization problems are the most common
> mistakes, but interrupt service routines are excessively and needlessly
> long, essentially preventing any time analysis.

Synchronization problems can be avoided with a disciplined design.
This would also help identify cases of priority inversion.

Long ISRs are usually a consequence of a lack of structure.  No
mechanism to systematically pass data/events between foreground
and background so the "processing" finds its way into the ISR.

Logical flaws should have been easily identifiable from the
specification (assuming it is serving ts proper role).

> 5. overly complicated in the commanding interface; the software tries to
> handle a command queue with no control over the queue itself (queue
> reset, queue status).

This also seems like it should have been apparent in the specification
(else how does the commanding agent know what rules *it* must live by??)

> 6. lacking control over the CCD; there's a register implemented in the
> hardware which gives control over the CCD functionalities, but has been
> ignored in the current implementation.

<frown>  Sorry, I don't understand the role it plays (in the hardware
or system itself).  On the surface, it seems to suggest:  "How the
hell could the device *function* if it has no control over its
transducer(s)?"

> 7. reinventing the wheel with basic functions; the C-runtime library
> provided by AD is completely ignored and a long list of functions have
> been implemented apparently for no reasons.

Are you *sure* that the library is disused OUT OF IGNORANCE?  (from
your other comments, this seems like it is probably the case)  But,
be aware that sometimes vendor supplied libraries are provided as
check-off items and often not suited to a particular environment
(e.g., reentrancy).

> 8. not utilizing the available bandwidth; there's a serial port through
> which the DSP can write its data to an output buffer, but the bandwidth
> available is reduced to essentially 256/1920 due to the handshake
> protocol implemented (where on the other side no one is really
> performing any hand shake).

Huh?  I assume you mean the "other party" has no *need* for the
handshaking (so it is wasted overhead)?  Is the "handshaking"
intended as a pacing mechanism or as an acknowledgement/verification
mechanism?

> This limit poses a very hard constraint on
> the science data, to the point where the information is not enough to
> reconstruct pointing direction with the accuracy needed.

Meaning you can't get data often enough to point the satellite
(or your instrument therein) *at* the correct target?

> 9. not maintained anymore; the previous software developer left and a
> new team took over. We tried to recover from the ashes all that we could
> from the previous work, but apparently a huge management flaw lost
> control over the schedule and deliverables that we are now facing a
> touch choice: where to start from?

Understood.  This is actually a common event.  Especially if for
projects that aren't particularly "well designed/implemented"
(its too easy/tempting to quietly "slip away" when no one is watching)

> 10. not designed for testing; there are a lot of functions that are not
> observable and there's no logging mechanism in the code for tracing
> either. It's what I usually call "plug and pray" system.
>
> I think there are other items that would call for a redesign, as lack of
> documentation, lack of revision control system, lack of test campaigns,
> lack of tools to work with the software, ...

All these are "strongly desired" -- especially with the stakes as they
are (I suspect it is very difficult/costly to get instruments flying!)

>> Then, add to that list any "issues" that have been uncovered but
>> not yet (satisfactorily) addressed in the conceptual design of
>> the device.  E.g., any "mysterious/anomalous" behaviors that
>> may have been noticed (and possibly "resolved themselves" *or*
>> were handled by resetting the device).
>>
>> IMO, you'll get more *practical* "return for this time invested
>> than rethinking an implementation methodology (which you might
>> do wrong)
>
> I agree that redesign is not the solution, if you want is just a mean to
> get it working. We normally follow an "iterative and incremental "
> model, in order not to invest too much time in the "wrong design" and
> leave room for adjustment along the way.

I tend to favor heavily front-loaded processes -- putting lots of
effort into nailing down all the details in a specification which
can then be followed, almost blindly.  But this requires folks
who are good at challenging assumptions to be able to foresee
the things that can go wrong -- and fortify the specification
against them.  Frankly, I don't know how else to develop especially
in an environment where you have no physical control over the
"what if's"  (what if the spacecraft is pointed the wrong way?
what if communications are interrupted at this point?  etc.)

In addition to serving as a map that "implementors" (coders?) can
"just follow", a good spec gives you a contract that you can
test against.  And, for teams large enough to support a division
of functions, lets the "test team" start designing test platforms
in which the *desired* functionality can be verified as well as
*stressed* in ways that might not have been imagined.  This can
speed up deployment (if all goes well) and/or bring problems in
the design (or staff!) out into the open, early enough that
you can address them ("Sheesh!  Bob writes crappy code!  Maybe
we should think of having Tom take over some of his duties?"
or "Ooops!  This compression algorithm takes far too long to
execute.  Perhaps we should aim for lower compression rates
in favor of speedier results?")

> Unfortunately the previous team
> didn't have any guideline to follow and spent 99% in development and %1
> in "hope it works". To their credit they didn't have any support from
> any external test team and they inevitably fell short of feedback as well.
>
>>> The application is a Star Tracker which is using a CCD aiming at stars
>>> while the DSP processes and compresses the images for transmission to
>>> ground.
>>
>> Is the device space-based?  E.g., can you make significant improvements
>> in power consumption (or other resource limitations)?
>
> It is space-based and it is currently flying. Actually the goal here is
> to *have* a design, which seems to me the current software is lacking of
> completely. Well I can imagine it's hard to explain how did we get to
> this point and I also believe it would be interesting to analyze what
> happened to got us to this point in order not to repeat the same

Worthwhile for *management's* use -- but doesnt do enough to sort
out the mess in front of you.

> mistakes, but unfortunately the situation is what it is and something
> has to be done if we ever want to correlate our photon reconstruction
> capabilities with pointing information.
>
> Actually I tried to get the chance to have some good reference to foster

That can be a good approach -- depending on the dynamics of
your particular team.  What you want to avoid is the distraction
of people focussing on "arguing with the examples/guidelines"
instead of learning from them and modifying them to fit *your*
needs.

[think about how "coding guidelines" end up diverting energy
into arguing about silly details -- instead of recognizing the
*need* for SOME SORT OF 'standard']

> discussion in the team as well. I believe that following some simple and
> well described approach can enable us to bend it later on to our
> particular needs (which might not be clear at this point).

I like to approach designs by enumerating the "things it must do"
from a functional perspective.  I.e., "activities":  control the
transducer, capture data from the transducer, process that data,
transmit that data... (an oversimplification of your device -- but
I don't know enough about it to comment in detail).  Note these
are all verbs -- active.

Then, identify the communications between these "activities".
And, the resource requirements of each.

[this is all informal, "shirt-cuff" at this point]

This gives me an idea of how finely I can partition the design.
The resource requirements tell me what constraints exist in
terms of how much can happen concurrently.  E.g., do I have
enough memory/CPU to *collect* (new) data while processing
(previous) data AND transmitting (old) data?  If not, what
value judgement can I make to best use the resources that I *do*
have to maximize the functionality of the device?  (e.g., if
I have lots of memory but very little CPU, I might prefer
to gather as much raw data *now* -- while some observable event
is happening -- and worry about processing and transmitting it
*later*)

This gives my first rough partitioning of tasks/threads/processes
and "memory regions".  It also shows where the data is flowing and
any other communication paths (even those that are *implicit*).
Synchronization needs then become obvious. And, performance
bottlenecks can be evaluated with an eye towards how the
design can be changed to improve that.

E.g., if an (earth-based) command station has to *review*/analyze
data from the device before it can reposition/target it, then
the time from collection thru processing and transmission
is a critical path limiting how quickly the *overall* system
(satellite plus ground control) can react.  If the times
associated with any of those tasks are long-ish, you can
rethink those aspects of the design with an eye towards
short-cutting them.  So, perhaps providing a mechanism
to transmit *unprocessed* data (if the processing activity
was the bottleneck) or collect over an abbreviated time window
(if the collection activity was the bottleneck).

Once the activities and communications are identified, I
look to see what services I want from an OS -- and the
resources available for it.  IMO, choice of services goes
a *LONG* way to imposing structure on an application!
And, it acts as an implicit "language" by which the
application can communicate with other developers about
what its doing at any particular place.

E.g., do you need fat pipes for your communications?
Are you better off passing pointers to memory regions
to reduce bcopy()'s?  Can you tolerate shared instances
of data?  Or do you *need* private instances?  How finely
must you resolve time?  What are the longest intervals
that you need be concerned with?

I would also set aside some resources for (one or more)
"black boxes".  These can be invaluable in post-mortems.
Ideally, if you have a ground-based prototype that you
can modify, consider adding additional memory to it
(even if it is "secondary" memory) for these black boxes.
Having *lots* of data can really help identify what is
happening when things turn ugly.  (much easier than
trying to reproduce a particular experiment -- which
might not be possible! -- so you can reinstrument it!)
Litter your code with invariant assertions so you see
every "can't happen" when it *does* happen! :>

Finally, testing is critical.  The goal *should* be to
"break" the device. Really!  And then look at the
conditions in which it did "break" and see how those
relate to your actual deployment ("Well, the system
crapped out when the 40KHz signal from the switching
power supply was coupled to the NMI pin...")

Mark Borgerson (sp?) would be a good contact -- he
posts here often (and, IIRC, designs data collection
devices for submersible deployments... similar issues
to what you face, I suspect.)

Paul Bennett (I see a post from him elsewhere in this
thread) will be a good resource regarding "getting it
right, on paper, *first*" -- since mistakes are
probably costly for you (in terms of access to the
actual device as well a lost opportunity for your
"customers")

There are others who can offer good advice, as well.
The actual nature of your project is what prompted
me to suggest these two folks...