Help me Plz!Debugging RAM version of build,produces rt output,Bin version does not!

Gurus,
I have worked in Lots of RTOS projects for consumer electronics
applications.I have encountered the following problem in couple of
instances and till now not able to track the reason behind it,because
each time my solution varies.Given below is my brief of the problem:

We have a consumer electronics appliance in which when the Target boots
up and puts a logo screen,when the user presses a key in the remote,the
target crashes.This happens in the ROM version of build.
When we debug it in RAM version,it never happens.I am clueless what
could be possible reason for this.
My little knowledge of RTOS says that this should be due to some timing
implications which when we step through debugger in RAM version makes
it work and since ROM version is non debuggable,we dont get required
delay for it to work.Is this understanding correct?
I am wondering how experts around here have solved these sorts of
problems where a RAM version when debugged works and ROM version of
same software fails!

In another case I encountered similar problem and traced the route
cause for an first enum parameter which was initialised to 1 instead of
zero and when we set the initial param to zero,it worked.I was clue
less how this made the Debugged RAM version work and fail in ROM
version.

It would be helpful if anyone can share their experience on how they
tackle such problems and whats the best way to debug such problems.
Advanced thanks to all your replys and looking farward for your pearls
of wisdom,
Regards,
s.subbarayan

[F'up2 cut down --- neglected by OP]

In comp.arch.embedded ssubbarayan <ssubba@gmail.com> wrote:

> We have a consumer electronics appliance in which when the Target boots
> up and puts a logo screen,when the user presses a key in the remote,the
> target crashes.This happens in the ROM version of build.
> When we debug it in RAM version,it never happens.I am clueless what
> could be possible reason for this.

Given the minimal amount of information you give, it's pretty much
guaranteed that nobody else can have more of a clue than you do.

[...]
> since ROM version is non debuggable,

And why would that be?  There's no particularly strict reason ROM
builds of a program should be non-debuggable.  They may well be
*harder* to debug than RAM builds, because some of the usual
techniques (e.g. run-time code modifications) aren't immediately
applicable to ROM programs.  But it's not generally impossible.

Or, to put it more provocatively: if your project and CPU is fat
enough that investing in an RTOS makes sense, how come you don't have
JTAG debugging?

And how is that ROM build actually different from the RAM build?  The
one special case you mentioned hints that you didn't understand your
tools well enough to turn on all needed options for them to support a
feature of the standardized language like static initializers.

-- 
Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de)
Even if all the snow were burnt, ashes would remain.

On 15 Dec, in article
     <1134708837.542201.170420@o13g2000cwo.googlegroups.com>
     ssubba@gmail.com "ssubbarayan" wrote:
>Gurus,
>I have worked in Lots of RTOS projects for consumer electronics
>applications.I have encountered the following problem in couple of
>instances and till now not able to track the reason behind it,because
>each time my solution varies.Given below is my brief of the problem:
>
>We have a consumer electronics appliance in which when the Target boots
>up and puts a logo screen,when the user presses a key in the remote,the
>target crashes.This happens in the ROM version of build.
>When we debug it in RAM version,it never happens.I am clueless what
>could be possible reason for this.
>My little knowledge of RTOS says that this should be due to some timing
>implications which when we step through debugger in RAM version makes
>it work and since ROM version is non debuggable,we dont get required
>delay for it to work.Is this understanding correct?

Probably, or even some variable that should be in RAM actually being in
ROM or not waiting for hardware correctly, not intialising ports correctly
(including serial ones), that happens to work in RAM versions.

RAM based debugger does things that are not done without debugger.
 
>I am wondering how experts around here have solved these sorts of
>problems where a RAM version when debugged works and ROM version of
>same software fails!

Use a scope/logic analyser to see what happens when, or toggle LED, or
even send messages to a serial port.

>In another case I encountered similar problem and traced the route
>cause for an first enum parameter which was initialised to 1 instead of
>zero and when we set the initial param to zero,it worked.I was clue
>less how this made the Debugged RAM version work and fail in ROM
>version.

Sounds more like you have RAM based variables that your code assumes are
zeroed, but your startup code does not initialise them to zero. This could
easily be the difference between your RAM based debugger and ROM version.

>It would be helpful if anyone can share their experience on how they
>tackle such problems and whats the best way to debug such problems.
>Advanced thanks to all your replys and looking farward for your pearls
>of wisdom,

Your scenarios suggest that you have spent time in hosted Application
Programming and transferred to embedded programming, without understanding
the differences.

Without knowing if these are all on the same RTOS, or host processor
anything other than the above general hand waving.
 
>Regards,
>s.subbarayan
>
>

-- 
Paul Carpenter          | paul@pcserviceselectronics.co.uk
<http://www.pcserviceselectronics.co.uk/>    PC Services
<http://www.gnuh8.org.uk/>              GNU H8 & mailing list info
<http://www.badweb.org.uk/>             For those web sites you hate

Hans-Bernhard Broeker wrote:
> [F'up2 cut down --- neglected by OP]
>
> In comp.arch.embedded ssubbarayan <ssubba@gmail.com> wrote:
>
> > We have a consumer electronics appliance in which when the Target boots
> > up and puts a logo screen,when the user presses a key in the remote,the
> > target crashes.This happens in the ROM version of build.
> > When we debug it in RAM version,it never happens.I am clueless what
> > could be possible reason for this.
>
> Given the minimal amount of information you give, it's pretty much
> guaranteed that nobody else can have more of a clue than you do.
>
> [...]
> > since ROM version is non debuggable,
>
> And why would that be?  There's no particularly strict reason ROM
> builds of a program should be non-debuggable.  They may well be
> *harder* to debug than RAM builds, because some of the usual
> techniques (e.g. run-time code modifications) aren't immediately
> applicable to ROM programs.  But it's not generally impossible.
>
> Or, to put it more provocatively: if your project and CPU is fat
> enough that investing in an RTOS makes sense, how come you don't have
> JTAG debugging?
>
> And how is that ROM build actually different from the RAM build?  The
> one special case you mentioned hints that you didn't understand your
> tools well enough to turn on all needed options for them to support a
> feature of the standardized language like static initializers.
>
> --
> Hans-Bernhard Broeker (broeker@physik.rwth-aachen.de)
> Even if all the snow were burnt, ashes would remain.

Hans,
Thanks for your reply.I guess I need to elaborate you to make it more
understandable.Its not non debuggable as u mentioned and I agree ROM
debuggers are available in fact.But in my company we have not procured
one and since we are at fag end of project,my clients are against
budjetting it now.Ok heres what it happens:
Our target Board boots up and puts a screen like this:
Wait a moment please.... along with Logo of my company.When we press
any remote key at this display our target crashes in the ROM version.We
use ST ELECTRONICS PROPRIETORY MICRO CONTROLLER  for video
applications.They have provided a ICE for debugging RAM version.

Now when we build the same and download it to RAM this scenerio never
happens.By the way I would like to know about "static initializers"
mentioned by you?
The only difference between these 2 versions is the way the code is
built.The ROM version actually combines thirdparty software along with
our code to make final image.We dont have source code for third party
software only binary is available.

Hope now I have made it clear....
Regards,
s.subbarayan

On 16 Dec, in article
     <1134738076.105723.202110@z14g2000cwz.googlegroups.com>
     ssubba@gmail.com "ssubbarayan" wrote:
>Hans-Bernhard Broeker wrote:
>> [F'up2 cut down --- neglected by OP]
>> In comp.arch.embedded ssubbarayan <ssubba@gmail.com> wrote:
>>
>> > We have a consumer electronics appliance in which when the Target boots
>> > up and puts a logo screen,when the user presses a key in the remote,the
>> > target crashes.This happens in the ROM version of build.
>> > When we debug it in RAM version,it never happens.I am clueless what
>> > could be possible reason for this.
>>
>> Given the minimal amount of information you give, it's pretty much
>> guaranteed that nobody else can have more of a clue than you do.
>>
>> [...]
>> > since ROM version is non debuggable,
>>
>> And why would that be?  There's no particularly strict reason ROM
>> builds of a program should be non-debuggable.  They may well be
>> *harder* to debug than RAM builds, because some of the usual
>> techniques (e.g. run-time code modifications) aren't immediately
>> applicable to ROM programs.  But it's not generally impossible.
>>
>> Or, to put it more provocatively: if your project and CPU is fat
>> enough that investing in an RTOS makes sense, how come you don't have
>> JTAG debugging?
>>
>> And how is that ROM build actually different from the RAM build?  The
>> one special case you mentioned hints that you didn't understand your
>> tools well enough to turn on all needed options for them to support a
>> feature of the standardized language like static initializers.
>>

>Hans,
>Thanks for your reply.I guess I need to elaborate you to make it more
>understandable.Its not non debuggable as u mentioned and I agree ROM
>debuggers are available in fact.But in my company we have not procured
>one and since we are at fag end of project,my clients are against
>budjetting it now.Ok heres what it happens:
>Our target Board boots up and puts a screen like this:
>Wait a moment please.... along with Logo of my company.When we press
>any remote key at this display our target crashes in the ROM version.We
>use ST ELECTRONICS PROPRIETORY MICRO CONTROLLER  for video
>applications.They have provided a ICE for debugging RAM version.

So you need to find out from ST ELECTRONICS what their ICE does and the
differences in building a RAM image.

>Now when we build the same and download it to RAM this scenerio never
>happens.By the way I would like to know about "static initializers"
>mentioned by you?

What language are you writing your application in? It suggests a lack of
understanding of the language you are using.

>The only difference between these 2 versions is the way the code is
>built.The ROM version actually combines thirdparty software along with
>our code to make final image.We dont have source code for third party
>software only binary is available.

Is the 3rd party image that you link with to build the ROM version the
EXACT same image you use for RAM version?

It still suggests to me that there is a startup code difference between
the images about initialising variables that is hidden by the RAM version.
Possibly the RAM version of the image ensures that parts are actually zero
before use but the ROM 3rd part image does not and relies on your startup
code.

Possibly you are initialising areas in the startup incorrectly, i.e.
initialising parts of RAM that are correct for ROM version but when linked
with the ROM version the starts of variable area is at a different address
to RAM version but the RAM image version are being initialised.

You need to check that all the changes you have to make between RAM and ROM
versions and ensure they are ALL being changed correctly. You need to find
out what ALL the differences between RAM and ROM versions of the 3rd party
image are.

The ICE may well be zeroing memory before download and hiding the issues.
 
>Hope now I have made it clear....
>Regards,
>s.subbarayan
>
>

-- 
Paul Carpenter          | paul@pcserviceselectronics.co.uk
<http://www.pcserviceselectronics.co.uk/>    PC Services
<http://www.gnuh8.org.uk/>              GNU H8 & mailing list info
<http://www.badweb.org.uk/>             For those web sites you hate

Paul Carpenter wrote:
> On 16 Dec, in article
>      <1134738076.105723.202110@z14g2000cwz.googlegroups.com>
>      ssubba@gmail.com "ssubbarayan" wrote:
> >Hans-Bernhard Broeker wrote:
> >> [F'up2 cut down --- neglected by OP]
> >> In comp.arch.embedded ssubbarayan <ssubba@gmail.com> wrote:
> >>
> >> > We have a consumer electronics appliance in which when the Target boots
> >> > up and puts a logo screen,when the user presses a key in the remote,the
> >> > target crashes.This happens in the ROM version of build.
> >> > When we debug it in RAM version,it never happens.I am clueless what
> >> > could be possible reason for this.
> >>
> >> Given the minimal amount of information you give, it's pretty much
> >> guaranteed that nobody else can have more of a clue than you do.
> >>
> >> [...]
> >> > since ROM version is non debuggable,
> >>
> >> And why would that be?  There's no particularly strict reason ROM
> >> builds of a program should be non-debuggable.  They may well be
> >> *harder* to debug than RAM builds, because some of the usual
> >> techniques (e.g. run-time code modifications) aren't immediately
> >> applicable to ROM programs.  But it's not generally impossible.
> >>
> >> Or, to put it more provocatively: if your project and CPU is fat
> >> enough that investing in an RTOS makes sense, how come you don't have
> >> JTAG debugging?
> >>
> >> And how is that ROM build actually different from the RAM build?  The
> >> one special case you mentioned hints that you didn't understand your
> >> tools well enough to turn on all needed options for them to support a
> >> feature of the standardized language like static initializers.
> >>
>
> >Hans,
> >Thanks for your reply.I guess I need to elaborate you to make it more
> >understandable.Its not non debuggable as u mentioned and I agree ROM
> >debuggers are available in fact.But in my company we have not procured
> >one and since we are at fag end of project,my clients are against
> >budjetting it now.Ok heres what it happens:
> >Our target Board boots up and puts a screen like this:
> >Wait a moment please.... along with Logo of my company.When we press
> >any remote key at this display our target crashes in the ROM version.We
> >use ST ELECTRONICS PROPRIETORY MICRO CONTROLLER  for video
> >applications.They have provided a ICE for debugging RAM version.
>
> So you need to find out from ST ELECTRONICS what their ICE does and the
> differences in building a RAM image.
>
> >Now when we build the same and download it to RAM this scenerio never
> >happens.By the way I would like to know about "static initializers"
> >mentioned by you?
>
> What language are you writing your application in? It suggests a lack of
> understanding of the language you are using.
>
> >The only difference between these 2 versions is the way the code is
> >built.The ROM version actually combines thirdparty software along with
> >our code to make final image.We dont have source code for third party
> >software only binary is available.
>
> Is the 3rd party image that you link with to build the ROM version the
> EXACT same image you use for RAM version?
>
> It still suggests to me that there is a startup code difference between
> the images about initialising variables that is hidden by the RAM version.
> Possibly the RAM version of the image ensures that parts are actually zero
> before use but the ROM 3rd part image does not and relies on your startup
> code.
>
> Possibly you are initialising areas in the startup incorrectly, i.e.
> initialising parts of RAM that are correct for ROM version but when linked
> with the ROM version the starts of variable area is at a different address
> to RAM version but the RAM image version are being initialised.
>
> You need to check that all the changes you have to make between RAM and ROM
> versions and ensure they are ALL being changed correctly. You need to find
> out what ALL the differences between RAM and ROM versions of the 3rd party
> image are.
>
> The ICE may well be zeroing memory before download and hiding the issues.
>
> >Hope now I have made it clear....
> >Regards,
> >s.subbarayan
> >
> >
>
> --
> Paul Carpenter          | paul@pcserviceselectronics.co.uk
> <http://www.pcserviceselectronics.co.uk/>    PC Services
> <http://www.gnuh8.org.uk/>              GNU H8 & mailing list info
> <http://www.badweb.org.uk/>             For those web sites you hate

Paul,
Thanks for your detailed mail.It was really helpful to me to understand
the differences.As you asked,I am giving below these information:
I use C language and proprietory RTOS given by ST MICRO ELECTRONICS by
name OS20.
I would like to understand from you how to make sure the startup code
intialises all the variables properly?
As you suggested,we once had a problem when building the ROM version
which said,
def_bss section memory bigger then available memory and this did not
happen in my RAM version.I believe its a proof that what you are saying
is happening to me.
Is it possible for you to confirm me whether this message confirms the
same?

Looking farward for your pearls of wisdom,
Advanced thanks once again,
Regards,
s.subbarayan

Paul Carpenter wrote:
> On 16 Dec, in article
>      <1134738076.105723.202110@z14g2000cwz.googlegroups.com>
>      ssubba@gmail.com "ssubbarayan" wrote:
> >Hans-Bernhard Broeker wrote:
> >> [F'up2 cut down --- neglected by OP]
> >> In comp.arch.embedded ssubbarayan <ssubba@gmail.com> wrote:
> >>
> >> > We have a consumer electronics appliance in which when the Target boots
> >> > up and puts a logo screen,when the user presses a key in the remote,the
> >> > target crashes.This happens in the ROM version of build.
> >> > When we debug it in RAM version,it never happens.I am clueless what
> >> > could be possible reason for this.
> >>
> >> Given the minimal amount of information you give, it's pretty much
> >> guaranteed that nobody else can have more of a clue than you do.
> >>
> >> [...]
> >> > since ROM version is non debuggable,
> >>
> >> And why would that be?  There's no particularly strict reason ROM
> >> builds of a program should be non-debuggable.  They may well be
> >> *harder* to debug than RAM builds, because some of the usual
> >> techniques (e.g. run-time code modifications) aren't immediately
> >> applicable to ROM programs.  But it's not generally impossible.
> >>
> >> Or, to put it more provocatively: if your project and CPU is fat
> >> enough that investing in an RTOS makes sense, how come you don't have
> >> JTAG debugging?
> >>
> >> And how is that ROM build actually different from the RAM build?  The
> >> one special case you mentioned hints that you didn't understand your
> >> tools well enough to turn on all needed options for them to support a
> >> feature of the standardized language like static initializers.
> >>
>
> >Hans,
> >Thanks for your reply.I guess I need to elaborate you to make it more
> >understandable.Its not non debuggable as u mentioned and I agree ROM
> >debuggers are available in fact.But in my company we have not procured
> >one and since we are at fag end of project,my clients are against
> >budjetting it now.Ok heres what it happens:
> >Our target Board boots up and puts a screen like this:
> >Wait a moment please.... along with Logo of my company.When we press
> >any remote key at this display our target crashes in the ROM version.We
> >use ST ELECTRONICS PROPRIETORY MICRO CONTROLLER  for video
> >applications.They have provided a ICE for debugging RAM version.
>
> So you need to find out from ST ELECTRONICS what their ICE does and the
> differences in building a RAM image.
>
> >Now when we build the same and download it to RAM this scenerio never
> >happens.By the way I would like to know about "static initializers"
> >mentioned by you?
>
> What language are you writing your application in? It suggests a lack of
> understanding of the language you are using.
>
> >The only difference between these 2 versions is the way the code is
> >built.The ROM version actually combines thirdparty software along with
> >our code to make final image.We dont have source code for third party
> >software only binary is available.
>
> Is the 3rd party image that you link with to build the ROM version the
> EXACT same image you use for RAM version?
>
> It still suggests to me that there is a startup code difference between
> the images about initialising variables that is hidden by the RAM version.
> Possibly the RAM version of the image ensures that parts are actually zero
> before use but the ROM 3rd part image does not and relies on your startup
> code.
>
> Possibly you are initialising areas in the startup incorrectly, i.e.
> initialising parts of RAM that are correct for ROM version but when linked
> with the ROM version the starts of variable area is at a different address
> to RAM version but the RAM image version are being initialised.
>
> You need to check that all the changes you have to make between RAM and ROM
> versions and ensure they are ALL being changed correctly. You need to find
> out what ALL the differences between RAM and ROM versions of the 3rd party
> image are.
>
> The ICE may well be zeroing memory before download and hiding the issues.
>
> >Hope now I have made it clear....
> >Regards,
> >s.subbarayan
> >
> >
>
> --
> Paul Carpenter          | paul@pcserviceselectronics.co.uk
> <http://www.pcserviceselectronics.co.uk/>    PC Services
> <http://www.gnuh8.org.uk/>              GNU H8 & mailing list info
> <http://www.badweb.org.uk/>             For those web sites you hate

Paul,
Thanks for your detailed mail.It was really helpful to me to understand
the differences.As you asked,I am giving below these information:
I use C language and proprietory RTOS given by ST MICRO ELECTRONICS by
name OS20.
I would like to understand from you how to make sure the startup code
intialises all the variables properly?
As you suggested,we once had a problem when building the ROM version
which said,
def_bss section memory bigger then available memory and this did not
happen in my RAM version.I believe its a proof that what you are saying
is happening to me.
Is it possible for you to confirm me whether this message confirms the
same?

Incase you are confirming it,Can you help me to understand how to
resolve it?I mean what should be my approach and where should I start
looking to overcome this problem?

Looking farward for your pearls of wisdom,
Advanced thanks once again,
Regards,
s.subbarayan

On 18 Dec, in article
     <1134899638.078138.78490@g49g2000cwa.googlegroups.com>
     ssubba@gmail.com "ssubbarayan" wrote:
>Paul Carpenter wrote:
>> On 16 Dec, in article
>>      <1134738076.105723.202110@z14g2000cwz.googlegroups.com>
>>      ssubba@gmail.com "ssubbarayan" wrote:
>> >Hans-Bernhard Broeker wrote:
>> >> In comp.arch.embedded ssubbarayan <ssubba@gmail.com> wrote:
>> >>
>> >> > We have a consumer electronics appliance in which when the Target boots
>> >> > up and puts a logo screen,when the user presses a key in the remote,the
>> >> > target crashes.This happens in the ROM version of build.
>> >> > When we debug it in RAM version,it never happens.I am clueless what
>> >> > could be possible reason for this.
>> >>
>> >> Given the minimal amount of information you give, it's pretty much
>> >> guaranteed that nobody else can have more of a clue than you do.

.........

>> >> And how is that ROM build actually different from the RAM build?  The
>> >> one special case you mentioned hints that you didn't understand your
>> >> tools well enough to turn on all needed options for them to support a
>> >> feature of the standardized language like static initializers.
>>
>> >Hans,
....
>> >Our target Board boots up and puts a screen like this:
>> >Wait a moment please.... along with Logo of my company.When we press
>> >any remote key at this display our target crashes in the ROM version.We
>> >use ST ELECTRONICS PROPRIETORY MICRO CONTROLLER  for video
>> >applications.They have provided a ICE for debugging RAM version.
>>
>> So you need to find out from ST ELECTRONICS what their ICE does and the
>> differences in building a RAM image.
>>
>> >Now when we build the same and download it to RAM this scenerio never
>> >happens.By the way I would like to know about "static initializers"
>> >mentioned by you?
>>
>> What language are you writing your application in? It suggests a lack of
>> understanding of the language you are using.
>>
>> >The only difference between these 2 versions is the way the code is
>> >built.The ROM version actually combines thirdparty software along with
>> >our code to make final image.We dont have source code for third party
>> >software only binary is available.
>>
>> Is the 3rd party image that you link with to build the ROM version the
>> EXACT same image you use for RAM version?
>>
>> It still suggests to me that there is a startup code difference between
>> the images about initialising variables that is hidden by the RAM version.
>> Possibly the RAM version of the image ensures that parts are actually zero
>> before use but the ROM 3rd part image does not and relies on your startup
>> code.
>>
>> Possibly you are initialising areas in the startup incorrectly, i.e.
>> initialising parts of RAM that are correct for ROM version but when linked
>> with the ROM version the starts of variable area is at a different address
>> to RAM version but the RAM image version are being initialised.
>>
>> You need to check that all the changes you have to make between RAM and ROM
>> versions and ensure they are ALL being changed correctly. You need to find
>> out what ALL the differences between RAM and ROM versions of the 3rd party
>> image are.
>>
>> The ICE may well be zeroing memory before download and hiding the issues.

>Paul,
>Thanks for your detailed mail.It was really helpful to me to understand
>the differences.As you asked,I am giving below these information:
>I use C language and proprietory RTOS given by ST MICRO ELECTRONICS by
>name OS20.

Not familiar SPECIFICALLY with that OS, but aware of other systems and OS.

>I would like to understand from you how to make sure the startup code
>intialises all the variables properly?

This should be explained in manuals for the version of C you are using and
the RTOS manual.

>As you suggested,we once had a problem when building the ROM version
>which said,
>def_bss section memory bigger then available memory and this did not
>happen in my RAM version.I believe its a proof that what you are saying
>is happening to me.

What it means to me is that either

  1/  When you run in a RAM version the total size of RAM is way bigger
      than the combined size of RAM and ROM on your 'ROM' system hardware.

  2/  You have not properly defined your RAM areas for what RAM you
      have in your 'ROM' hardware.

  3/  You have used up too much RAM in your 'ROM' system for variables
      than there is available.

  4/  You have not checked the actual sizes of your programme and data
      areas that the linked version gives you. There must be a MAP file
      somewhere you can examine.

>Is it possible for you to confirm me whether this message confirms the
>same?

No not without a LOT more information, which considering you give the
APPEARANCE of being an application programmer doing embedded work thinking
he has the 'unlimited' memory of a PC, means the problems could be many.

You do not appear to have the grasp of your tools (compiler and OS), hardware
and how to structure for embedded version.

>Incase you are confirming it,Can you help me to understand how to
>resolve it?I mean what should be my approach and where should I start
>looking to overcome this problem?

If this is a commercial product, I suggest you need to hire someone in
who has experience of embedded systems even on consultancy basis who is
allowed to see the hardware, circuit diagrams, manuals, tools being used,
source code and project files.

-- 
Paul Carpenter          | paul@pcserviceselectronics.co.uk
<http://www.pcserviceselectronics.co.uk/>    PC Services
<http://www.gnuh8.org.uk/>              GNU H8 & mailing list info
<http://www.badweb.org.uk/>             For those web sites you hate