Unexplained Hang During Boot

Bill Pringlemeir wrote:
>>eon_blue_80@verizon.net wrote:
> 
> 
>>Another possibility is that errant code is corrupting memory during
>>the boot process.
> 
> 
> This is *unlikely* as the OP noted that adding un-executed code would
> cause the problem.  If the code is directly corrupting memory this
> would be unlikely to introduce the problem.  Especially if the added
> code make no types of allocation, nor writes to memory.  If simply
> changing the cache on/off will cause the crash, I find it extremely
> unlikely that it is a memory corruption.

I have seen this happen in the past in this manner. By adding code into 
the code segement, you move the relative position of stuff around. Even 
if the code you added does not get executed, if the I/O drivers are at 
opposite end of the link map from the boot code, just increasing or 
decreasing the relative separation of components can cause the 
corruption to occur in a place that does not get executed during the 
boot process or causes a different kind of problem. C libraries are 
another good candidate for winding up at the far end of the link map. If 
you are lucky, this will show up as an illegal instruction trap, and if 
you are unlucky, it shows up as branches to nowhere or tight loops.


> So there is a quick way to rule this out.  Disable/enable the cache
> with a crashing image.  Often you can arrange the code so that the
> size is the same, just a constant has changed to disable/enable the
> cache.

This would be a good first step. The OP sounded like he was fishing for 
ideas, so I threw out a couple that I have run into in the past.

Bob

> This would be a good first step. The OP sounded like he was fishing for
> ideas, so I threw out a couple that I have run into in the past.

I also would tip on cache handling problems in the code. Forgotten
flush of the i-cache is something I have had to chase with my early
versions.
 There is one more possibility I know of. If the processor is a 405,
check
its errata sheet. I recently discovered (while considering a device,
I opted not to use it) a late published error to be saying basically
you may not use its cache in copyback mode, it does not work.
Use write through....

Dimiter

------------------------------------------------------
Dimiter Popoff               Transgalactic Instruments

http://www.tgi-sci.com
------------------------------------------------------

MetalHead wrote:
> Bill Pringlemeir wrote:
> >>eon_blue_80@verizon.net wrote:
> >
> >
> >>Another possibility is that errant code is corrupting memory during
> >>the boot process.
> >
> >
> > This is *unlikely* as the OP noted that adding un-executed code would
> > cause the problem.  If the code is directly corrupting memory this
> > would be unlikely to introduce the problem.  Especially if the added
> > code make no types of allocation, nor writes to memory.  If simply
> > changing the cache on/off will cause the crash, I find it extremely
> > unlikely that it is a memory corruption.
>
> I have seen this happen in the past in this manner. By adding code into
> the code segement, you move the relative position of stuff around. Even
> if the code you added does not get executed, if the I/O drivers are at
> opposite end of the link map from the boot code, just increasing or
> decreasing the relative separation of components can cause the
> corruption to occur in a place that does not get executed during the
> boot process or causes a different kind of problem. C libraries are
> another good candidate for winding up at the far end of the link map. If
> you are lucky, this will show up as an illegal instruction trap, and if
> you are unlucky, it shows up as branches to nowhere or tight loops.
>
>
> > So there is a quick way to rule this out.  Disable/enable the cache
> > with a crashing image.  Often you can arrange the code so that the
> > size is the same, just a constant has changed to disable/enable the
> > cache.
>
> This would be a good first step. The OP sounded like he was fishing for
> ideas, so I threw out a couple that I have run into in the past.
> 
> Bob

Didi wrote:

>  There is one more possibility I know of. If the processor is a 405,
> check
> its errata sheet. I recently discovered (while considering a device,
> I opted not to use it) a late published error to be saying basically
> you may not use its cache in copyback mode, it does not work.
> Use write through....

We haven't used write-through, ever, on the 405GPr and it has had narry
a problem with copy-back at least for the past 4 years of the product
life (thousands of blade servers). Do you have an errata number or doc.
I could look at WRT to this cache bug? If you are referring to CPU_213
you need only to set CCR0 as specified. Setting write-through mode is
simply too big a hammer (for us).
-- 
- Mark

eon_blue_80@verizon.net wrote:

> I am experiencing a very bizarre problem with vxWorks and I am hoping
> that someone might be able to offer some suggestions on where to start
> looking to determine the root of the problem.
> 
> VxWorks is being used on a Synergy Microsystems VME SBC which is PPC
> based.  The problem seems to arise at random times after rebuilding the
> OS image.  For instance, by commenting out a single 'printf' statement
> such as "printf("Message Received\n"); in an application level piece of
> code that is not even invoked; and rebuilding the image, the image can
> hang while booting (early in the boot procedure).  Uncomment this
> 'printf' statement, rebuild the image, and the OS will boot without
> error. Note that this routine is not called at any time during the boot
> procedure so the code containing that printf is never even executed.

Reading your post, it's not clear how many different
physical units you've tried this on.  If the answer
is one, the problem could be a bad byte with a bad
bit of flash memory.

>  If you are referring to CPU_213 you need only to set CCR0 as specified.

This is what I was referring to, apparently you have it under control.
It was
enough to stop me from using the 405 (I opted for the 5200).

Dimiter

------------------------------------------------------
Dimiter Popoff               Transgalactic Instruments

http://www.tgi-sci.com
------------------------------------------------------

mrfirmware wrote:
> Didi wrote:
>
> >  There is one more possibility I know of. If the processor is a 405,
> > check
> > its errata sheet. I recently discovered (while considering a device,
> > I opted not to use it) a late published error to be saying basically
> > you may not use its cache in copyback mode, it does not work.
> > Use write through....
>
> We haven't used write-through, ever, on the 405GPr and it has had narry
> a problem with copy-back at least for the past 4 years of the product
> life (thousands of blade servers). Do you have an errata number or doc.
> I could look at WRT to this cache bug? If you are referring to CPU_213
> you need only to set CCR0 as specified. Setting write-through mode is
> simply too big a hammer (for us).
> -- 
> - Mark

Thank you everyone for all of your suggestions.  These suggestions will
be a great help when troubleshooting future problems.

As far as the original problem goes, using I/O probing we were able to
successfully narrow the error down to a relatively large segment of the
BSP.  Apparently there is a problem in the SCSI section of the BSP
(wild pointer or out of order type operation??) that causes the image
to hang when the bytes of the image are aligned in just the right way.
We have made a decision to disable SCSI support within the OS (which
has corrected the problem).  Hopefully, if time ever becomes available,
we can look into the SCSI section of the BSP; and find the exact bug.