[cross-post] nested interrupts

Dear all,

I would like to understand more about nested interrupts since it looks
to me they break my code and I'm not sure why.

I have three 'items' in my software: a serial port, a fifo and a timer.
The serial port is an external hardware that sends back an interrupt
when it's free to send another byte, so I decided to have an interrupt
service routine (ISR) that reads from the fifo and puts the byte into
the serial port. With this choice the main program should start the
first byte transmission from the fifo and then everything is handled by
the ISR. Therefore the code looks like this:

> int main() {
> ...
>   while(1) {
>     write_to_fifo(data);
>     if (state != TRANSMITTING) {
>       if (read_from_fifo(&byte) == OK) {
>         write_to_sport(byte);
>         state = TRANSMITTING;
>       }
>     }
>   }
> }

the variable 'state' is needed in order to prevent the main to write
into the serial port while the ISR is already doing this.
The ISR would look like the following:
> 
> void isr_sport(int signal) {
> 
>   if (read_from_fifo(&byte) == OK) write_to_sport(byte);
>   else                             state == DONE;
> }

When the fifo will be empty the ISR will change the state to DONE such
that main can start a new transmission again.

This flow works perfectly as expected but now I decided to add a timer
interrupt which is simply incrementing my 'system time' variable,
nothing else.
If the 'interrupt nesting' is disabled everything works as expected, but
if it is enabled something breaks. After some bytes transmitted to the
serial port the program simply runs through the main, as if the 'state'
variable was not set to DONE when the ISR emptied the fifo.

My simple 'naive' model for this is that while the serial port ISR was
running it was interrupted by the timer interrupt (higher priority) but
once the timer ISR was completed, instead of returning to the serial
port ISR it returned directly to the main, missing the instruction to
change the state. At this point there's no chance for the ISR to change
the 'state' variable and the main will wait forever a never changing state.

But is this 'possible'? I started looking at the assembler for the
interrupt dispatcher and interrupt handling routines but still didn't
manage to find a potential flaw (even though I admit I did not fully
understand everything there).

Any suggestion or pointer is more than welcome.
Thanks a lot,

Al

p.s.: the target is an ADSP21020 and the compiler is g21k.

-- 
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

On 07/01/2012 03:27 PM, alb wrote:
>>
>> void isr_sport(int signal) {
>>
>>    if (read_from_fifo(&byte) == OK) write_to_sport(byte);
>>    else                             state == DONE;
>> }
>

Try 'state = DONE' instead.

On 7/1/2012 4:19 PM, Arlet Ottens wrote:
> On 07/01/2012 03:27 PM, alb wrote:
>>>
>>> void isr_sport(int signal) {
>>>
>>>    if (read_from_fifo(&byte) == OK) write_to_sport(byte);
>>>    else                             state == DONE;
>>> }
>>
> 
> Try 'state = DONE' instead.

Ok, that was a typo when posting, my apologies. But as I mentioned, the
logic works for some time until the 'clash' with the timer interrupt.

On 7/1/12 11:21 AM, alb wrote:
> On 7/1/2012 4:19 PM, Arlet Ottens wrote:
>> On 07/01/2012 03:27 PM, alb wrote:
>>>>
>>>> void isr_sport(int signal) {
>>>>
>>>>     if (read_from_fifo(&byte) == OK) write_to_sport(byte);
>>>>     else                             state == DONE;
>>>> }
>>>
>>
>> Try 'state = DONE' instead.
>
> Ok, that was a typo when posting, my apologies. But as I mentioned, the
> logic works for some time until the 'clash' with the timer interrupt.

i have absolutely no idea what processor this runs on, but to do nested 
interrupts without too much trouble, you need to have incremental 
priority levels for the interrupts, where only a higher priority 
interrupt source may interrupt another process.  some processors allow 
you to assign the priority of your choice to each interrupt source.

of course, for each interrupt, you must fully save and restore every 
register that is used in the interrupt and that the "status register" or 
"condition code register" or whatever they call it, must be restored 
exactly.

then the last issue is the system stack.  if you have lotsa nested 
interrupts, your stack (where you're saving all these registers) will 
get deeper and deeper.  make sure that neither the stack tramples over 
some other in-use memory nor that some other process accidentally 
tramples over the stack.

other than that, everything should work transparently with no difference 
*except* for time.  any process that is interrupted will get back to 
doing its job and will pick up exactly where it left off, *except* it's 
at a later time than would be if it were not interrupted.  sometimes, in 
a real-time embedded system, that delay in time kills you if the process 
is reading or writing I/O and the outside world is unhappy about the delay.

that's all the advice i can throw at you.  it's been 2 decades since i 
had to worry about nested interrupts.  chips have evolved/changed since 
then, but the principles are the same.


-- 

r b-j                  rbj@audioimagination.com

"Imagination is more important than knowledge."

On 7/1/2012 6:36 PM, robert bristow-johnson wrote:
> i have absolutely no idea what processor this runs on,

it is in the p.s. of the OP: ADSP21020 with g21k compiler.

> but to do nested
> interrupts without too much trouble, you need to have incremental
> priority levels for the interrupts, where only a higher priority
> interrupt source may interrupt another process.  some processors allow
> you to assign the priority of your choice to each interrupt source.

In the case of the processor I'm using the priority is fixed but I have
a choice to use a high priority interrupt or low priority for the timer.
In this case I'm using the high priority timer interrupt.

> 
> of course, for each interrupt, you must fully save and restore every
> register that is used in the interrupt and that the "status register" or
> "condition code register" or whatever they call it, must be restored
> exactly.

This part is done by the library and this is where I fear something is
not properly working.

> 
> then the last issue is the system stack.  if you have lotsa nested
> interrupts, your stack (where you're saving all these registers) will
> get deeper and deeper.  make sure that neither the stack tramples over
> some other in-use memory nor that some other process accidentally
> tramples over the stack.

I understand that the 'system stack' may be critical but here there
should not be any problem since there's nothing going deeper than two
interrupt calls. Maybe I should check what the interrupt call does at
the assembler level and make sure that everything is in place.

> 
> other than that, everything should work transparently with no difference
> *except* for time.  any process that is interrupted will get back to
> doing its job and will pick up exactly where it left off, *except* it's
> at a later time than would be if it were not interrupted.  sometimes, in
> a real-time embedded system, that delay in time kills you if the process
> is reading or writing I/O and the outside world is unhappy about the delay.
> 

No issue on delays, that's why I can even live without nested interrupt
and wait for the current interrupt is served before serving the next
one. The point is that I would like to understand the reason for such a
strange behavior since I fear that it will byte me back later if I don't
pin it down at this stage.

> that's all the advice i can throw at you.  it's been 2 decades since i
> had to worry about nested interrupts.  chips have evolved/changed since
> then, but the principles are the same.
> 

The processor and the toolchain I'm working with is exactly 2 decades old...

Le 01/07/2012 15:27, alb a �crit :
> Dear all,
>
> I would like to understand more about nested interrupts since it looks
> to me they break my code and I'm not sure why.
>
> I have three 'items' in my software: a serial port, a fifo and a timer.
> The serial port is an external hardware that sends back an interrupt
> when it's free to send another byte, so I decided to have an interrupt
> service routine (ISR) that reads from the fifo and puts the byte into
> the serial port. With this choice the main program should start the
> first byte transmission from the fifo and then everything is handled by
> the ISR. Therefore the code looks like this:
>
>> int main() {
>> ...
>>    while(1) {
>>      write_to_fifo(data);
>>      if (state != TRANSMITTING) {
>>        if (read_from_fifo(&byte) == OK) {
>>          write_to_sport(byte);
>>          state = TRANSMITTING;
>>        }
>>      }
>>    }
>> }
>
> the variable 'state' is needed in order to prevent the main to write
> into the serial port while the ISR is already doing this.
> The ISR would look like the following:
>>
>> void isr_sport(int signal) {
>>
>>    if (read_from_fifo(&byte) == OK) write_to_sport(byte);
>>    else                             state == DONE;
>> }
>
> When the fifo will be empty the ISR will change the state to DONE such
> that main can start a new transmission again.
>
> This flow works perfectly as expected but now I decided to add a timer
> interrupt which is simply incrementing my 'system time' variable,
> nothing else.
> If the 'interrupt nesting' is disabled everything works as expected, but
> if it is enabled something breaks. After some bytes transmitted to the
> serial port the program simply runs through the main, as if the 'state'
> variable was not set to DONE when the ISR emptied the fifo.
>
> My simple 'naive' model for this is that while the serial port ISR was
> running it was interrupted by the timer interrupt (higher priority) but
> once the timer ISR was completed, instead of returning to the serial
> port ISR it returned directly to the main, missing the instruction to
> change the state. At this point there's no chance for the ISR to change
> the 'state' variable and the main will wait forever a never changing state.
>
> But is this 'possible'? I started looking at the assembler for the
> interrupt dispatcher and interrupt handling routines but still didn't
> manage to find a potential flaw (even though I admit I did not fully
> understand everything there).
>
> Any suggestion or pointer is more than welcome.
> Thanks a lot,
>
I haven't seen any flaw in either your code or your concept.
It would be useful to have the assembly code, particularly
that of the ISR and the pre and post ISR code to come up
with something.

On 12-07-01 16:27 , alb wrote:
> Dear all,
>
> I would like to understand more about nested interrupts since it looks
> to me they break my code and I'm not sure why.
>
> I have three 'items' in my software: a serial port, a fifo and a timer.
> The serial port is an external hardware that sends back an interrupt
> when it's free to send another byte, so I decided to have an interrupt
> service routine (ISR) that reads from the fifo and puts the byte into
> the serial port. With this choice the main program should start the
> first byte transmission from the fifo and then everything is handled by
> the ISR. Therefore the code looks like this:
>
>> int main() {
>> ...
>>    while(1) {
>>      write_to_fifo(data);
>>      if (state != TRANSMITTING) {
>>        if (read_from_fifo(&byte) == OK) {
>>          write_to_sport(byte);

At this point you have a race between main() and isr_sport(): both will 
assign something to "state". Normally main() will no doubt have time to 
execute the assignment below, before isr_sport() is called, but if 
something else interrupts main() at this point, isr_sport() may reach 
its statement "state = DONE" first, which means that main() then 
overwrites DONE with TRANSMITTING in "state", and the logic fails.

>>          state = TRANSMITTING;
>>        }
>>      }
>>    }
>> }
>
> the variable 'state' is needed in order to prevent the main to write
> into the serial port while the ISR is already doing this.
> The ISR would look like the following:
>>
>> void isr_sport(int signal) {
>>
>>    if (read_from_fifo(&byte) == OK) write_to_sport(byte);
>>    else                             state = DONE;
>> }
>
> When the fifo will be empty the ISR will change the state to DONE such
> that main can start a new transmission again.
>
> This flow works perfectly as expected but now I decided to add a timer
> interrupt which is simply incrementing my 'system time' variable,
> nothing else.
> If the 'interrupt nesting' is disabled everything works as expected, but
> if it is enabled something breaks. After some bytes transmitted to the
> serial port the program simply runs through the main, as if the 'state'
> variable was not set to DONE when the ISR emptied the fifo.

This symptom can result from the race condition above.

I'm not sure how the dependence on interrupt nesting arises. Perhaps, 
when nesting is disabled, and the serial-port interrupt is signalled 
while the timer interrupt is being handled, one or a few instructions in 
main() are executed between the return from the timer interrupt handler 
and before the serial-port handler is executed. This might give main() 
enough time to execute its "state" assignment before isr_sport() runs.

To remove the race, move the assignment "state = TRANSMITTING" before 
the write_to_sport(), in main().

By the way, check that you have declared "state" as volatile.

-- 
Niklas Holsti
Tidorum Ltd
niklas holsti tidorum fi
       .      @       .

On Sunday 01 July 2012 15:27 alb wrote:

> Dear all,
> 
> I would like to understand more about nested interrupts since it looks
> to me they break my code and I'm not sure why.
> 
> I have three 'items' in my software: a serial port, a fifo and a timer.
> The serial port is an external hardware that sends back an interrupt
> when it's free to send another byte, so I decided to have an interrupt
> service routine (ISR) that reads from the fifo and puts the byte into
> the serial port. With this choice the main program should start the
> first byte transmission from the fifo and then everything is handled by
> the ISR. Therefore the code looks like this:
> 
>> int main() {
>> ...
>>   while(1) {
>>     write_to_fifo(data);
>>     if (state != TRANSMITTING) {
>>       if (read_from_fifo(&byte) == OK) {
>>         write_to_sport(byte);
>>         state = TRANSMITTING;
>>       }
>>     }
>>   }
>> }
> 
> the variable 'state' is needed in order to prevent the main to write
> into the serial port while the ISR is already doing this.
> The ISR would look like the following:
>> 
>> void isr_sport(int signal) {
>> 
>>   if (read_from_fifo(&byte) == OK) write_to_sport(byte);
>>   else                             state == DONE;
>> }
> 
> When the fifo will be empty the ISR will change the state to DONE such
> that main can start a new transmission again.
> 
> This flow works perfectly as expected but now I decided to add a timer
> interrupt which is simply incrementing my 'system time' variable,
> nothing else.
> If the 'interrupt nesting' is disabled everything works as expected, but
> if it is enabled something breaks. After some bytes transmitted to the
> serial port the program simply runs through the main, as if the 'state'
> variable was not set to DONE when the ISR emptied the fifo.
> 
> My simple 'naive' model for this is that while the serial port ISR was
> running it was interrupted by the timer interrupt (higher priority) but
> once the timer ISR was completed, instead of returning to the serial
> port ISR it returned directly to the main, missing the instruction to
> change the state. At this point there's no chance for the ISR to change
> the 'state' variable and the main will wait forever a never changing
> state.
> 
> But is this 'possible'? I started looking at the assembler for the
> interrupt dispatcher and interrupt handling routines but still didn't
> manage to find a potential flaw (even though I admit I did not fully
> understand everything there).
> 
> Any suggestion or pointer is more than welcome.
> Thanks a lot,
> 
> Al
> 
> p.s.: the target is an ADSP21020 and the compiler is g21k.
> 

You have not shown, if somewhere interrupts are disabled. 
I see one possible race condition:

     if (state != TRANSMITTING) {
       if (read_from_fifo(&byte) == OK) {
           write_to_sport(byte);
----
	 isr_sport() is triggered and sets state=DONE
----
           state = TRANSMITTING;


Without the timer running this will probably never happen. But the timer ISR 
might just step in after write_to_port and consume CPU time.

The simplest way out of this is to disable interrupts before 
write_to_sport() and enable them after state = TRANSMITTING;

-- 
Reinhardt Behm
rb@rbehm.de

On Sun, 01 Jul 2012 12:36:02 -0400, robert bristow-johnson wrote:

> On 7/1/12 11:21 AM, alb wrote:
>> On 7/1/2012 4:19 PM, Arlet Ottens wrote:
>>> On 07/01/2012 03:27 PM, alb wrote:
>>>>>
>>>>> void isr_sport(int signal) {
>>>>>
>>>>>     if (read_from_fifo(&byte) == OK) write_to_sport(byte); else     
>>>>>                            state == DONE;
>>>>> }
>>>>
>>>>
>>> Try 'state = DONE' instead.
>>
>> Ok, that was a typo when posting, my apologies. But as I mentioned, the
>> logic works for some time until the 'clash' with the timer interrupt.
> 
> i have absolutely no idea what processor this runs on, but to do nested
> interrupts without too much trouble, you need to have incremental
> priority levels for the interrupts, where only a higher priority
> interrupt source may interrupt another process.  some processors allow
> you to assign the priority of your choice to each interrupt source.
> 
> of course, for each interrupt, you must fully save and restore every
> register that is used in the interrupt and that the "status register" or
> "condition code register" or whatever they call it, must be restored
> exactly.

A light went on:

Often the amount of stuff you have to save for an ISR is greater than the 
amount of stuff you have to save for an ordinary function call.

Most modern C compilers let you declare a function to be the target of an 
interrupt.  The syntax isn't standardized -- for some it's "void 
interrupt foo()", for some it's something like "void foo() attribute
("__interrupt__")", etc.  But it boils down to letting the compiler know 
that it's an interrupt service routine, so that it can do the extra saves 
& restores.

It may be that you were just lucking out without nested interrupts, and 
were constantly exposing yourself to trouble and the other shoe just 
never dropped.

So -- dig through the processor's documentation on its interrupt 
response, and look at the compiler's assembly output to see if it seems 
to be saving away enough stuff on each function call.  If not, dig 
through the compiler's documentation to see how to tell it that an ISR is 
an ISR.

And finally, if you can get by without nested interrupts -- don't use 
them.

-- 
My liberal friends think I'm a conservative kook.
My conservative friends think I'm a liberal kook.
Why am I not happy that they have found common ground?

Tim Wescott, Communications, Control, Circuits & Software
http://www.wescottdesign.com

On 7/1/12 1:04 PM, alb wrote:
> On 7/1/2012 6:36 PM, robert bristow-johnson wrote:
>> i have absolutely no idea what processor this runs on,
>
> it is in the p.s. of the OP: ADSP21020 with g21k compiler.
>
>> but to do nested
>> interrupts without too much trouble, you need to have incremental
>> priority levels for the interrupts, where only a higher priority
>> interrupt source may interrupt another process.  some processors allow
>> you to assign the priority of your choice to each interrupt source.
>
> In the case of the processor I'm using the priority is fixed but I have
> a choice to use a high priority interrupt or low priority for the timer.

i remember that.  TIMER was the only one that could do that.  and i 
remember being disgusted that there were some devices that we needed at 
a higher priority and couldn't implement it directly.

> In this case I'm using the high priority timer interrupt.
>

okay, what other peripherals do you have hooked up and what are their 
relative priorities?  and, then, about how often are these different 
interrupts expected to hit?  if you have a low-priority interrupt that 
is expected to hit at, say, 48 kHz, and there is a higher priority 
interrupt that is connected to a user control (and it has a long and 
complicated ISR, but one that doesn't need instant attention).  you 
could have a situation where this naturally high-priority device is held 
hostage to this naturally low-priority device which is fixed in the DSP 
at a higher level priority.

>>
>> of course, for each interrupt, you must fully save and restore every
>> register that is used in the interrupt and that the "status register" or
>> "condition code register" or whatever they call it, must be restored
>> exactly.
>
> This part is done by the library and this is where I fear something is
> not properly working.
>

>On 7/1/12 5:49 PM, Tim Wescott wrote:
>

>
> A light went on:
>
> Often the amount of stuff you have to save for an ISR is greater than the
> amount of stuff you have to save for an ordinary function call.
>
> Most modern C compilers let you declare a function to be the target of an
> interrupt.  The syntax isn't standardized -- for some it's "void
> interrupt foo()", for some it's something like "void foo() attribute
> ("__interrupt__")", etc.  But it boils down to letting the compiler know
> that it's an interrupt service routine, so that it can do the extra saves
> &  restores.
>

Tim, i'm sure the stub of code that connects to the C functions (that 
don't need to save/restore certain "scratch" registers and that sometime 
return values in registers) from the interrupt vector.  i imagine that 
code takes care of what needs to be saved and restored.

there is one thing about the 210xx in that it has an alternate register 
set.  we used that in a sorta inverted way.  we used the "alternate set" 
as the regular set (that most of the C code was using) and the "normal 
set" was used for these ISRs that didn't need to save/restore (they 
owned the registers they used).  the reason why was because it was 
easier to clear the interrupt enable bit and the bits for those 
alternate register sets with a single AND at the beginning of the 
interrupt and then restore as a final OR instruction before the RTI (or 
whatever it's called).  the only restriction was that any C code called 
by these interrupts (that flipped some of the alternate register sets) 
could not have register local variables.  but none of those ISRs were 
there to run C code.

anyway, if you can make use of the alternate register set for all of 
your ISRs, and commit some registers, you might not need to do any 
save/restore.

>
> And finally, if you can get by without nested interrupts -- don't use
> them.
>


>> then the last issue is the system stack.  if you have lotsa nested
>> interrupts, your stack (where you're saving all these registers) will
>> get deeper and deeper.  make sure that neither the stack tramples over
>> some other in-use memory nor that some other process accidentally
>> tramples over the stack.
>
> I understand that the 'system stack' may be critical but here there
> should not be any problem since there's nothing going deeper than two
> interrupt calls. Maybe I should check what the interrupt call does at
> the assembler level and make sure that everything is in place.
>
>>
>> other than that, everything should work transparently with no difference
>> *except* for time.  any process that is interrupted will get back to
>> doing its job and will pick up exactly where it left off, *except* it's
>> at a later time than would be if it were not interrupted.  sometimes, in
>> a real-time embedded system, that delay in time kills you if the process
>> is reading or writing I/O and the outside world is unhappy about the delay.
>>
>
> No issue on delays, that's why I can even live without nested interrupt
> and wait for the current interrupt is served before serving the next
> one. The point is that I would like to understand the reason for such a
> strange behavior since I fear that it will byte me back later if I don't
> pin it down at this stage.
>
>> that's all the advice i can throw at you.  it's been 2 decades since i
>> had to worry about nested interrupts.  chips have evolved/changed since
>> then, but the principles are the same.
>>
>
> The processor and the toolchain I'm working with is exactly 2 decades old...

well, that was about as long ago that i had been working on a SHArC.  it 
was ADSP21062 version 0.6 .  lotsa bugs in the tools and some ugly (and 
therefore useless) designs in the instruction set and architecture of 
the SHArC.  the MANT instruction (i think that's what it was called), is 
useless since it didn't return a float.  it makes it more difficult to 
do math and you can get a float version of a fractional number with a 
conversion and subtraction.



-- 

r b-j                  rbj@audioimagination.com

"Imagination is more important than knowledge."