EmbeddedRelated.com
Forums

[cross-post] nested interrupts

Started by alb July 1, 2012
On 7/2/2012 6:31 PM, Niklas Holsti wrote:
> On 12-07-02 13:37 , alb wrote: >> On 7/1/2012 8:19 PM, Niklas Holsti wrote: >> [...] >>>>> int main() { >>>>> ... >>>>> while(1) { >>>>> write_to_fifo(data); >>>>> if (state != TRANSMITTING) { >>>>> if (read_from_fifo(&byte) == OK) { >>>>> write_to_sport(byte); >>> >>> At this point you have a race between main() and isr_sport(): both will >>> assign something to "state". Normally main() will no doubt have time to >>> execute the assignment below, before isr_sport() is called, but if >>> something else interrupts main() at this point, isr_sport() may reach >>> its statement "state = DONE" first, which means that main() then >>> overwrites DONE with TRANSMITTING in "state", and the logic fails. >>> >> >> got your point about the race condition, but let me dig a little further >> here. At the point you mentioned the isr_sport is not running, since >> there's no transmission going on. > > The point I meant was just after the statement > > write_to_sport(byte); > > I assumed that this statement writes "byte" to the serial port, and that > (as usual for output ports) writing has the side effect of starting the > transmission of the byte. (If this is not so, your code does not show > how and where the serial transmission is started.)
That is correct.
> > Since the serial port is now transmitting, isr_sport() will be called > when the byte is transmitted and the serial-port interrupt occurs. In > this sense, main() and isr_sport() are concurrent at this point. If you > don't really know how fast things go (because of other interrupts or > delays that can happen), either main() or isr_sport() can reach its > assignment to "state" first. That is a race.
Got it. I was indeed assuming that time between write_to_sport() and state = TRANSMITTING was small enough to take allow the first transmission, but if something else delays the execution of the 'state = TRANSMITTING' instruction I may run in the race problem.
> >> The only interrupt that may kick in is >> the timer ISR. > > No, the serial-port interrupt can also occur, because you started > transmission by writing the byte to the serial port. *When* the > interrupt happens depends on the speed of the serial port. >
Indeed I think the hardware has a register to store the data and another to shift it out, hence as soon as the register is free to accept a new byte the interrupt will be asserted. This means that as soon as the first byte is written the interrupt is asserted (on next cycle), causing isr_sport() to run. [...]
>> If all this is true though then I should have this problem even without >> interrupt nesting. > > Maybe, maybe not. As I said, without interrupt nesting I'm guessing that > when the timer-interrupt handler executes return-from-interrupt, control > returns to the main() function for a couple of instructions (the 21020 > has a deep pipeline with two branch-delay slots) before the pending > serial-port interrupt forces control to isr_sport(). This may give > main() enough time between the two ISRs to execute the assignment "state > = TRANSMITTING", giving the right order of assignments. > > Also, the 21020 delays interrupt responses for some multi-cycle > instructions, including branch. Since the statement "state = > TRANSMITTING" is the last statement in a loop body, it is quite likely > that the compiler puts it in a delay slot of the "return to loop head" > branch. This makes it more likely that the assigment is executed between > the return from the timer-interrupt handler and the invocation of the > serial-port interrupt handler, when interrupt nesting is disabled. >
uhm... interesting. I should maybe try to understand how the program sequencer works. [...]
>>> By the way, check that you have declared "state" as volatile. >>> >> >> why? in the event of 'state' been optimized away then it should never >> work, right? > > Today the compiler may be kind to you, and not make trouble. Do you want > to be surprised tomorrow, after some small, unrelated change to the > program? > > The compiler is free to reorder writes to non-volatile variables, as > long as the mathematical effect is OK. If x and y are two non-volatile > variables (or if only one of them is volatile) and you write in your C code > > x = 1; > y = 2; > > the C compiler can translate it to an instruction sequence that first > assigns 2 to y and then assigns 1 to x. If x and y are both volatile, > the compiler must respect the written order of assignments (and also of > reads of the variables). > > So if "state" is not a volatile variable, moving its assignment > statement in the C source code may or may not move it in the assembly code. > > What is write_to_sport()? Is is a macro or a function? If it is a macro > (or a possibly in-lined function) of the form "portvar = byte", you > should also declare "portvar" as volatile, to ensure that the order of > assignments is as in the C source.
indeed it is a memory mapped register defined as volatile and I declared it such since I thought this is the "way" to tell the compiler that the content of this variable may change due to hardware operations. I was not aware that non volatile variables could be freely reordered. This will definitely screw up my logic, since I rely on things happening with certain chronological order (which is maybe a bad practice).
> > And of course you should also make sure that accesses to the FIFO are > protected against wrongful concurrent manipulations from main() and > isr_sport(), as was suggested in other posts.
I understand, but I thought that having the isr_sport() only reading and the main() only writing I would have avoided this problem. Truth is that the first byte is read out by the main... ouch.
On 7/2/2012 9:35 AM, #! wrote:
[...]
>>> int main() { >>> ... >>> while(1) { >>> write_to_fifo(data); >>> if (state != TRANSMITTING) { >>> if (read_from_fifo(&byte) == OK) { >>> write_to_sport(byte); >>> state = TRANSMITTING; >>> } >>> } >>> } >>> } >>
[...]
>>> void isr_sport(int signal) { >>> >>> if (read_from_fifo(&byte) == OK) write_to_sport(byte); >>> else state == DONE; >>> } >> > > The fifo is shared between main and the ISR. You are not preventing the > ISR triggering while inserting more data into the fifo. This would > result in a race condition.
the main() is only writing to the fifo and the isr_sport() only reading - except at the beginning of transmission - so why this should result in a race condition? In the write_to_fifo() I check if the fifo is full and if not I then write to it, incrementing the write pointer. In the read_from_fifo() I check if the fifo is empty and if not read from it, incrementing the read pointer. Maybe I'm missing something here.
On 7/3/2012 1:16 PM, alb wrote:
[...]
>> The fifo is shared between main and the ISR. You are not preventing the >> ISR triggering while inserting more data into the fifo. This would >> result in a race condition. > > the main() is only writing to the fifo and the isr_sport() only reading > - except at the beginning of transmission - so why this should result in > a race condition? > > In the write_to_fifo() I check if the fifo is full and if not I then > write to it, incrementing the write pointer. In the read_from_fifo() I > check if the fifo is empty and if not read from it, incrementing the > read pointer. Maybe I'm missing something here. >
I think I was really missing something here!!! These are the write_to_fifo() and read_from_fifo() functions:
> bool write_to_fifo(int8 d) { > > if ((fifo->wr != fifo->rd) || fifo->ef == EMPTY) { > fifo->bf[fifo->wr] = d; > fifo->wr = MOD(fifo->wr+1, DIM(fifo->bf)); > if (fifo->wr == fifo->rd) fifo->ef = FULL; > return TRUE; > } > > return FALSE; > } >
> bool read_from_fifo(int8 *d) { > > if ((fifo->wr != fifo->rd) || fifo->ef == FULL) { > *d = fifo->bf[fifo->rd]; > fifo->rd = MOD(fifo->rd+1, DIM(fifo->bf)); > if (fifo->rd == f->wr) fifo->ef = EMPTY; > return TRUE; > } > > return FALSE; > } >
where fifo has the following structure:
> typedef struct { > int8 bf[FIFO_LENGTH]; > int16 wr; // write pointer > int16 rd; // read pointer > bool ef; // fifo empty/full flag > } _fifo;
The race condition is on the ef flag, since both read_from_fifo() and write_from_fifo() will set it to either FULL or empty. Uhmmm... I guess I need to revise this part of the code. I suddenly realize I'm lacking of a methodology to write code which is interruptable, maybe some of you can point me to some good reference.
On 12-07-03 18:01 , alb wrote:
> On 7/3/2012 1:16 PM, alb wrote: > [...] >>> The fifo is shared between main and the ISR. You are not preventing the >>> ISR triggering while inserting more data into the fifo. This would >>> result in a race condition. >> >> the main() is only writing to the fifo and the isr_sport() only reading >> - except at the beginning of transmission - so why this should result in >> a race condition? >> >> In the write_to_fifo() I check if the fifo is full and if not I then >> write to it, incrementing the write pointer. In the read_from_fifo() I >> check if the fifo is empty and if not read from it, incrementing the >> read pointer. Maybe I'm missing something here. >> > > I think I was really missing something here!!! These are the > write_to_fifo() and read_from_fifo() functions:
[ elided code ] Without analysing that code in detail, at first glance it does look as if it needs to be protected against concurrent access from several threads/tasks/ISRs.
> where fifo has the following structure: > >> typedef struct { >> int8 bf[FIFO_LENGTH]; >> int16 wr; // write pointer >> int16 rd; // read pointer >> bool ef; // fifo empty/full flag >> } _fifo; > > The race condition is on the ef flag, since both read_from_fifo() and > write_from_fifo() will set it to either FULL or empty.
How can a "bool" variable have the values FULL or EMPTY? That contradicts the meaning of "bool". (Yes, I know that "bool" is just a fiction in traditional C, but that does not make it logically correct :-)) And what about the case when the FIFO is neither FULL, nor EMPTY, say it is about half-full? I like to keep a "count" field in FIFOs. Empty is then count == 0 and full is count == FIFO_LENGTH.
> Uhmmm... I guess I need to revise this part of the code.
At least for making the FIFO usable by concurrent threads.
> I suddenly realize I'm lacking of a methodology to write code which is > interruptable, maybe some of you can point me to some good reference.
The traditional way, with mutually exclusive critical sections: http://en.wikipedia.org/wiki/Mutual_exclusion. The new way (mainly useful for multicore processors IMO), with lock-free data structures: http://en.wikipedia.org/wiki/Lock-free. Note that both methods need a known ordering of actions on variables, so they need "volatile", or something like that (plus, in the more complex processors, memory barriers, etc.) If you were using an RT kernel, it would provide some of this stuff, usually at least interrupt-safe FIFO queues. -- Niklas Holsti Tidorum Ltd niklas holsti tidorum fi . @ .
On 12-07-03 13:17 , alb wrote:
> On 7/2/2012 6:31 PM, Niklas Holsti wrote: >> ... >> Since the serial port is now transmitting, isr_sport() will be called >> when the byte is transmitted and the serial-port interrupt occurs. In >> this sense, main() and isr_sport() are concurrent at this point. If you >> don't really know how fast things go (because of other interrupts or >> delays that can happen), either main() or isr_sport() can reach its >> assignment to "state" first. That is a race. > > Got it. I was indeed assuming that time between write_to_sport() and > state = TRANSMITTING was small enough to take allow the first > transmission, but if something else delays the execution of the 'state = > TRANSMITTING' instruction I may run in the race problem.
I would say that in that case the race may be won by isr_sport(), instead of (as you have assumed) the main() thread. This is just to emphasize that (IMO at least) the existence (definition) of the race condition depends only on the possible sequences of actions in the various threads, and not on their relative execution speeds. If you know the relative execution speeds, you can argue that you know which thread will win the race, and in that case the race condition can be left in the code without risk of errors (until someone changes the execution speeds ands forgets to re-examine the race...)
> Indeed I think the hardware has a register to store the data and another > to shift it out, hence as soon as the register is free to accept a new > byte the interrupt will be asserted. This means that as soon as the > first byte is written the interrupt is asserted (on next cycle), causing > isr_sport() to run.
In that case, the only reason why the code works at all (even without the timer interrupt) is the relatively long latency for interrupt response in the 21020, giving main() time to assign state = TRANSMITTING even while the serial interrupt is asserted.
>> Also, the 21020 delays interrupt responses for some multi-cycle >> instructions, including branch. Since the statement "state = >> TRANSMITTING" is the last statement in a loop body, it is quite likely >> that the compiler puts it in a delay slot of the "return to loop head" >> branch. This makes it more likely that the assigment is executed between >> the return from the timer-interrupt handler and the invocation of the >> serial-port interrupt handler, when interrupt nesting is disabled. >> > > uhm... interesting. I should maybe try to understand how the program > sequencer works.
"That way madness lies" :-) The 21020 program sequencer is a complex thing. When we built our 21020 WCET analyzer I had to try to understand how the zero-overhead, 6-deep-nested HW loop logic interacts with delayed branch instructions, with or without the loop-abort flag, and with special cases for loops that have few instructions and loops that have few iterations. Yuck.
>> What is write_to_sport()? Is is a macro or a function? If it is a macro >> (or a possibly in-lined function) of the form "portvar = byte", you >> should also declare "portvar" as volatile, to ensure that the order of >> assignments is as in the C source. > > indeed it is a memory mapped register defined as volatile and I declared > it such since I thought this is the "way" to tell the compiler that the > content of this variable may change due to hardware operations.
Yes, "volatile" means a bit more than that. Unfortunately (but understandably), the C standard has no strict, formal definition. There are also reports that C compilers frequently miscompile code that uses "volatile" -- at least in the sense that the compiled code does not do what the programmers wanted it to do.
> I was not aware that non volatile variables could be freely reordered. > This will definitely screw up my logic, since I rely on things happening > with certain chronological order (which is maybe a bad practice).
I think most multi-thread stuff (including interrupt handling) must rely on some ordering of actions. But one should try to enforce the order by logical conditions, not by timing that depends on execution speeds. -- Niklas Holsti Tidorum Ltd niklas holsti tidorum fi . @ .
On 04/07/12 03:01, alb wrote:
> On 7/3/2012 1:16 PM, alb wrote: > [...] >>> The fifo is shared between main and the ISR. You are not preventing the >>> ISR triggering while inserting more data into the fifo. This would >>> result in a race condition. >> >> the main() is only writing to the fifo and the isr_sport() only reading >> - except at the beginning of transmission - so why this should result in >> a race condition? >> >> In the write_to_fifo() I check if the fifo is full and if not I then >> write to it, incrementing the write pointer. In the read_from_fifo() I >> check if the fifo is empty and if not read from it, incrementing the >> read pointer. Maybe I'm missing something here. >> > > I think I was really missing something here!!! These are the > write_to_fifo() and read_from_fifo() functions: > >> bool write_to_fifo(int8 d) { >> >> if ((fifo->wr != fifo->rd) || fifo->ef == EMPTY) { >> fifo->bf[fifo->wr] = d; >> fifo->wr = MOD(fifo->wr+1, DIM(fifo->bf)); >> if (fifo->wr == fifo->rd) fifo->ef = FULL; >> return TRUE; >> } >> >> return FALSE; >> } >> > >> bool read_from_fifo(int8 *d) { >> >> if ((fifo->wr != fifo->rd) || fifo->ef == FULL) { >> *d = fifo->bf[fifo->rd]; >> fifo->rd = MOD(fifo->rd+1, DIM(fifo->bf)); >> if (fifo->rd == f->wr) fifo->ef = EMPTY; >> return TRUE; >> } >> >> return FALSE; >> } >> > > where fifo has the following structure: > >> typedef struct { >> int8 bf[FIFO_LENGTH]; >> int16 wr; // write pointer >> int16 rd; // read pointer >> bool ef; // fifo empty/full flag >> } _fifo; > > The race condition is on the ef flag, since both read_from_fifo() and > write_from_fifo() will set it to either FULL or empty. > Uhmmm... I guess I need to revise this part of the code. > > I suddenly realize I'm lacking of a methodology to write code which is > interruptable, maybe some of you can point me to some good reference. >
Yes I was thinking of the pointers. write_to_fifo() could be half way through adjusting a pointer value when an interrupt driven read_from_fifo() comes along and also adjusts the pointer values. Maybe try something like while(1) { temp_disable_sport_interrupt(); write_to_fifo(data); reenable_sport_interrupt(); if (state != TRANSMITTING) {
On 04/07/12 17:33, # wrote:
> On 04/07/12 03:01, alb wrote: >> On 7/3/2012 1:16 PM, alb wrote: >> [...] >>>> The fifo is shared between main and the ISR. You are not preventing the >>>> ISR triggering while inserting more data into the fifo. This would >>>> result in a race condition. >>> >>> the main() is only writing to the fifo and the isr_sport() only reading >>> - except at the beginning of transmission - so why this should result in >>> a race condition? >>> >>> In the write_to_fifo() I check if the fifo is full and if not I then >>> write to it, incrementing the write pointer. In the read_from_fifo() I >>> check if the fifo is empty and if not read from it, incrementing the >>> read pointer. Maybe I'm missing something here. >>> >> >> I think I was really missing something here!!! These are the >> write_to_fifo() and read_from_fifo() functions: >> >>> bool write_to_fifo(int8 d) { >>> >>> if ((fifo->wr != fifo->rd) || fifo->ef == EMPTY) { >>> fifo->bf[fifo->wr] = d; >>> fifo->wr = MOD(fifo->wr+1, DIM(fifo->bf)); >>> if (fifo->wr == fifo->rd) fifo->ef = FULL; >>> return TRUE; >>> } >>> >>> return FALSE; >>> } >>> >> >>> bool read_from_fifo(int8 *d) { >>> >>> if ((fifo->wr != fifo->rd) || fifo->ef == FULL) { >>> *d = fifo->bf[fifo->rd]; >>> fifo->rd = MOD(fifo->rd+1, DIM(fifo->bf)); >>> if (fifo->rd == f->wr) fifo->ef = EMPTY; >>> return TRUE; >>> } >>> >>> return FALSE; >>> } >>> >> >> where fifo has the following structure: >> >>> typedef struct { >>> int8 bf[FIFO_LENGTH]; >>> int16 wr; // write pointer >>> int16 rd; // read pointer >>> bool ef; // fifo empty/full flag >>> } _fifo; >> >> The race condition is on the ef flag, since both read_from_fifo() and >> write_from_fifo() will set it to either FULL or empty. >> Uhmmm... I guess I need to revise this part of the code. >> >> I suddenly realize I'm lacking of a methodology to write code which is >> interruptable, maybe some of you can point me to some good reference. >> > > Yes I was thinking of the pointers. > write_to_fifo() could be half way through adjusting a pointer value when > an interrupt driven read_from_fifo() comes along and also adjusts the > pointer values. >
Just looked at your code An interrupt could change the value of rd while evaluating this line if (fifo->wr == fifo->rd) fifo->ef = FULL; The value of wr may be partly incremented when an interrupt tries to evaluate this line if (fifo->rd == f->wr) fifo->ef = EMPTY;
"alb" <alessandro.basili@cern.ch> wrote in message
news:a5b1h6FkgiU1@mid.individual.net...
> Dear all, > > I would like to understand more about nested interrupts since it looks > to me they break my code and I'm not sure why. > > I have three 'items' in my software: a serial port, a fifo and a timer. > The serial port is an external hardware that sends back an interrupt > when it's free to send another byte, so I decided to have an interrupt > service routine (ISR) that reads from the fifo and puts the byte into > the serial port. With this choice the main program should start the > first byte transmission from the fifo and then everything is handled by > the ISR. Therefore the code looks like this: > > > int main() { > > ... > > while(1) { > > write_to_fifo(data); > > if (state != TRANSMITTING) { > > if (read_from_fifo(&byte) == OK) { > > write_to_sport(byte); > > state = TRANSMITTING; > > } > > } > > } > > } > > the variable 'state' is needed in order to prevent the main to write > into the serial port while the ISR is already doing this. > The ISR would look like the following:
The code above is flawed. Your main should not write to the sport at all. If no transmission is in progress, the TX interrupt of sport should be disabled. If you write something to the fifo, you just enable the TX interrupt, nothing more. At that time, the sport TX buffers are empty and an interrupt will occur immediately, reading a byte from the fifo and writing it to sport. When the next interrupt occurs, the next byte will be read from the fifo and written to the sport. If the fifo is empty when an interrupt occurs, the interrupt handler should disable the TX interrupt and exit. Meindert
On Jul 4, 9:12=A0am, "Meindert Sprang" <m...@NOJUNKcustomORSPAMware.nl>
wrote:
> "alb" <alessandro.bas...@cern.ch> wrote in message > > news:a5b1h6FkgiU1@mid.individual.net... > > > > > > > Dear all, > > > I would like to understand more about nested interrupts since it looks > > to me they break my code and I'm not sure why. > > > I have three 'items' in my software: a serial port, a fifo and a timer. > > The serial port is an external hardware that sends back an interrupt > > when it's free to send another byte, so I decided to have an interrupt > > service routine (ISR) that reads from the fifo and puts the byte into > > the serial port. With this choice the main program should start the > > first byte transmission from the fifo and then everything is handled by > > the ISR. Therefore the code looks like this: > > > > int main() { > > > ... > > > =A0 while(1) { > > > =A0 =A0 write_to_fifo(data); > > > =A0 =A0 if (state !=3D TRANSMITTING) { > > > =A0 =A0 =A0 if (read_from_fifo(&byte) =3D=3D OK) { > > > =A0 =A0 =A0 =A0 write_to_sport(byte); > > > =A0 =A0 =A0 =A0 state =3D TRANSMITTING; > > > =A0 =A0 =A0 } > > > =A0 =A0 } > > > =A0 } > > > } > > > the variable 'state' is needed in order to prevent the main to write > > into the serial port while the ISR is already doing this. > > The ISR would look like the following: > > The code above is flawed. Your main should not write to the sport at all.=
If
> no transmission is in progress, the TX interrupt of sport should be > disabled. If you write something to the fifo, you just enable the TX > interrupt, nothing more. At that time, the sport TX buffers are empty and=
an
> interrupt will occur immediately, reading a byte from the fifo and writin=
g
> it to sport. When the next interrupt occurs, the next byte will be read f=
rom
> the fifo and written to the sport. If the fifo is empty when an interrupt > occurs, the interrupt handler should disable the TX interrupt and exit. > > Meindert- Hide quoted text - > > - Show quoted text -
This is not true for all processors. Some of them have edge triggered interrupts the only interrupt when the transmitter finishes transmission.
On 7/3/2012 5:48 PM, Niklas Holsti wrote:
> On 12-07-03 18:01 , alb wrote: >> On 7/3/2012 1:16 PM, alb wrote: >> [...] >>>> The fifo is shared between main and the ISR. You are not preventing the >>>> ISR triggering while inserting more data into the fifo. This would >>>> result in a race condition. >>> >>> the main() is only writing to the fifo and the isr_sport() only reading >>> - except at the beginning of transmission - so why this should result in >>> a race condition? >>> >>> In the write_to_fifo() I check if the fifo is full and if not I then >>> write to it, incrementing the write pointer. In the read_from_fifo() I >>> check if the fifo is empty and if not read from it, incrementing the >>> read pointer. Maybe I'm missing something here. >>> >> >> I think I was really missing something here!!! These are the >> write_to_fifo() and read_from_fifo() functions: > > [ elided code ] > > Without analysing that code in detail, at first glance it does look as > if it needs to be protected against concurrent access from several > threads/tasks/ISRs.
I realized that. Here I can only think about disabling the interrupt temporarily, since interrupts will be still latched and servicing resumed on interrupt enabling. Otherwise I would need to invent some 'locking' mechanism with possibility to postpone a write/read to/from the fifo.
> >> where fifo has the following structure: >> >>> typedef struct { >>> int8 bf[FIFO_LENGTH]; >>> int16 wr; // write pointer >>> int16 rd; // read pointer >>> bool ef; // fifo empty/full flag >>> } _fifo; >> >> The race condition is on the ef flag, since both read_from_fifo() and >> write_from_fifo() will set it to either FULL or empty. > > How can a "bool" variable have the values FULL or EMPTY? That > contradicts the meaning of "bool". (Yes, I know that "bool" is just a > fiction in traditional C, but that does not make it logically correct :-))
A fifo is fully described with a read pointer, a write pointer, the size and a bit to distinguish EMPTY/FULL when pointers are the same. I do not see any logical flow here.
> > And what about the case when the FIFO is neither FULL, nor EMPTY, say it > is about half-full?
if (wr - rd != 0) fifo is neither empty nor full.
>> I suddenly realize I'm lacking of a methodology to write code which is >> interruptable, maybe some of you can point me to some good reference. > > The traditional way, with mutually exclusive critical sections: > http://en.wikipedia.org/wiki/Mutual_exclusion. > > The new way (mainly useful for multicore processors IMO), with lock-free > data structures: http://en.wikipedia.org/wiki/Lock-free. >
Thanks for the pointers!
> Note that both methods need a known ordering of actions on variables, so > they need "volatile", or something like that (plus, in the more complex > processors, memory barriers, etc.) If you were using an RT kernel, it > would provide some of this stuff, usually at least interrupt-safe FIFO > queues. >
Certainly an RTOS will facilitate some of these stuff, but I'm afraid I will bump in several other problems, considering that there's very little activity on this type of DSP (after all is a legacy DSP).