How to manipulate a receiving buffer

i am writting problem to receive data from a interrupt-driven serial
port, and i think my code will always have change of lossing data.

in the receving ISR, i get a byte from a register and put it into a
buffer,

static unsigned char buff[ BUFF_SIZE ];
static buff_len = 0;

__interrupt  on_byte_in( void )
{
       if ( buffer_len < sizeof( buff ) )
           buffer[ buffer_len++ ] = the_byte_in_the_register;
}

another function used to fetch the buffer data,

flush_receiving_buff( void )
{
   for ( size_t i = 0; i < buff_len; ++i ) {
      process the buff[ i ];
   buff_len = 0;                         ------------- (a)
}

my question is that, during the execution of statement (a), one or
more new bytes might come in and code above will lost them.  if you
folks see the problem, can you please suggest me a solution? i believe
the question is very basic and must have been well resolved.

thanks in advance.

--
woody

> __interrupt  on_byte_in( void )
> {
>       if ( buffer_len < sizeof( buff ) )
>           buffer[ buffer_len++ ] = the_byte_in_the_register;
> }
>
> flush_receiving_buff( void )
> {
>   for ( size_t i = 0; i < buff_len; ++i ) {
>      process the buff[ i ];
>   buff_len = 0;                         ------------- (a)
> }
>
> my question is that, during the execution of statement (a), one or
> more new bytes might come in and code above will lost them.  if you
> folks see the problem, can you please suggest me a solution?

Yes, it's basic.  You'll need to disable your receive interrupt during 
statement (a) execution.  Doing so will prevent the ISR from executing and 
changing buff_len while your flush function is also changing it.

It falls under the topic of "critical sections" of code execution.

JJS

On 4=D4=C23=C8=D5, =CF=C2=CE=E711=CA=B111=B7=D6, "John Speth" <johnsp...@ya=
hoo.com> wrote:
> > __interrupt  on_byte_in( void )
> > {
> >       if ( buffer_len < sizeof( buff ) )
> >           buffer[ buffer_len++ ] =3D the_byte_in_the_register;
> > }
>
> > flush_receiving_buff( void )
> > {
> >   for ( size_t i =3D 0; i < buff_len; ++i ) {
> >      process the buff[ i ];
> >   buff_len =3D 0;                         ------------- (a)
> > }
>
> > my question is that, during the execution of statement (a), one or
> > more new bytes might come in and code above will lost them.  if you
> > folks see the problem, can you please suggest me a solution?
>
> Yes, it's basic.  You'll need to disable your receive interrupt during
> statement (a) execution.  Doing so will prevent the ISR from executing and
> changing buff_len while your flush function is also changing it.
>
> It falls under the topic of "critical sections" of code execution.
>
> JJS

after an interrupt was disabled during the execuation of (a) by the
means you described, will it pop up again after the enable-interrupt
statement? if not, i think the problem is still not resolved.

thanks.

On Apr 3, 5:11 pm, "John Speth" <johnsp...@yahoo.com> wrote:
> > __interrupt  on_byte_in( void )
> > {
> >       if ( buffer_len < sizeof( buff ) )
> >           buffer[ buffer_len++ ] = the_byte_in_the_register;
> > }
>
> > flush_receiving_buff( void )
> > {
> >   for ( size_t i = 0; i < buff_len; ++i ) {
> >      process the buff[ i ];
> >   buff_len = 0;                         ------------- (a)
> > }
>
> > my question is that, during the execution of statement (a), one or
> > more new bytes might come in and code above will lost them.  if you
> > folks see the problem, can you please suggest me a solution?
>
> Yes, it's basic.  You'll need to disable your receive interrupt during
> statement (a) execution.  Doing so will prevent the ISR from executing and
> changing buff_len while your flush function is also changing it.

It would also be a good idea to declare 'buff_len' as volatile.

Alternatively, you could use something similar to this, and leave
interrupts enabled:

static unsigned char buff[ BUFF_SIZE ];
static volatile unsigned char buff_head;
static unsigned char buff_tail;

__interrupt  on_byte_in( void )
{
    if ( (unsigned char) (buff_head - buff_tail) < BUFF_SIZE )
        buff[ buff_head++  % BUFF_SIZE ] = the_byte_in_the_register;

}

flush_receiving_buff( void )
{
    while( buff_head != buff_tail )
        process the buff[ buff_tail++ % BUFF_SIZE  ];
}

This works only if BUFF_SIZE < 256. Powers of two are recommended to
avoid modulo operation. If your compiler doesn't optimize away the
modulo, rewrite it as '& (BUFF_SIZE-1)'. For larger BUFF_SIZE, use a
bigger type.

On 4=D4=C24=C8=D5, =C9=CF=CE=E712=CA=B106=B7=D6, "Arlet" <usene...@c-scape.=
nl> wrote:
> On Apr 3, 5:11 pm, "John Speth" <johnsp...@yahoo.com> wrote:
>
>
>
> > > __interrupt  on_byte_in( void )
> > > {
> > >       if ( buffer_len < sizeof( buff ) )
> > >           buffer[ buffer_len++ ] =3D the_byte_in_the_register;
> > > }
>
> > > flush_receiving_buff( void )
> > > {
> > >   for ( size_t i =3D 0; i < buff_len; ++i ) {
> > >      process the buff[ i ];
> > >   buff_len =3D 0;                         ------------- (a)
> > > }
>
> > > my question is that, during the execution of statement (a), one or
> > > more new bytes might come in and code above will lost them.  if you
> > > folks see the problem, can you please suggest me a solution?
>
> > Yes, it's basic.  You'll need to disable your receive interrupt during
> > statement (a) execution.  Doing so will prevent the ISR from executing =
and
> > changing buff_len while your flush function is also changing it.
>
> It would also be a good idea to declare 'buff_len' as volatile.
>
> Alternatively, you could use something similar to this, and leave
> interrupts enabled:
>
> static unsigned char buff[ BUFF_SIZE ];
> static volatile unsigned char buff_head;
> static unsigned char buff_tail;
>
> __interrupt  on_byte_in( void )
> {
>     if ( (unsigned char) (buff_head - buff_tail) < BUFF_SIZE )
>         buff[ buff_head++  % BUFF_SIZE ] =3D the_byte_in_the_register;
>
> }
>
> flush_receiving_buff( void )
> {
>     while( buff_head !=3D buff_tail )
>         process the buff[ buff_tail++ % BUFF_SIZE  ];
>
> }
>
> This works only if BUFF_SIZE < 256. Powers of two are recommended to
> avoid modulo operation. If your compiler doesn't optimize away the
> modulo, rewrite it as '& (BUFF_SIZE-1)'. For larger BUFF_SIZE, use a
> bigger type.

Arlet,

1, does you ment to say, by letting the on_byte_in wrte only one
variable and letting the flush_receiving_buff write only another
variable, hence the code above don't need to be pretected into a
disable/enable interrupt section?
2, what's the exact functionality of keyword 'volatile', what's the
different if don't use it?
3, can you please answer me another question: will a disabled
interrupt pop up again after it is enabled?

thank you very much.

-
woody

On Apr 3, 7:13 pm, "Steven Woody" <narkewo...@gmail.com> wrote:
> On 4=D4=C24=C8=D5, =C9=CF=CE=E712=CA=B106=B7=D6, "Arlet" <usene...@c-scap=
e=2Enl> wrote:
>
> > On Apr 3, 5:11 pm, "John Speth" <johnsp...@yahoo.com> wrote:
>
> > > > __interrupt  on_byte_in( void )
> > > > {
> > > >       if ( buffer_len < sizeof( buff ) )
> > > >           buffer[ buffer_len++ ] =3D the_byte_in_the_register;
> > > > }
>
> > > > flush_receiving_buff( void )
> > > > {
> > > >   for ( size_t i =3D 0; i < buff_len; ++i ) {
> > > >      process the buff[ i ];
> > > >   buff_len =3D 0;                         ------------- (a)
> > > > }
>
> > > > my question is that, during the execution of statement (a), one or
> > > > more new bytes might come in and code above will lost them.  if you
> > > > folks see the problem, can you please suggest me a solution?
>
> > > Yes, it's basic.  You'll need to disable your receive interrupt during
> > > statement (a) execution.  Doing so will prevent the ISR from executin=
g and
> > > changing buff_len while your flush function is also changing it.
>
> > It would also be a good idea to declare 'buff_len' as volatile.
>
> > Alternatively, you could use something similar to this, and leave
> > interrupts enabled:
>
> > static unsigned char buff[ BUFF_SIZE ];
> > static volatile unsigned char buff_head;
> > static unsigned char buff_tail;
>
> > __interrupt  on_byte_in( void )
> > {
> >     if ( (unsigned char) (buff_head - buff_tail) < BUFF_SIZE )
> >         buff[ buff_head++  % BUFF_SIZE ] =3D the_byte_in_the_register;
>
> > }
>
> > flush_receiving_buff( void )
> > {
> >     while( buff_head !=3D buff_tail )
> >         process the buff[ buff_tail++ % BUFF_SIZE  ];
>
> > }
>
> > This works only if BUFF_SIZE < 256. Powers of two are recommended to
> > avoid modulo operation. If your compiler doesn't optimize away the
> > modulo, rewrite it as '& (BUFF_SIZE-1)'. For larger BUFF_SIZE, use a
> > bigger type.
>
> Arlet,
>
> 1, does you ment to say, by letting the on_byte_in wrte only one
> variable and letting the flush_receiving_buff write only another
> variable, hence the code above don't need to be pretected into a
> disable/enable interrupt section?

True. Instead of having a single 'length', this code uses two
variables, and the 'length' is the difference between them. The
interrupt handler only changes head, and the main code only changes
tail, so there's no conflict. It does require atomic reads, though. On
an 8-bit architecture, where the CPU doesn't have a 16-bit read,
you'll be restricted to 8-bit head/tail counters. For most
applications on an 8-bit platform this is sufficient for a UART
buffer.

> 2, what's the exact functionality of keyword 'volatile', what's the
> different if don't use it?

'volatile' tells the compiler not to optimize away memory references.
For example in the while loop:

    while( buff_head !=3D buff_tail )

Without the volatile, the compiler could assume that buff_head never
changes, and may therefore load the value in a register before the
while loop, and never read it again. With the 'volatile' keyword, the
buff_head variable will be read from memory before every comparison.
Since buff_tail is not modified in the interrupt handler, it does not
need to be volatile.

> 3, can you please answer me another question: will a disabled
> interrupt pop up again after it is enabled?

On any sane architecture, yes.

On 4=D4=C24=C8=D5, =C9=CF=CE=E71=CA=B137=B7=D6, "Arlet" <usene...@c-scape.n=
l> wrote:
> On Apr 3, 7:13 pm, "Steven Woody" <narkewo...@gmail.com> wrote:
>
>
>
> > On 4=D4=C24=C8=D5, =C9=CF=CE=E712=CA=B106=B7=D6, "Arlet" <usene...@c-sc=
ape.nl> wrote:
>
> > > On Apr 3, 5:11 pm, "John Speth" <johnsp...@yahoo.com> wrote:
>
> > > > > __interrupt  on_byte_in( void )
> > > > > {
> > > > >       if ( buffer_len < sizeof( buff ) )
> > > > >           buffer[ buffer_len++ ] =3D the_byte_in_the_register;
> > > > > }
>
> > > > > flush_receiving_buff( void )
> > > > > {
> > > > >   for ( size_t i =3D 0; i < buff_len; ++i ) {
> > > > >      process the buff[ i ];
> > > > >   buff_len =3D 0;                         ------------- (a)
> > > > > }
>
> > > > > my question is that, during the execution of statement (a), one or
> > > > > more new bytes might come in and code above will lost them.  if y=
ou
> > > > > folks see the problem, can you please suggest me a solution?
>
> > > > Yes, it's basic.  You'll need to disable your receive interrupt dur=
ing
> > > > statement (a) execution.  Doing so will prevent the ISR from execut=
ing and
> > > > changing buff_len while your flush function is also changing it.
>
> > > It would also be a good idea to declare 'buff_len' as volatile.
>
> > > Alternatively, you could use something similar to this, and leave
> > > interrupts enabled:
>
> > > static unsigned char buff[ BUFF_SIZE ];
> > > static volatile unsigned char buff_head;
> > > static unsigned char buff_tail;
>
> > > __interrupt  on_byte_in( void )
> > > {
> > >     if ( (unsigned char) (buff_head - buff_tail) < BUFF_SIZE )
> > >         buff[ buff_head++  % BUFF_SIZE ] =3D the_byte_in_the_register;
>
> > > }
>
> > > flush_receiving_buff( void )
> > > {
> > >     while( buff_head !=3D buff_tail )
> > >         process the buff[ buff_tail++ % BUFF_SIZE  ];
>
> > > }
>
> > > This works only if BUFF_SIZE < 256. Powers of two are recommended to
> > > avoid modulo operation. If your compiler doesn't optimize away the
> > > modulo, rewrite it as '& (BUFF_SIZE-1)'. For larger BUFF_SIZE, use a
> > > bigger type.
>
> > Arlet,
>
> > 1, does you ment to say, by letting the on_byte_in wrte only one
> > variable and letting the flush_receiving_buff write only another
> > variable, hence the code above don't need to be pretected into a
> > disable/enable interrupt section?
>
> True. Instead of having a single 'length', this code uses two
> variables, and the 'length' is the difference between them. The
> interrupt handler only changes head, and the main code only changes
> tail, so there's no conflict. It does require atomic reads, though. On
> an 8-bit architecture, where the CPU doesn't have a 16-bit read,
> you'll be restricted to 8-bit head/tail counters. For most
> applications on an 8-bit platform this is sufficient for a UART
> buffer.

thanks for you answer. and, even  in a 8-bit architecture, one need to
receive a network packet which may longer than 255 bytes :-(  in the
case, data might be lost. am i right?

>
> > 2, what's the exact functionality of keyword 'volatile', what's the
> > different if don't use it?
>
> 'volatile' tells the compiler not to optimize away memory references.
> For example in the while loop:
>
>     while( buff_head !=3D buff_tail )
>
> Without the volatile, the compiler could assume that buff_head never
> changes, and may therefore load the value in a register before the
> while loop, and never read it again. With the 'volatile' keyword, the
> buff_head variable will be read from memory before every comparison.
> Since buff_tail is not modified in the interrupt handler, it does not
> need to be volatile.

thank you very much!

>
> > 3, can you please answer me another question: will a disabled
> > interrupt pop up again after it is enabled?
>
> On any sane architecture, yes.

so, if in a disable/enable interrupt secion, there are two bytes comes
in, the first byte will be lost since the interrupt will appear only
once after the enable interrupt instruction. am i righ? ( supposing i
am talking about a UART with only one byte FIFO ).

-
woody

Steven Woody wrote:
> "John Speth" <johnsp...@yahoo.com> wrote:
>>
... snip ...
>>
>>> flush_receiving_buff( void )
>>> {
>>>   for ( size_t i = 0; i < buff_len; ++i ) {
>>>      process the buff[ i ];
>>>   buff_len = 0;                         ------------- (a)
>>> }
>>
>>> my question is that, during the execution of statement (a), one
>>> or more new bytes might come in and code above will lost them. 
>>> if you folks see the problem, can you please suggest me a
>>> solution?
>>
>> Yes, it's basic.  You'll need to disable your receive interrupt
>> during statement (a) execution.  Doing so will prevent the ISR
>> from executing and changing buff_len while your flush function is
>> also changing it.
>>
>> It falls under the topic of "critical sections" of code execution.
> 
> after an interrupt was disabled during the execuation of (a) by
> the means you described, will it pop up again after the enable-
> interrupt statement? if not, i think the problem is still not
> resolved.

Then you control things at a finer level.

   i = 0;
   WHILE buffernotempty DO BEGIN
     istate = disableints;
     mybuff[i] = getbytefrombuffer;
     enableints(istate);
     i = i + 1;
     END;
      
-- 
Chuck F (cbfalconer at maineline dot net)
   Available for consulting/temporary embedded and systems.
   <http://cbfalconer.home.att.net>



-- 
Posted via a free Usenet account from http://www.teranews.com

On Apr 3, 8:45 pm, "Steven Woody" <narkewo...@gmail.com> wrote:

> > > > Alternatively, you could use something similar to this, and leave
> > > > interrupts enabled:
>
> > > > static unsigned char buff[ BUFF_SIZE ];
> > > > static volatile unsigned char buff_head;
> > > > static unsigned char buff_tail;
>
> > > > __interrupt  on_byte_in( void )
> > > > {
> > > >     if ( (unsigned char) (buff_head - buff_tail) < BUFF_SIZE )
> > > >         buff[ buff_head++  % BUFF_SIZE ] = the_byte_in_the_register;
>
> > > > }
>
> > > > flush_receiving_buff( void )
> > > > {
> > > >     while( buff_head != buff_tail )
> > > >         process the buff[ buff_tail++ % BUFF_SIZE  ];
>
> > > > }
>
> > > > This works only if BUFF_SIZE < 256. Powers of two are recommended to
> > > > avoid modulo operation. If your compiler doesn't optimize away the
> > > > modulo, rewrite it as '& (BUFF_SIZE-1)'. For larger BUFF_SIZE, use a
> > > > bigger type.
>
> > > Arlet,
>
> > > 1, does you ment to say, by letting the on_byte_in wrte only one
> > > variable and letting the flush_receiving_buff write only another
> > > variable, hence the code above don't need to be pretected into a
> > > disable/enable interrupt section?
>
> > True. Instead of having a single 'length', this code uses two
> > variables, and the 'length' is the difference between them. The
> > interrupt handler only changes head, and the main code only changes
> > tail, so there's no conflict. It does require atomic reads, though. On
> > an 8-bit architecture, where the CPU doesn't have a 16-bit read,
> > you'll be restricted to 8-bit head/tail counters. For most
> > applications on an 8-bit platform this is sufficient for a UART
> > buffer.
>
> thanks for you answer. and, even  in a 8-bit architecture, one need to
> receive a network packet which may longer than 255 bytes :-(  in the
> case, data might be lost. am i right?

You could have a 2-part solution. The first would be a smallish FIFO,
like above, big enough to store all the data that comes in while your
main code is busy with something else. In a second step, you pull all
the data out of the FIFO, and put it in a larger buffer for
processing. Only the first FIFO needs to worry about critical sections
and interrupts.

If you don't want to do that, you'll need to disable interrupts in
some way, yes.

> > > 3, can you please answer me another question: will a disabled
> > > interrupt pop up again after it is enabled?
>
> > On any sane architecture, yes.
>
> so, if in a disable/enable interrupt secion, there are two bytes comes
> in, the first byte will be lost since the interrupt will appear only
> once after the enable interrupt instruction. am i righ? ( supposing i
> am talking about a UART with only one byte FIFO ).

Yes, if interrupts are disabled for an extended time, you may lose
data from your peripheral.

Steven Woody wrote:
> On 4&#4294967295;&#4294967295;4&#4294967295;&#4294967295;, &#4294967295;&#4294967295;&#4294967295;&#4294967295;1&#689;37&#4294967295;&#4294967295;, "Arlet" <usene...@c-scape.nl> wrote:
>> On Apr 3, 7:13 pm, "Steven Woody" <narkewo...@gmail.com> wrote:
>>
>>
>>
>>> On 4&#4294967295;&#4294967295;4&#4294967295;&#4294967295;, &#4294967295;&#4294967295;&#4294967295;&#4294967295;12&#689;06&#4294967295;&#4294967295;, "Arlet" <usene...@c-scape.nl> wrote:
>>>> On Apr 3, 5:11 pm, "John Speth" <johnsp...@yahoo.com> wrote:
>>>>>> __interrupt  on_byte_in( void )
>>>>>> {
>>>>>>       if ( buffer_len < sizeof( buff ) )
>>>>>>           buffer[ buffer_len++ ] = the_byte_in_the_register;
>>>>>> }
>>>>>> flush_receiving_buff( void )
>>>>>> {
>>>>>>   for ( size_t i = 0; i < buff_len; ++i ) {
>>>>>>      process the buff[ i ];
>>>>>>   buff_len = 0;                         ------------- (a)
>>>>>> }
>>>>>> my question is that, during the execution of statement (a), one or
>>>>>> more new bytes might come in and code above will lost them.  if you
>>>>>> folks see the problem, can you please suggest me a solution?
>>>>> Yes, it's basic.  You'll need to disable your receive interrupt during
>>>>> statement (a) execution.  Doing so will prevent the ISR from executing and
>>>>> changing buff_len while your flush function is also changing it.
>>>> It would also be a good idea to declare 'buff_len' as volatile.
>>>> Alternatively, you could use something similar to this, and leave
>>>> interrupts enabled:
>>>> static unsigned char buff[ BUFF_SIZE ];
>>>> static volatile unsigned char buff_head;
>>>> static unsigned char buff_tail;
>>>> __interrupt  on_byte_in( void )
>>>> {
>>>>     if ( (unsigned char) (buff_head - buff_tail) < BUFF_SIZE )
>>>>         buff[ buff_head++  % BUFF_SIZE ] = the_byte_in_the_register;
>>>> }
>>>> flush_receiving_buff( void )
>>>> {
>>>>     while( buff_head != buff_tail )
>>>>         process the buff[ buff_tail++ % BUFF_SIZE  ];
>>>> }
>>>> This works only if BUFF_SIZE < 256. Powers of two are recommended to
>>>> avoid modulo operation. If your compiler doesn't optimize away the
>>>> modulo, rewrite it as '& (BUFF_SIZE-1)'. For larger BUFF_SIZE, use a
>>>> bigger type.
>>> Arlet,
>>> 1, does you ment to say, by letting the on_byte_in wrte only one
>>> variable and letting the flush_receiving_buff write only another
>>> variable, hence the code above don't need to be pretected into a
>>> disable/enable interrupt section?
>> True. Instead of having a single 'length', this code uses two
>> variables, and the 'length' is the difference between them. The
>> interrupt handler only changes head, and the main code only changes
>> tail, so there's no conflict. It does require atomic reads, though. On
>> an 8-bit architecture, where the CPU doesn't have a 16-bit read,
>> you'll be restricted to 8-bit head/tail counters. For most
>> applications on an 8-bit platform this is sufficient for a UART
>> buffer.
> 
> thanks for you answer. and, even  in a 8-bit architecture, one need to
> receive a network packet which may longer than 255 bytes :-(  in the
> case, data might be lost. am i right?
> 
>>> 2, what's the exact functionality of keyword 'volatile', what's the
>>> different if don't use it?
>> 'volatile' tells the compiler not to optimize away memory references.
>> For example in the while loop:
>>
>>     while( buff_head != buff_tail )
>>
>> Without the volatile, the compiler could assume that buff_head never
>> changes, and may therefore load the value in a register before the
>> while loop, and never read it again. With the 'volatile' keyword, the
>> buff_head variable will be read from memory before every comparison.
>> Since buff_tail is not modified in the interrupt handler, it does not
>> need to be volatile.
> 
> thank you very much!
> 
>>> 3, can you please answer me another question: will a disabled
>>> interrupt pop up again after it is enabled?
>> On any sane architecture, yes.
> 
> so, if in a disable/enable interrupt secion, there are two bytes comes
> in, the first byte will be lost since the interrupt will appear only
> once after the enable interrupt instruction. am i righ? ( supposing i
> am talking about a UART with only one byte FIFO ).
> 
One thing wasn't clear from the original problem description: will the
fn. flush_receiving_buff() always clear the buffer to empty (for
example, if a complete packet is been received?
If so, you might consider using two receiving buffers, controlled by a
pointer. __interrupt  on_byte_in() will load the buffer indicated by the
pointer. Once a full packet is received, the pointer is switched to the
second buffer, & the first is processed at your leisure.
This is the classic double buffer setup. Of course, the actual switching
of the pointer may need protecting as a critical section (eg by
disabling interrupts).