upsid...@downunder.com ha scritto:
> In the interrupt service routine (which should be kept very simple to
> avoid blocking other interrupts) just use the idle timeout.
> 
> It is to the upper level logic to find out the actual framing, e.g. by
> knowing the structure of the expected response (such as ending in
> <LF>).

I implemented the mechanism you suggested: mark the end of a message
after a small timeout (4-5 characters frame length) without receiving
any character.
In this way, I will have a buffer with the complete message from the modem,
answer from a previous command or unsolicited code... most of the time.

During developing I noticed the following. When an incoming SMS arrives,
the modem sends an unsolicited message +CNMI to inform about the event.
This message could appear concatenated to an answer to a previous AT 
command. In other words, the modem doesn't insert a pause, even small, 
between the answer to a previous AT command and the +CNMI unsolicited 
message.
Detecting the start and stop of a message just using the timeout between
receiving characters is a broken approach: I could have a concatenated 
message (answer plus unsolicited).

I know the application layer can break this concatenated message
depending on the content and expected answer, knowing the previous AT
command send, but this approach will complicate the problem.
I hoped the modem gave the opportunity to the host microcontroller to
separate the messages at a low-level, without parsing the content, breaking
the stream of incoming characters in messages separated at the presence
of a long pause between characters. Unfortunately this approach doesn't 
work, at least with my modem (SIM900 of SimCom).

IMHO, if the application layer should be the only responsible to separate 
the stream of incoming characters in messages, parsing the content, the 
low-level timeout mechanism isn't needed at all. 
It's sufficient to have a FIFO buffer where the incoming characters 
will be pushed in ISR and popped in the application code.

On 2014-05-10, upsidedown@downunder.com <upsidedown@downunder.com> wrote:
> On Sat, 10 May 2014 09:15:37 +0200, pozz <pozzugno@gmail.com> wrote:
>
>>Il 09/05/2014 14:29, Simon Clubley ha scritto:
>>> I would have implemented this as a state machine in the upper layers of
>>> the code with clearly defined state transitions depending on (1) timeouts,
>>> (2) the responses you are waiting for and (3) unsolicited events such
>>> as ring.
>>>
>>> The next layer down would include a parser which extracts any printable
>>> text, on a per line basis, from the modem and either tokenises it or
>>> passes it as-is back up to the state machine.
>>
>>Even the "per line basis" parser approach doesn't work everytime.
>>The answer to the AT command that sends SMS isn't <CR><LF> as usual, but 
>>it is the 2-characters prompt "> ". The lower layer (the driver?) can't 
>>decide to wait for <CR><LF> before passing the incoming characters to 
>>the application layer.
>>
>>One possibility is to change at run-time the ending suffix: normally it 
>>is <CR><LF>, but it could be "> ". Anyway this suffix will be used only 
>>for answers to pass to the application layer. At the same time 
>>unsolicited messages (ending with <CR><LF>) could arrive and must be 
>>managed.
>>

In that case, my first try at implementing this would be to switch to
traditional per character parsing and tokenising the response. If you
see "> ", then that's one token and it can be pushed upstairs without
any wait.

If OTOH you see a alphanumeric character, you continue scanning until
you see a non-alphanumeric character, (although you may choose to treat
a space as part of the sequence and look for a non-printable character
instead).

>>
>>> If the state machine sees a ring while waiting for OK, it can decide
>>> if it wants to transition to a new state at that point or wait for
>>> the expected response (or timeout).
>>>
>>> That means in the event of RING followed on the next line by OK, your
>>> state machine would see two distinct responses from the modem.
>>
>>You're thinking of a lower level that is able to separate two messages, 
>>even if they are strictly *consecutive*: answer-unsolicited or 
>>unsolicited-answer.
>>
>>This is the starting question: what is the best method to separate 
>>incoming characters from modem into *distinct* messages in *all* 
>>possible situations (answers, unsolicited...)?
>><CR><LF> prefix/suffix could be a method, but in some situation the 
>>suffix changes.
>>
>>Are you thinking of a lower level that pass to the application layer 
>>*all* the messages: answers and unsolicited? IMHO it's not feasible. All 
>>the upper complex FSMs (there are many, one for sensing SMS, another for 
>>receiving SMS, another for making voice calls, another for making data 
>>calls, ...) should decide, in *every* state, what to do in case of 
>>unsolicited messages.
>>

The FSM should know what are the possible responses in each state, both
expected and unsolicited. I agree it can get complicated, but if you
are getting those messages you have to handle them somehow, even if
you transition back to the same state, thereby ignoring the message.

>>
>>> You would of course have to make sure your state machine always
>>> terminated any possible successful connect so you were not running up
>>> connect charges after recovery from any unexpected events.
>>>
>>> Is the above how others here would implement this or would you
>>> use a different approach ?
>
> Why bother with a fuss like that above. If your application layer
> doesn't understand the response, just resend the original command a
> few times until you get a valid response or declare the connection
> dead. 
>

It matters because you need to handle error recovery in a robust manner.
Your commands may have side effects, such as changing a configuration,
sending a message, or establishing a connection.

You can't just blindly send them multiple times. You might get away with
that in some circumstances, but not always.

There's enough lousy embedded software out there without us adding to
the problem. :-)

Simon.

-- 
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

On Sat, 10 May 2014 09:15:37 +0200, pozz <pozzugno@gmail.com> wrote:

>Il 09/05/2014 14:29, Simon Clubley ha scritto:
>> I would have implemented this as a state machine in the upper layers of
>> the code with clearly defined state transitions depending on (1) timeouts,
>> (2) the responses you are waiting for and (3) unsolicited events such
>> as ring.
>>
>> The next layer down would include a parser which extracts any printable
>> text, on a per line basis, from the modem and either tokenises it or
>> passes it as-is back up to the state machine.
>
>Even the "per line basis" parser approach doesn't work everytime.
>The answer to the AT command that sends SMS isn't <CR><LF> as usual, but 
>it is the 2-characters prompt "> ". The lower layer (the driver?) can't 
>decide to wait for <CR><LF> before passing the incoming characters to 
>the application layer.
>
>One possibility is to change at run-time the ending suffix: normally it 
>is <CR><LF>, but it could be "> ". Anyway this suffix will be used only 
>for answers to pass to the application layer. At the same time 
>unsolicited messages (ending with <CR><LF>) could arrive and must be 
>managed.
>
>
>> If the state machine sees a ring while waiting for OK, it can decide
>> if it wants to transition to a new state at that point or wait for
>> the expected response (or timeout).
>>
>> That means in the event of RING followed on the next line by OK, your
>> state machine would see two distinct responses from the modem.
>
>You're thinking of a lower level that is able to separate two messages, 
>even if they are strictly *consecutive*: answer-unsolicited or 
>unsolicited-answer.
>
>This is the starting question: what is the best method to separate 
>incoming characters from modem into *distinct* messages in *all* 
>possible situations (answers, unsolicited...)?
><CR><LF> prefix/suffix could be a method, but in some situation the 
>suffix changes.
>
>Are you thinking of a lower level that pass to the application layer 
>*all* the messages: answers and unsolicited? IMHO it's not feasible. All 
>the upper complex FSMs (there are many, one for sensing SMS, another for 
>receiving SMS, another for making voice calls, another for making data 
>calls, ...) should decide, in *every* state, what to do in case of 
>unsolicited messages.
>
>
>> You would of course have to make sure your state machine always
>> terminated any possible successful connect so you were not running up
>> connect charges after recovery from any unexpected events.
>>
>> Is the above how others here would implement this or would you
>> use a different approach ?

Why bother with a fuss like that above. If your application layer
doesn't understand the response, just resend the original command a
few times until you get a valid response or declare the connection
dead. 

For the resend interval use something odd like 3.13 or 2.71 seconds,
since if there are similar timers at the end, they would typically
expire after 1, 2 or 3 seconds :-).

On Sat, 10 May 2014 08:45:09 +0200, pozz <pozzugno@gmail.com> wrote:

>Il 09/05/2014 19:21, upsidedown@downunder.com ha scritto:
>>> I'm not considering full-duplex traffic, my example is half-duplex.
>>>
>>> 1. Host microcontroller sends completely the command AT<CR><LF>
>>> 2. Modem answers with <CR><LF>OK<CR><LF> (during this msg RING event occurs)
>>> 3. Modem transmits unsolicited message <CR><LF>RING<CR><LF>
>>
>> If the modem sends "RING" at the same time time as you start sending
>> ATxxxxx<CR>, then it is definitely full duplex. This is possible in
>> RS-232 and RS-422, but not in 2 wire RS-485.
>
>Yes, you're right. But my example above aren't full-duplex. AT command 
>is *completely* send out (step 1 is finished) when the modem starts 
>answering (step 2). And the modem *completely* answered with OK (step 2 
>is finished) when it starts emitting unsolicited RING message (step 3).
>
>This is definitely a half-duplex communication.
>
>
>>> My question was: does the modem waits between steps 2 and 3? If the internal
>>> firmware of the modem enqueue messages in a global output buffer, it is
>>> possible to have two consecutive messages without a delay between them.
>>
>> I have no idea how each modem on the market behaves.
>
>If the modem doesn't *guarantee* a minimum delay (4-5 characters time) 
>between two output messages, are you agree your method (separating 
>frames thanks to a delay) doesn't work?

In the interrupt service routine (which should be kept very simple to
avoid blocking other interrupts) just use the idle timeout.

It is to the upper level logic to find out the actual framing, e.g. by
knowing the structure of the expected response (such as ending in
<LF>).

If it can't make any sense, just resend the command after a while
(100-1000 ms). This works well in practice.

Il 09/05/2014 14:29, Simon Clubley ha scritto:
> I would have implemented this as a state machine in the upper layers of
> the code with clearly defined state transitions depending on (1) timeouts,
> (2) the responses you are waiting for and (3) unsolicited events such
> as ring.
>
> The next layer down would include a parser which extracts any printable
> text, on a per line basis, from the modem and either tokenises it or
> passes it as-is back up to the state machine.

Even the "per line basis" parser approach doesn't work everytime.
The answer to the AT command that sends SMS isn't <CR><LF> as usual, but 
it is the 2-characters prompt "> ". The lower layer (the driver?) can't 
decide to wait for <CR><LF> before passing the incoming characters to 
the application layer.

One possibility is to change at run-time the ending suffix: normally it 
is <CR><LF>, but it could be "> ". Anyway this suffix will be used only 
for answers to pass to the application layer. At the same time 
unsolicited messages (ending with <CR><LF>) could arrive and must be 
managed.


> If the state machine sees a ring while waiting for OK, it can decide
> if it wants to transition to a new state at that point or wait for
> the expected response (or timeout).
>
> That means in the event of RING followed on the next line by OK, your
> state machine would see two distinct responses from the modem.

You're thinking of a lower level that is able to separate two messages, 
even if they are strictly *consecutive*: answer-unsolicited or 
unsolicited-answer.

This is the starting question: what is the best method to separate 
incoming characters from modem into *distinct* messages in *all* 
possible situations (answers, unsolicited...)?
<CR><LF> prefix/suffix could be a method, but in some situation the 
suffix changes.

Are you thinking of a lower level that pass to the application layer 
*all* the messages: answers and unsolicited? IMHO it's not feasible. All 
the upper complex FSMs (there are many, one for sensing SMS, another for 
receiving SMS, another for making voice calls, another for making data 
calls, ...) should decide, in *every* state, what to do in case of 
unsolicited messages.


> You would of course have to make sure your state machine always
> terminated any possible successful connect so you were not running up
> connect charges after recovery from any unexpected events.
>
> Is the above how others here would implement this or would you
> use a different approach ?

Il 09/05/2014 15:03, R.Wieser ha scritto:
> Hello Pozzugno,
>
>> Indeed your imagination is wrong :-)
>
> Well, you're right about that humans could take a long time to finish typing
> a command, so the "from the begin" idea is out.  But I still do not think
> that there will be much chance (if any) that an unsolicited message will be
> send while another command is being executed (for multiple reasons).

Why not? I'm thinking how the embedded GSM modem firmware could work.
There are many threads running that can push output messages in a output 
queue.

One thread could manage the GSM network, so it receives signals that 
incoming call is in progress. This thread could push "RING" message in
the output buffer queue.

Another thread could manage the AT requests on the serial port and could 
push "OK" answers.

At a certain time, the output queue could contain two messages, emitted 
one after the other in a transmitting UART ISR.


>> It couldn't work. When I decide to send last char (even
>> after a long delay),  at the same time the modem could
>> decide to send an unsolicited message.
>
> You're right, stated like that it seems like an unsolvable "race condition"
> problem.  Nevertheless, it must have been solved time-and-again, otherwise
> simple old-style modem communication between computers would never have
> passed its beta stage. :-)

I agree with you and this is why I asked for a solution of this problem. 
I don't think the problem hasn't a solution, I'm quite sure I'm using a 
wrong method.


> Is it possible that there are few different unsolicited messages and by
> their fixed nature are fairly easy to recognise/discern from an expected
> result, and by that feat rather easily removed when its recieved before an
> expected reply ?

Indeed it seems the only possible solution, even if it isn't simple to 
implement. The receiving driver should detect a certain number of 
unsolicited messages and filter them out to the application layer that 
wants only the answers to commands. There many unsolicited messages:

   <CR><LF>RING<CR><LF>
   <CR><LF>CONNECT ...<CR><LF>
   <CR><LF>BUSY<CR><LF>
   <CR><LF>DIALTONE<CR><LF>
   <CR><LF>+CGMS...<CR><LF> (or something similar)

I have the possibility to implement *all* the possible unsolicited 
messages, but I don't think it is feasible because there are many and 
some of them couldln't be documented.

Alternatively I can capture and filter only the unsolicited messages I 
want to decode and analyze. The other unsolicited messages will not be 
recognized as such so they must be passed to the application layer.
The application should be ready to receive not only answers to commands 
it send, but also other messages that should be discarded because not 
useful unsolicited messages.

Il 09/05/2014 19:21, upsidedown@downunder.com ha scritto:
>> I'm not considering full-duplex traffic, my example is half-duplex.
>>
>> 1. Host microcontroller sends completely the command AT<CR><LF>
>> 2. Modem answers with <CR><LF>OK<CR><LF> (during this msg RING event occurs)
>> 3. Modem transmits unsolicited message <CR><LF>RING<CR><LF>
>
> If the modem sends "RING" at the same time time as you start sending
> ATxxxxx<CR>, then it is definitely full duplex. This is possible in
> RS-232 and RS-422, but not in 2 wire RS-485.

Yes, you're right. But my example above aren't full-duplex. AT command 
is *completely* send out (step 1 is finished) when the modem starts 
answering (step 2). And the modem *completely* answered with OK (step 2 
is finished) when it starts emitting unsolicited RING message (step 3).

This is definitely a half-duplex communication.


>> My question was: does the modem waits between steps 2 and 3? If the internal
>> firmware of the modem enqueue messages in a global output buffer, it is
>> possible to have two consecutive messages without a delay between them.
>
> I have no idea how each modem on the market behaves.

If the modem doesn't *guarantee* a minimum delay (4-5 characters time) 
between two output messages, are you agree your method (separating 
frames thanks to a delay) doesn't work?

On Fri, 9 May 2014 01:52:15 -0700 (PDT), giuseppe.modugno@gmail.com
wrote:

>upsid...@downunder.com ha scritto:
>> >This could work only if the modem *guarantees* a sufficient delay between 
>> >the last character of a message and the first character of the next message. 
>> >
>> >Consider the previous example (AT command from micro, OK answer from modem
>> >and immediately after RING unsolicited message). What happens if the RING
>> >event is generated *in the middle* of OK answer transmission? I think the
>> >modem firmware buffer the RING event and put out the unsolicited message as
>> >soon as the serial port is free, i.e. immediately after the last character
>> >of the OK answer. Do the modem firmware wait for some delay before putting
>> >out the next message?
>> >
>> >In case you're right, how could I know the amount of this delay, considering
>> >this behaviour isn't documented? It is very difficult to measure it, because
>> >I need to reproduce the events above... how? 
>> 
>> Typically, the AT command set is just half duplex request/response
>> traffic, so framing by idle periods is quite reliable. However, if
>> there are full duplex traffic, just read until next idle timeout. Let
>> the higher level logic figure out what is response to a request and
>> what is spontaneous "RING" messages.
>
>I'm not considering full-duplex traffic, my example is half-duplex.
>
>1. Host microcontroller sends completely the command AT<CR><LF>
>2. Modem answers with <CR><LF>OK<CR><LF> (during this msg RING event occurs)
>3. Modem transmits unsolicited message <CR><LF>RING<CR><LF>

If the modem sends "RING" at the same time time as you start sending
ATxxxxx<CR>, then it is definitely full duplex. This is possible in
RS-232 and RS-422, but not in 2 wire RS-485.

>My question was: does the modem waits between steps 2 and 3? If the internal
>firmware of the modem enqueue messages in a global output buffer, it is 
>possible to have two consecutive messages without a delay between them. 

I have no idea how each modem on the market behaves.

Hello Pozzugno,

> Indeed your imagination is wrong :-)

Well, you're right about that humans could take a long time to finish typing
a command, so the "from the begin" idea is out.  But I still do not think
that there will be much chance (if any) that an unsolicited message will be
send while another command is being executed (for multiple reasons).

> It couldn't work. When I decide to send last char (even
> after a long delay),  at the same time the modem could
> decide to send an unsolicited message.

You're right, stated like that it seems like an unsolvable "race condition"
problem.  Nevertheless, it must have been solved time-and-again, otherwise
simple old-style modem communication between computers would never have
passed its beta stage. :-)

Is it possible that there are few different unsolicited messages and by
their fixed nature are fairly easy to recognise/discern from an expected
result, and by that feat rather easily removed when its recieved before an
expected reply ?

> I don't think RTS and CTS signals move with the transmitting
> of requests, answers or unsolicited messages.

I don't think so either.  I rather thought it could be a by-product of the
modem being busy internally (executing the command you just send, or simple
being busy sending you data), and therefore not able to receive any data
itself.

Well, I think I'm at the end of my ideas.  Sorry I could not help you.

Regards,
Rudy Wieser


-- Origional message:
<pozzugno@gmail.com> schreef in berichtnieuws
0e8ed80a-2360-41dc-94dd-7a632df171be@googlegroups.com...
> R.Wieser ha scritto:
> > > This is indeed a possible strategy, but it works only if the
> > > modem doesn't decide to transmit an unsolicited message
> > > immediately before the answer of the command send by
> > > the application (not the driver).
> >
> > You will have to check the docs ofcourse, but I could imagine that such
> > unsolicited messages will not occur between the end (and possibly even
the
> > begin!) of the request you send and the end of the reply for request
that it
> > sends back.
>
> Indeed your imagination is wrong :-)
> I think AT commands are oriented to human interaction and they surely can
be
> send by typing the characters on a keyboard (you have the possibility to
> correct what you have typed by pressing backspace). This means that the
> command you are sending can take a long time to finish. In the meantime,
> the modem can send you an unsolicited message. I already test this
situation.
>
> > As for your described problem ?  Check, just before sending the last
> > byte/char (the CR/LF) of your request if something has come in.  If so,
> > delay sending that last char until you have parsed/purged the received
data.
>
> It couldn't work. When I decide to send last char (even after a long
delay),
> at the same time the modem could decide to send an unsolicited message.
>
> > > Should the program, in every point it uses the modem,
> > > decode an answer that  could be mixed with unsolicited
> > > message? I don't think it's a good idea.
> >
> > You're quite correct in that, which is the reason I think that the
writers
> > of that embedded modem software will have thought of it to and (quite
> > likely) will hold off any such unsolicited message until a running
request
> > has been completeley finished.  Again, I would suggest you check the
> > (interfacing) specs coming with(?) that modem for it.
>
> Unfortunately you're wrong

>
> > One other thing: does that modem have any hardware control (RTS/CTS) ?
In
> > that case I could imagine that a CTS stays false for as long as a
command is
> > executing and data is send your way (assuming half-duplex here).  If an
> > unsolicited message is than "tacked on" the reply you should see the CTS
> > going true in between.
>
> Why? RTS/CTS are used for flow-control: the receiving could say to the
> transmitter to stop transmitting if the incoming queue is almost full.
> I don't think RTS and CTS signals move with the transmitting of requests,
> answers or unsolicited messages.

On 2014-05-09, pozzugno@gmail.com <pozzugno@gmail.com> wrote:
> R.Wieser ha scritto:
>> > This is indeed a possible strategy, but it works only if the
>> > modem doesn't decide to transmit an unsolicited message
>> > immediately before the answer of the command send by
>> > the application (not the driver).
>> 
>> You will have to check the docs ofcourse, but I could imagine that such
>> unsolicited messages will not occur between the end (and possibly even the
>> begin!) of the request you send and the end of the reply for request that it
>> sends back.
>
> Indeed your imagination is wrong :-)
> I think AT commands are oriented to human interaction and they surely can be
> send by typing the characters on a keyboard (you have the possibility to
> correct what you have typed by pressing backspace). This means that the 
> command you are sending can take a long time to finish. In the meantime, 
> the modem can send you an unsolicited message. I already test this situation.
>

To be honest, I'm not really seeing what the problem is.

I would have implemented this as a state machine in the upper layers of
the code with clearly defined state transitions depending on (1) timeouts,
(2) the responses you are waiting for and (3) unsolicited events such
as ring.

The next layer down would include a parser which extracts any printable
text, on a per line basis, from the modem and either tokenises it or
passes it as-is back up to the state machine.

If the state machine sees a ring while waiting for OK, it can decide
if it wants to transition to a new state at that point or wait for
the expected response (or timeout).

That means in the event of RING followed on the next line by OK, your
state machine would see two distinct responses from the modem.

You would of course have to make sure your state machine always
terminated any possible successful connect so you were not running up
connect charges after recovery from any unexpected events.

Is the above how others here would implement this or would you
use a different approach ?

Simon.

-- 
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world