AT commands: how to detect unsolicited result codes

Hello Pozzugno,

> This is indeed a possible strategy, but it works only if the
> modem doesn't decide to transmit an unsolicited message
> immediately before the answer of the command send by
> the application (not the driver).

You will have to check the docs ofcourse, but I could imagine that such
unsolicited messages will not occur between the end (and possibly even the
begin!) of the request you send and the end of the reply for request that it
sends back.

As for your described problem ?  Check, just before sending the last
byte/char (the CR/LF) of your request if something has come in.  If so,
delay sending that last char until you have parsed/purged the received data.

> Should the program, in every point it uses the modem,
> decode an answer that  could be mixed with unsolicited
> message? I don't think it's a good idea.

You're quite correct in that, which is the reason I think that the writers
of that embedded modem software will have thought of it to and (quite
likely) will hold off any such unsolicited message until a running request
has been completeley finished.  Again, I would suggest you check the
(interfacing) specs coming with(?) that modem for it.

One other thing: does that modem have any hardware control (RTS/CTS) ?  In
that case I could imagine that a CTS stays false for as long as a command is
executing and data is send your way (assuming half-duplex here).  If an
unsolicited message is than "tacked on" the reply you should see the CTS
going true in between.

Regards,
Rudy Wieser


-- Origional message:
<pozzugno@gmail.com> schreef in berichtnieuws
e1e19b93-1420-4977-bcfa-7975a0a14b96@googlegroups.com...
Il giorno gioved&#4294967295; 8 maggio 2014 21:12:25 UTC+2, R.Wieser
ha scritto:
> > The previous driver is broken for this answer
> > because it would detect two answers (the first
> > terminated at the first <LF>).
>
> You are making your life too difficult:  Its not upto your driver to make
> sense of the chars/bytes that it receives, thats the task of the program
> thats actually sending the requests and receiving the replies.
>
> In other words: Its upto the program using the driver to hold off sending
a
> new command until after the full reply of the previous command is
received.
> And as that program knows what commands it is sending it also knows what
the
> format of the reply should be.  That makes it *much* easier for it to
> determine if it should stop after the first EOL, or if it needs a few more
> lines for the full answer. :-)

This is indeed a possible strategy, but it works only if the modem doesn't
decide to transmit an unsolicited message immediately before the answer of
the command send by the application (not the driver). Or you're not
interested in unsolicited message (but I need to decode them).

Should the program, in every point it uses the modem, decode an answer that
could be mixed with unsolicited message? I don't think it's a good idea.

R.Wieser ha scritto:
> > This is indeed a possible strategy, but it works only if the
> > modem doesn't decide to transmit an unsolicited message
> > immediately before the answer of the command send by
> > the application (not the driver).
> 
> You will have to check the docs ofcourse, but I could imagine that such
> unsolicited messages will not occur between the end (and possibly even the
> begin!) of the request you send and the end of the reply for request that it
> sends back.

Indeed your imagination is wrong :-)
I think AT commands are oriented to human interaction and they surely can be
send by typing the characters on a keyboard (you have the possibility to
correct what you have typed by pressing backspace). This means that the 
command you are sending can take a long time to finish. In the meantime, 
the modem can send you an unsolicited message. I already test this situation.


> As for your described problem ?  Check, just before sending the last
> byte/char (the CR/LF) of your request if something has come in.  If so,
> delay sending that last char until you have parsed/purged the received data.

It couldn't work. When I decide to send last char (even after a long delay), 
at the same time the modem could decide to send an unsolicited message.


> > Should the program, in every point it uses the modem,
> > decode an answer that  could be mixed with unsolicited
> > message? I don't think it's a good idea.
> 
> You're quite correct in that, which is the reason I think that the writers
> of that embedded modem software will have thought of it to and (quite
> likely) will hold off any such unsolicited message until a running request
> has been completeley finished.  Again, I would suggest you check the
> (interfacing) specs coming with(?) that modem for it.

Unfortunately you're wrong :-( 

> One other thing: does that modem have any hardware control (RTS/CTS) ?  In
> that case I could imagine that a CTS stays false for as long as a command is
> executing and data is send your way (assuming half-duplex here).  If an
> unsolicited message is than "tacked on" the reply you should see the CTS
> going true in between.

Why? RTS/CTS are used for flow-control: the receiving could say to the
transmitter to stop transmitting if the incoming queue is almost full.
I don't think RTS and CTS signals move with the transmitting of requests,
answers or unsolicited messages.

On 2014-05-09, pozzugno@gmail.com <pozzugno@gmail.com> wrote:
> R.Wieser ha scritto:
>> > This is indeed a possible strategy, but it works only if the
>> > modem doesn't decide to transmit an unsolicited message
>> > immediately before the answer of the command send by
>> > the application (not the driver).
>> 
>> You will have to check the docs ofcourse, but I could imagine that such
>> unsolicited messages will not occur between the end (and possibly even the
>> begin!) of the request you send and the end of the reply for request that it
>> sends back.
>
> Indeed your imagination is wrong :-)
> I think AT commands are oriented to human interaction and they surely can be
> send by typing the characters on a keyboard (you have the possibility to
> correct what you have typed by pressing backspace). This means that the 
> command you are sending can take a long time to finish. In the meantime, 
> the modem can send you an unsolicited message. I already test this situation.
>

To be honest, I'm not really seeing what the problem is.

I would have implemented this as a state machine in the upper layers of
the code with clearly defined state transitions depending on (1) timeouts,
(2) the responses you are waiting for and (3) unsolicited events such
as ring.

The next layer down would include a parser which extracts any printable
text, on a per line basis, from the modem and either tokenises it or
passes it as-is back up to the state machine.

If the state machine sees a ring while waiting for OK, it can decide
if it wants to transition to a new state at that point or wait for
the expected response (or timeout).

That means in the event of RING followed on the next line by OK, your
state machine would see two distinct responses from the modem.

You would of course have to make sure your state machine always
terminated any possible successful connect so you were not running up
connect charges after recovery from any unexpected events.

Is the above how others here would implement this or would you
use a different approach ?

Simon.

-- 
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
Microsoft: Bringing you 1980s technology to a 21st century world

Hello Pozzugno,

> Indeed your imagination is wrong :-)

Well, you're right about that humans could take a long time to finish typing
a command, so the "from the begin" idea is out.  But I still do not think
that there will be much chance (if any) that an unsolicited message will be
send while another command is being executed (for multiple reasons).

> It couldn't work. When I decide to send last char (even
> after a long delay),  at the same time the modem could
> decide to send an unsolicited message.

You're right, stated like that it seems like an unsolvable "race condition"
problem.  Nevertheless, it must have been solved time-and-again, otherwise
simple old-style modem communication between computers would never have
passed its beta stage. :-)

Is it possible that there are few different unsolicited messages and by
their fixed nature are fairly easy to recognise/discern from an expected
result, and by that feat rather easily removed when its recieved before an
expected reply ?

> I don't think RTS and CTS signals move with the transmitting
> of requests, answers or unsolicited messages.

I don't think so either.  I rather thought it could be a by-product of the
modem being busy internally (executing the command you just send, or simple
being busy sending you data), and therefore not able to receive any data
itself.

Well, I think I'm at the end of my ideas.  Sorry I could not help you.

Regards,
Rudy Wieser


-- Origional message:
<pozzugno@gmail.com> schreef in berichtnieuws
0e8ed80a-2360-41dc-94dd-7a632df171be@googlegroups.com...
> R.Wieser ha scritto:
> > > This is indeed a possible strategy, but it works only if the
> > > modem doesn't decide to transmit an unsolicited message
> > > immediately before the answer of the command send by
> > > the application (not the driver).
> >
> > You will have to check the docs ofcourse, but I could imagine that such
> > unsolicited messages will not occur between the end (and possibly even
the
> > begin!) of the request you send and the end of the reply for request
that it
> > sends back.
>
> Indeed your imagination is wrong :-)
> I think AT commands are oriented to human interaction and they surely can
be
> send by typing the characters on a keyboard (you have the possibility to
> correct what you have typed by pressing backspace). This means that the
> command you are sending can take a long time to finish. In the meantime,
> the modem can send you an unsolicited message. I already test this
situation.
>
> > As for your described problem ?  Check, just before sending the last
> > byte/char (the CR/LF) of your request if something has come in.  If so,
> > delay sending that last char until you have parsed/purged the received
data.
>
> It couldn't work. When I decide to send last char (even after a long
delay),
> at the same time the modem could decide to send an unsolicited message.
>
> > > Should the program, in every point it uses the modem,
> > > decode an answer that  could be mixed with unsolicited
> > > message? I don't think it's a good idea.
> >
> > You're quite correct in that, which is the reason I think that the
writers
> > of that embedded modem software will have thought of it to and (quite
> > likely) will hold off any such unsolicited message until a running
request
> > has been completeley finished.  Again, I would suggest you check the
> > (interfacing) specs coming with(?) that modem for it.
>
> Unfortunately you're wrong

>
> > One other thing: does that modem have any hardware control (RTS/CTS) ?
In
> > that case I could imagine that a CTS stays false for as long as a
command is
> > executing and data is send your way (assuming half-duplex here).  If an
> > unsolicited message is than "tacked on" the reply you should see the CTS
> > going true in between.
>
> Why? RTS/CTS are used for flow-control: the receiving could say to the
> transmitter to stop transmitting if the incoming queue is almost full.
> I don't think RTS and CTS signals move with the transmitting of requests,
> answers or unsolicited messages.

On Fri, 9 May 2014 01:52:15 -0700 (PDT), giuseppe.modugno@gmail.com
wrote:

>upsid...@downunder.com ha scritto:
>> >This could work only if the modem *guarantees* a sufficient delay between 
>> >the last character of a message and the first character of the next message. 
>> >
>> >Consider the previous example (AT command from micro, OK answer from modem
>> >and immediately after RING unsolicited message). What happens if the RING
>> >event is generated *in the middle* of OK answer transmission? I think the
>> >modem firmware buffer the RING event and put out the unsolicited message as
>> >soon as the serial port is free, i.e. immediately after the last character
>> >of the OK answer. Do the modem firmware wait for some delay before putting
>> >out the next message?
>> >
>> >In case you're right, how could I know the amount of this delay, considering
>> >this behaviour isn't documented? It is very difficult to measure it, because
>> >I need to reproduce the events above... how? 
>> 
>> Typically, the AT command set is just half duplex request/response
>> traffic, so framing by idle periods is quite reliable. However, if
>> there are full duplex traffic, just read until next idle timeout. Let
>> the higher level logic figure out what is response to a request and
>> what is spontaneous "RING" messages.
>
>I'm not considering full-duplex traffic, my example is half-duplex.
>
>1. Host microcontroller sends completely the command AT<CR><LF>
>2. Modem answers with <CR><LF>OK<CR><LF> (during this msg RING event occurs)
>3. Modem transmits unsolicited message <CR><LF>RING<CR><LF>

If the modem sends "RING" at the same time time as you start sending
ATxxxxx<CR>, then it is definitely full duplex. This is possible in
RS-232 and RS-422, but not in 2 wire RS-485.

>My question was: does the modem waits between steps 2 and 3? If the internal
>firmware of the modem enqueue messages in a global output buffer, it is 
>possible to have two consecutive messages without a delay between them. 

I have no idea how each modem on the market behaves.

Il 09/05/2014 19:21, upsidedown@downunder.com ha scritto:
>> I'm not considering full-duplex traffic, my example is half-duplex.
>>
>> 1. Host microcontroller sends completely the command AT<CR><LF>
>> 2. Modem answers with <CR><LF>OK<CR><LF> (during this msg RING event occurs)
>> 3. Modem transmits unsolicited message <CR><LF>RING<CR><LF>
>
> If the modem sends "RING" at the same time time as you start sending
> ATxxxxx<CR>, then it is definitely full duplex. This is possible in
> RS-232 and RS-422, but not in 2 wire RS-485.

Yes, you're right. But my example above aren't full-duplex. AT command 
is *completely* send out (step 1 is finished) when the modem starts 
answering (step 2). And the modem *completely* answered with OK (step 2 
is finished) when it starts emitting unsolicited RING message (step 3).

This is definitely a half-duplex communication.


>> My question was: does the modem waits between steps 2 and 3? If the internal
>> firmware of the modem enqueue messages in a global output buffer, it is
>> possible to have two consecutive messages without a delay between them.
>
> I have no idea how each modem on the market behaves.

If the modem doesn't *guarantee* a minimum delay (4-5 characters time) 
between two output messages, are you agree your method (separating 
frames thanks to a delay) doesn't work?

Il 09/05/2014 15:03, R.Wieser ha scritto:
> Hello Pozzugno,
>
>> Indeed your imagination is wrong :-)
>
> Well, you're right about that humans could take a long time to finish typing
> a command, so the "from the begin" idea is out.  But I still do not think
> that there will be much chance (if any) that an unsolicited message will be
> send while another command is being executed (for multiple reasons).

Why not? I'm thinking how the embedded GSM modem firmware could work.
There are many threads running that can push output messages in a output 
queue.

One thread could manage the GSM network, so it receives signals that 
incoming call is in progress. This thread could push "RING" message in
the output buffer queue.

Another thread could manage the AT requests on the serial port and could 
push "OK" answers.

At a certain time, the output queue could contain two messages, emitted 
one after the other in a transmitting UART ISR.


>> It couldn't work. When I decide to send last char (even
>> after a long delay),  at the same time the modem could
>> decide to send an unsolicited message.
>
> You're right, stated like that it seems like an unsolvable "race condition"
> problem.  Nevertheless, it must have been solved time-and-again, otherwise
> simple old-style modem communication between computers would never have
> passed its beta stage. :-)

I agree with you and this is why I asked for a solution of this problem. 
I don't think the problem hasn't a solution, I'm quite sure I'm using a 
wrong method.


> Is it possible that there are few different unsolicited messages and by
> their fixed nature are fairly easy to recognise/discern from an expected
> result, and by that feat rather easily removed when its recieved before an
> expected reply ?

Indeed it seems the only possible solution, even if it isn't simple to 
implement. The receiving driver should detect a certain number of 
unsolicited messages and filter them out to the application layer that 
wants only the answers to commands. There many unsolicited messages:

   <CR><LF>RING<CR><LF>
   <CR><LF>CONNECT ...<CR><LF>
   <CR><LF>BUSY<CR><LF>
   <CR><LF>DIALTONE<CR><LF>
   <CR><LF>+CGMS...<CR><LF> (or something similar)

I have the possibility to implement *all* the possible unsolicited 
messages, but I don't think it is feasible because there are many and 
some of them couldln't be documented.

Alternatively I can capture and filter only the unsolicited messages I 
want to decode and analyze. The other unsolicited messages will not be 
recognized as such so they must be passed to the application layer.
The application should be ready to receive not only answers to commands 
it send, but also other messages that should be discarded because not 
useful unsolicited messages.

Il 09/05/2014 14:29, Simon Clubley ha scritto:
> I would have implemented this as a state machine in the upper layers of
> the code with clearly defined state transitions depending on (1) timeouts,
> (2) the responses you are waiting for and (3) unsolicited events such
> as ring.
>
> The next layer down would include a parser which extracts any printable
> text, on a per line basis, from the modem and either tokenises it or
> passes it as-is back up to the state machine.

Even the "per line basis" parser approach doesn't work everytime.
The answer to the AT command that sends SMS isn't <CR><LF> as usual, but 
it is the 2-characters prompt "> ". The lower layer (the driver?) can't 
decide to wait for <CR><LF> before passing the incoming characters to 
the application layer.

One possibility is to change at run-time the ending suffix: normally it 
is <CR><LF>, but it could be "> ". Anyway this suffix will be used only 
for answers to pass to the application layer. At the same time 
unsolicited messages (ending with <CR><LF>) could arrive and must be 
managed.


> If the state machine sees a ring while waiting for OK, it can decide
> if it wants to transition to a new state at that point or wait for
> the expected response (or timeout).
>
> That means in the event of RING followed on the next line by OK, your
> state machine would see two distinct responses from the modem.

You're thinking of a lower level that is able to separate two messages, 
even if they are strictly *consecutive*: answer-unsolicited or 
unsolicited-answer.

This is the starting question: what is the best method to separate 
incoming characters from modem into *distinct* messages in *all* 
possible situations (answers, unsolicited...)?
<CR><LF> prefix/suffix could be a method, but in some situation the 
suffix changes.

Are you thinking of a lower level that pass to the application layer 
*all* the messages: answers and unsolicited? IMHO it's not feasible. All 
the upper complex FSMs (there are many, one for sensing SMS, another for 
receiving SMS, another for making voice calls, another for making data 
calls, ...) should decide, in *every* state, what to do in case of 
unsolicited messages.


> You would of course have to make sure your state machine always
> terminated any possible successful connect so you were not running up
> connect charges after recovery from any unexpected events.
>
> Is the above how others here would implement this or would you
> use a different approach ?

On Sat, 10 May 2014 08:45:09 +0200, pozz <pozzugno@gmail.com> wrote:

>Il 09/05/2014 19:21, upsidedown@downunder.com ha scritto:
>>> I'm not considering full-duplex traffic, my example is half-duplex.
>>>
>>> 1. Host microcontroller sends completely the command AT<CR><LF>
>>> 2. Modem answers with <CR><LF>OK<CR><LF> (during this msg RING event occurs)
>>> 3. Modem transmits unsolicited message <CR><LF>RING<CR><LF>
>>
>> If the modem sends "RING" at the same time time as you start sending
>> ATxxxxx<CR>, then it is definitely full duplex. This is possible in
>> RS-232 and RS-422, but not in 2 wire RS-485.
>
>Yes, you're right. But my example above aren't full-duplex. AT command 
>is *completely* send out (step 1 is finished) when the modem starts 
>answering (step 2). And the modem *completely* answered with OK (step 2 
>is finished) when it starts emitting unsolicited RING message (step 3).
>
>This is definitely a half-duplex communication.
>
>
>>> My question was: does the modem waits between steps 2 and 3? If the internal
>>> firmware of the modem enqueue messages in a global output buffer, it is
>>> possible to have two consecutive messages without a delay between them.
>>
>> I have no idea how each modem on the market behaves.
>
>If the modem doesn't *guarantee* a minimum delay (4-5 characters time) 
>between two output messages, are you agree your method (separating 
>frames thanks to a delay) doesn't work?

In the interrupt service routine (which should be kept very simple to
avoid blocking other interrupts) just use the idle timeout.

It is to the upper level logic to find out the actual framing, e.g. by
knowing the structure of the expected response (such as ending in
<LF>).

If it can't make any sense, just resend the command after a while
(100-1000 ms). This works well in practice.

On Sat, 10 May 2014 09:15:37 +0200, pozz <pozzugno@gmail.com> wrote:

>Il 09/05/2014 14:29, Simon Clubley ha scritto:
>> I would have implemented this as a state machine in the upper layers of
>> the code with clearly defined state transitions depending on (1) timeouts,
>> (2) the responses you are waiting for and (3) unsolicited events such
>> as ring.
>>
>> The next layer down would include a parser which extracts any printable
>> text, on a per line basis, from the modem and either tokenises it or
>> passes it as-is back up to the state machine.
>
>Even the "per line basis" parser approach doesn't work everytime.
>The answer to the AT command that sends SMS isn't <CR><LF> as usual, but 
>it is the 2-characters prompt "> ". The lower layer (the driver?) can't 
>decide to wait for <CR><LF> before passing the incoming characters to 
>the application layer.
>
>One possibility is to change at run-time the ending suffix: normally it 
>is <CR><LF>, but it could be "> ". Anyway this suffix will be used only 
>for answers to pass to the application layer. At the same time 
>unsolicited messages (ending with <CR><LF>) could arrive and must be 
>managed.
>
>
>> If the state machine sees a ring while waiting for OK, it can decide
>> if it wants to transition to a new state at that point or wait for
>> the expected response (or timeout).
>>
>> That means in the event of RING followed on the next line by OK, your
>> state machine would see two distinct responses from the modem.
>
>You're thinking of a lower level that is able to separate two messages, 
>even if they are strictly *consecutive*: answer-unsolicited or 
>unsolicited-answer.
>
>This is the starting question: what is the best method to separate 
>incoming characters from modem into *distinct* messages in *all* 
>possible situations (answers, unsolicited...)?
><CR><LF> prefix/suffix could be a method, but in some situation the 
>suffix changes.
>
>Are you thinking of a lower level that pass to the application layer 
>*all* the messages: answers and unsolicited? IMHO it's not feasible. All 
>the upper complex FSMs (there are many, one for sensing SMS, another for 
>receiving SMS, another for making voice calls, another for making data 
>calls, ...) should decide, in *every* state, what to do in case of 
>unsolicited messages.
>
>
>> You would of course have to make sure your state machine always
>> terminated any possible successful connect so you were not running up
>> connect charges after recovery from any unexpected events.
>>
>> Is the above how others here would implement this or would you
>> use a different approach ?

Why bother with a fuss like that above. If your application layer
doesn't understand the response, just resend the original command a
few times until you get a valid response or declare the connection
dead. 

For the resend interval use something odd like 3.13 or 2.71 seconds,
since if there are similar timers at the end, they would typically
expire after 1, 2 or 3 seconds :-).