AT commands: how to detect unsolicited result codes

As you know, modems (GSM, GSM/GPRS, PSTN, ...) are controlled
by a host microcontroller through AT commands send to a asyncronous
serial port. The modem answers with a message.
Commands and answers are ASCII-based and terminated with <CR> 
and/or <CR><LF>.

Usually echo is activated for human interaction, but it is usually
disabled when the modem is controlled by a microcontroller.

For example (-> micro-to-uC, <- modem-to-uC):
  -> AT<CR>
  <- <CR><LF>OK<CR><LF>
  -> AT+IPR=57600<CR>
  <- <CR><LF>OK<CR><LF>

Writing a modem driver is simple: push each received character in
a FIFO buffer inside the ISR. Main task pop the characters in the
FIFO until the termination (<LF), when a complete answer is found.

The transmit task is a little harder: a command should be sent
only after receiving the *complete* answer of the previous command.
Indeed I tried to send a second commands immediately after the
first one, but the modem answered and processed only the first.

I have two problems in this scenario.

The first is that some answers have termination characters
<CR><LF> embedded. For example:

  -> ATI4<CR>
  <- SIMCOM_Ltd SIMCOM_SIM900<CR><LF><CR><LF>OK<CR><LF>

The previous driver is broken for this answer because it would
detect two answers (the first terminated at the first <LF>).

I could modify the driver to detect the 4-characters termination 
"OK<CR><LF>", but some answers could end with "ERROR<CR><LF>".
I could detect both of them as answer termination, but I think
the modem is free to emit other types of answers with a different
scheme.

The second problem is with unsolicited result codes that are 
messages asyncronously transmitted from the modem to the 
host microcontroller.
They relate to very different functionalities (GSM registration 
status, incoming SMS, incoming voice/data call...) and, most 
of them (but not all!), can be disabled.

How to modify the driver to understand those unsolicited messages 
too?
I'm not an expert of modems, but I think (and *hope*) the modem
doesn't emit an unsolicited message when it has already started
answering to a previous command send from microcontroller.
Otherwise the answer and the unsolicited message overlap and would
be impossible to distinguish them.

It seems the answers are human-oriented, but it's difficult to
automate a process to decode them.

Any suggestions?

On Thu, 8 May 2014 05:21:50 -0700 (PDT), pozzugno@gmail.com wrote:

>As you know, modems (GSM, GSM/GPRS, PSTN, ...) are controlled
>by a host microcontroller through AT commands send to a asyncronous
>serial port. The modem answers with a message.
>Commands and answers are ASCII-based and terminated with <CR> 
>and/or <CR><LF>.
>
>Usually echo is activated for human interaction, but it is usually
>disabled when the modem is controlled by a microcontroller.
>
>For example (-> micro-to-uC, <- modem-to-uC):
>  -> AT<CR>
>  <- <CR><LF>OK<CR><LF>
>  -> AT+IPR=57600<CR>
>  <- <CR><LF>OK<CR><LF>
>
>Writing a modem driver is simple: push each received character in
>a FIFO buffer inside the ISR. Main task pop the characters in the
>FIFO until the termination (<LF), when a complete answer is found.
>
>The transmit task is a little harder: a command should be sent
>only after receiving the *complete* answer of the previous command.
>Indeed I tried to send a second commands immediately after the
>first one, but the modem answered and processed only the first.
>
>I have two problems in this scenario.
>
>The first is that some answers have termination characters
><CR><LF> embedded. For example:
>
>  -> ATI4<CR>
>  <- SIMCOM_Ltd SIMCOM_SIM900<CR><LF><CR><LF>OK<CR><LF>
>
>The previous driver is broken for this answer because it would
>detect two answers (the first terminated at the first <LF>).
>
>I could modify the driver to detect the 4-characters termination 
>"OK<CR><LF>", but some answers could end with "ERROR<CR><LF>".
>I could detect both of them as answer termination, but I think
>the modem is free to emit other types of answers with a different
>scheme.
>
>The second problem is with unsolicited result codes that are 
>messages asyncronously transmitted from the modem to the 
>host microcontroller.
>They relate to very different functionalities (GSM registration 
>status, incoming SMS, incoming voice/data call...) and, most 
>of them (but not all!), can be disabled.
>
>How to modify the driver to understand those unsolicited messages 
>too?
>I'm not an expert of modems, but I think (and *hope*) the modem
>doesn't emit an unsolicited message when it has already started
>answering to a previous command send from microcontroller.
>Otherwise the answer and the unsolicited message overlap and would
>be impossible to distinguish them.
>
>It seems the answers are human-oriented, but it's difficult to
>automate a process to decode them.
>
>Any suggestions?

Just read until timeout. If there are a few character times idle after
the last character, you could assume that the message has ended,
especially if the last character was <LF>.

If autobauding is enabled, there can be a quite long time after the
"AT" characters echoed and the rest of the response.

I had a problem with the updated version of the Siemens GPRS modem
TC35i (but not with the original TC35), which often failed to handle
ATxyz<CR> sent in a row. 

Finally I had to modify the command transmission to
1.) Send "A"
2.) Wait for echo of "A"
3.) Send "T"
4.) Wait for echo for "T"
5.) Send xyz<CR> in a row.

Which worked every time. The autobauding seemed to slowed considerably
from the original modem version.

Il giorno gioved� 8 maggio 2014 14:50:24 UTC+2, 
upsid...@downunder.com ha scritto:
> Just read until timeout. If there are a few character times idle after
> the last character, you could assume that the message has ended,
> especially if the last character was <LF>.

It's a nice idea... does it really work?

The receiving state machine should arm two timeouts: one for the
no answer (5 seconds? I think it could depends from the actual command
send) and another one for intra-character time (100ms?).

The only problem I see with your approach is with unsolicited result 
codes that could be sent randomly from modem. Consider the following:

  -> AT<CR>
  <- <CR><LF>OK<CR><LF>
  <- <CR><LF>RING<CR><LF>

If the first character of RING unsolicited message is send immediately
after the last character of OK answer to AT command, your state machine
could erroneously think it's a single message (and process it in that way).


> If autobauding is enabled, there can be a quite long time after the
> "AT" characters echoed and the rest of the response.
> 
> I had a problem with the updated version of the Siemens GPRS modem
> TC35i (but not with the original TC35), which often failed to handle
> ATxyz<CR> sent in a row. 
> 
> Finally I had to modify the command transmission to
> 1.) Send "A"
> 2.) Wait for echo of "A"
> 3.) Send "T"
> 4.) Wait for echo for "T"
> 5.) Send xyz<CR> in a row.
> 
> Which worked every time. The autobauding seemed to slowed considerably
> from the original modem version.

This is why I'll never use autobauding with a host microcontroller. Why
don't you disable it?

On Thu, 8 May 2014 08:04:45 -0700 (PDT), giuseppe.modugno@gmail.com
wrote:

>Il giorno gioved� 8 maggio 2014 14:50:24 UTC+2, 
>upsid...@downunder.com ha scritto:
>> Just read until timeout. If there are a few character times idle after
>> the last character, you could assume that the message has ended,
>> especially if the last character was <LF>.
>
>It's a nice idea... does it really work?

I have used this method with MC 68360 and various Power PC with QUICC
coprocessor for more than a decade. Of course, any controller with a
free timer with microsecond timer will do.

>The receiving state machine should arm two timeouts: one for the
>no answer (5 seconds? I think it could depends from the actual command
>send) and another one for intra-character time (100ms?).
>
>The only problem I see with your approach is with unsolicited result 
>codes that could be sent randomly from modem. Consider the following:
>
>  -> AT<CR>
>  <- <CR><LF>OK<CR><LF>
>  <- <CR><LF>RING<CR><LF>
>
>If the first character of RING unsolicited message is send immediately
>after the last character of OK answer to AT command, your state machine
>could erroneously think it's a single message (and process it in that way).

Since this is simple 7 bit ASCII, you could insert special character
like TIMEOUTxxxms into the reception queue and let the high level
logic make sense of that frame.

> The first is that some answers have termination characters
> <CR><LF> embedded. For example:
>
>    -> ATI4<CR>
>    <- SIMCOM_Ltd SIMCOM_SIM900<CR><LF><CR><LF>OK<CR><LF>
>
> The previous driver is broken for this answer because it would
> detect two answers (the first terminated at the first <LF>).
>
> I could modify the driver to detect the 4-characters termination
> "OK<CR><LF>", but some answers could end with "ERROR<CR><LF>".
> I could detect both of them as answer termination, but I think
> the modem is free to emit other types of answers with a different
> scheme.

Just throw away LFs in the Rx ISR and use CRs for the official 
termination byte.

JJS

Hello Pozzugno(?),

> The previous driver is broken for this answer
> because it would detect two answers (the first
> terminated at the first <LF>).

You are making your life too difficult:  Its not upto your driver to make
sense of the chars/bytes that it receives, thats the task of the program
thats actually sending the requests and receiving the replies.

In other words: Its upto the program using the driver to hold off sending a
new command until after the full reply of the previous command is received.
And as that program knows what commands it is sending it also knows what the
format of the reply should be.  That makes it *much* easier for it to
determine if it should stop after the first EOL, or if it needs a few more
lines for the full answer. :-)

Oh sure, you can let the driver do it, but that would mean that it must be
intelligent enough to parse the outgoing text to be able to figure out how
many lines of response it should expect.  ... which is something you should
not want to ( a] that is not what a driver is for b] you would get more
instead of fewer problems).

My two cents.

Regards,
Rudy Wieser


-- Origional message:
<pozzugno@gmail.com> schreef in berichtnieuws
438ca738-188f-4346-91ee-232de68e52ce@googlegroups.com...
> As you know, modems (GSM, GSM/GPRS, PSTN, ...) are controlled
> by a host microcontroller through AT commands send to a asyncronous
> serial port. The modem answers with a message.
> Commands and answers are ASCII-based and terminated with <CR>
> and/or <CR><LF>.
>
> Usually echo is activated for human interaction, but it is usually
> disabled when the modem is controlled by a microcontroller.
>
> For example (-> micro-to-uC, <- modem-to-uC):
>   -> AT<CR>
>   <- <CR><LF>OK<CR><LF>
>   -> AT+IPR=57600<CR>
>   <- <CR><LF>OK<CR><LF>
>
> Writing a modem driver is simple: push each received character in
> a FIFO buffer inside the ISR. Main task pop the characters in the
> FIFO until the termination (<LF), when a complete answer is found.
>
> The transmit task is a little harder: a command should be sent
> only after receiving the *complete* answer of the previous command.
> Indeed I tried to send a second commands immediately after the
> first one, but the modem answered and processed only the first.
>
> I have two problems in this scenario.
>
> The first is that some answers have termination characters
> <CR><LF> embedded. For example:
>
>   -> ATI4<CR>
>   <- SIMCOM_Ltd SIMCOM_SIM900<CR><LF><CR><LF>OK<CR><LF>
>
> The previous driver is broken for this answer because it would
> detect two answers (the first terminated at the first <LF>).
>
> I could modify the driver to detect the 4-characters termination
> "OK<CR><LF>", but some answers could end with "ERROR<CR><LF>".
> I could detect both of them as answer termination, but I think
> the modem is free to emit other types of answers with a different
> scheme.
>
> The second problem is with unsolicited result codes that are
> messages asyncronously transmitted from the modem to the
> host microcontroller.
> They relate to very different functionalities (GSM registration
> status, incoming SMS, incoming voice/data call...) and, most
> of them (but not all!), can be disabled.
>
> How to modify the driver to understand those unsolicited messages
> too?
> I'm not an expert of modems, but I think (and *hope*) the modem
> doesn't emit an unsolicited message when it has already started
> answering to a previous command send from microcontroller.
> Otherwise the answer and the unsolicited message overlap and would
> be impossible to distinguish them.
>
> It seems the answers are human-oriented, but it's difficult to
> automate a process to decode them.
>
> Any suggestions?

upsid...@downunder.com ha scritto:
> On Thu, 8 May 2014 08:04:45 -0700 (PDT), gius...@gmail.com
> wrote:
[...]
> >The only problem I see with your approach is with unsolicited result 
> >codes that could be sent randomly from modem. Consider the following:
> >
> >  -> AT<CR>
> >  <- <CR><LF>OK<CR><LF>
> >  <- <CR><LF>RING<CR><LF>
> >
> >If the first character of RING unsolicited message is send immediately
> >after the last character of OK answer to AT command, your state machine
> >could erroneously think it's a single message (and process it in that way).
> 
> Since this is simple 7 bit ASCII, you could insert special character
> like TIMEOUTxxxms into the reception queue and let the high level
> logic make sense of that frame.

Do you mean pushing into receiving buffer a non-ASCII character (like 0xFF)
to mark the end of the previous message?
This could work only if the modem *guarantees* a sufficient delay between 
the last character of a message and the first character of the next message. 

Consider the previous example (AT command from micro, OK answer from modem
and immediately after RING unsolicited message). What happens if the RING
event is generated *in the middle* of OK answer transmission? I think the
modem firmware buffer the RING event and put out the unsolicited message as
soon as the serial port is free, i.e. immediately after the last character
of the OK answer. Do the modem firmware wait for some delay before putting
out the next message?

In case you're right, how could I know the amount of this delay, considering
this behaviour isn't documented? It is very difficult to measure it, because
I need to reproduce the events above... how? 

What is the delay you use to mark the end of a message with your modem?
I'm using Simcom.

Il giorno gioved� 8 maggio 2014 21:12:25 UTC+2, R.Wieser 
ha scritto:
> > The previous driver is broken for this answer
> > because it would detect two answers (the first
> > terminated at the first <LF>).
> 
> You are making your life too difficult:  Its not upto your driver to make
> sense of the chars/bytes that it receives, thats the task of the program
> thats actually sending the requests and receiving the replies.
> 
> In other words: Its upto the program using the driver to hold off sending a
> new command until after the full reply of the previous command is received.
> And as that program knows what commands it is sending it also knows what the
> format of the reply should be.  That makes it *much* easier for it to
> determine if it should stop after the first EOL, or if it needs a few more
> lines for the full answer. :-)

This is indeed a possible strategy, but it works only if the modem doesn't
decide to transmit an unsolicited message immediately before the answer of
the command send by the application (not the driver). Or you're not 
interested in unsolicited message (but I need to decode them).

Should the program, in every point it uses the modem, decode an answer that 
could be mixed with unsolicited message? I don't think it's a good idea.

On Fri, 9 May 2014 00:21:14 -0700 (PDT), pozzugno@gmail.com wrote:

>upsid...@downunder.com ha scritto:
>> On Thu, 8 May 2014 08:04:45 -0700 (PDT), gius...@gmail.com
>> wrote:
>[...]
>> >The only problem I see with your approach is with unsolicited result 
>> >codes that could be sent randomly from modem. Consider the following:
>> >
>> >  -> AT<CR>
>> >  <- <CR><LF>OK<CR><LF>
>> >  <- <CR><LF>RING<CR><LF>
>> >
>> >If the first character of RING unsolicited message is send immediately
>> >after the last character of OK answer to AT command, your state machine
>> >could erroneously think it's a single message (and process it in that way).
>> 
>> Since this is simple 7 bit ASCII, you could insert special character
>> like TIMEOUTxxxms into the reception queue and let the high level
>> logic make sense of that frame.
>
>Do you mean pushing into receiving buffer a non-ASCII character (like 0xFF)
>to mark the end of the previous message?

Or even set the highest bit on the last byte :-)

>This could work only if the modem *guarantees* a sufficient delay between 
>the last character of a message and the first character of the next message. 
>
>Consider the previous example (AT command from micro, OK answer from modem
>and immediately after RING unsolicited message). What happens if the RING
>event is generated *in the middle* of OK answer transmission? I think the
>modem firmware buffer the RING event and put out the unsolicited message as
>soon as the serial port is free, i.e. immediately after the last character
>of the OK answer. Do the modem firmware wait for some delay before putting
>out the next message?
>
>In case you're right, how could I know the amount of this delay, considering
>this behaviour isn't documented? It is very difficult to measure it, because
>I need to reproduce the events above... how? 
>

Typically, the AT command set is just half duplex request/response
traffic, so framing by idle periods is quite reliable. However, if
there are full duplex traffic, just read until next idle timeout. Let
the higher level logic figure out what is response to a request and
what is spontaneous "RING" messages.

Of course, the receiver buffer must be able to contain a few of the
longest messages.

>What is the delay you use to mark the end of a message with your modem?
>I'm using Simcom.

In most cases 2-5 character time seems to work reliably, without
spending too long waiting after the whole message. At 9600 bit/s the
character time is about 1 ms, so an idle timeout of 2-5 ms should be
OK.

So even if the delays between characters is longer you expected
(especially after "AT" if autobauding is used), you get some extra
idle timeouts in the buffer, but the higher level logic could ignore
then, if it is within a known response sequence.

upsid...@downunder.com ha scritto:
> >This could work only if the modem *guarantees* a sufficient delay between 
> >the last character of a message and the first character of the next message. 
> >
> >Consider the previous example (AT command from micro, OK answer from modem
> >and immediately after RING unsolicited message). What happens if the RING
> >event is generated *in the middle* of OK answer transmission? I think the
> >modem firmware buffer the RING event and put out the unsolicited message as
> >soon as the serial port is free, i.e. immediately after the last character
> >of the OK answer. Do the modem firmware wait for some delay before putting
> >out the next message?
> >
> >In case you're right, how could I know the amount of this delay, considering
> >this behaviour isn't documented? It is very difficult to measure it, because
> >I need to reproduce the events above... how? 
> 
> Typically, the AT command set is just half duplex request/response
> traffic, so framing by idle periods is quite reliable. However, if
> there are full duplex traffic, just read until next idle timeout. Let
> the higher level logic figure out what is response to a request and
> what is spontaneous "RING" messages.

I'm not considering full-duplex traffic, my example is half-duplex.

1. Host microcontroller sends completely the command AT<CR><LF>
2. Modem answers with <CR><LF>OK<CR><LF> (during this msg RING event occurs)
3. Modem transmits unsolicited message <CR><LF>RING<CR><LF>

My question was: does the modem waits between steps 2 and 3? If the internal
firmware of the modem enqueue messages in a global output buffer, it is 
possible to have two consecutive messages without a delay between them.