Designing a protocol for the first time

Greetings,

I wish to transfer binary data in the simplest way possible from a PC to 
either an AVR or TMS320F2812 embedded target via USB.

The PC will be running a Python/wx.Python program and use PySerial to 
send data out through the virtual COMM port produced by using a FTDI 
FT245BM USB-to-FIFO parallel interface chip on the target.  Ideally this 
program will be useable on Linux as well as Windoze machines.

I estimate that I need up to 120kB/s (without overhead) of data 
throughput to transfer 30000 points per second of 32-bit words 
containing vector graphic X,Y,RGB points for a laser show project.

The FTDI chip says it can do 300kB/s with the virtual COMM driver, so 
this should work unless they are fibbing.

I have gained some experience recently by writing a set of low-level 
serial driver functions for the F2812.  Then I wrote a command parser 
for it so the F2812 can respond to text commands and send back various 
reports.  This is all via text terminal.  That was my first foray into 
communication interface programming.  It was fun and not too difficult.

Now I need to transfer binary data.  The project here won't need any 
human readable command line.  I also don't want to increase overhead by 
having to reserve the 8th bit for example to distinguish control codes 
from data, or use some text rendition of the data, etc.  Just straight 
binary.

Thus, I suspect the simplest way for the target to behave is to 
implement a state machine something like this:

data:                   state:

<none yet>              1. wait for start code
0x1234                  2. (start code received) wait for size field
0x0123                  3. (16-bit size field received, specifying 291
                            words to follow) wait for data
0xdddddddd              4. (data incoming) count data words until 291
0xdddddddd                  received
0xdddddddd
...
0xdddddddd              5. (291st word received) wait for CRC state
0x3E5B                  6. (16-bit CRC received) goto state 1

Now of course, each state transistion can send a code back to the sender 
telling whether things are proceeding smoothly.  Another state 
transition path to add might be:

data:

<none yet>              1. wait for start code
0x5678                  7. (request for ACK received) send ACK, goto
                            state 1

That way if the receiver is stuck waiting for data and the sender 
doesn't know what state receiver is in, the sender can request ACK until 
the data receive counter transitions things back to state 1.

Is there anything more needed than this (aside from additional fields in 
the protocol)?

Since there is a specific code to tell the receiver that a data package 
has begun, when the receiver is in state 1, any sort of commands can 
also be issued by using any codes other than the 0x1234 example.

Any comments would be welcome regarding my budding protocol idea.


Good day!




-- 
_____________________
Christopher R. Carlen
crobc@bogus-remove-me.sbcglobal.net
SuSE 9.1 Linux 2.6.5

CC wrote:
> Greetings,
> 
> I wish to transfer binary data in the simplest way possible from a PC to 
> either an AVR or TMS320F2812 embedded target via USB.
> 
> The PC will be running a Python/wx.Python program and use PySerial to 
> send data out through the virtual COMM port produced by using a FTDI 
> FT245BM USB-to-FIFO parallel interface chip on the target.  Ideally this 
> program will be useable on Linux as well as Windoze machines.
> 
> I estimate that I need up to 120kB/s (without overhead) of data 
> throughput to transfer 30000 points per second of 32-bit words 
> containing vector graphic X,Y,RGB points for a laser show project.
> 
> The FTDI chip says it can do 300kB/s with the virtual COMM driver, so 
> this should work unless they are fibbing.
> 
> I have gained some experience recently by writing a set of low-level 
> serial driver functions for the F2812.  Then I wrote a command parser 
> for it so the F2812 can respond to text commands and send back various 
> reports.  This is all via text terminal.  That was my first foray into 
> communication interface programming.  It was fun and not too difficult.
> 
> Now I need to transfer binary data.  The project here won't need any 
> human readable command line.  I also don't want to increase overhead by 
> having to reserve the 8th bit for example to distinguish control codes 
> from data, or use some text rendition of the data, etc.  Just straight 
> binary.
> 
> Thus, I suspect the simplest way for the target to behave is to 
> implement a state machine something like this:
> 
> data:                   state:
> 
> <none yet>              1. wait for start code
> 0x1234                  2. (start code received) wait for size field
> 0x0123                  3. (16-bit size field received, specifying 291
>                            words to follow) wait for data
> 0xdddddddd              4. (data incoming) count data words until 291
> 0xdddddddd                  received
> 0xdddddddd
> ...
> 0xdddddddd              5. (291st word received) wait for CRC state
> 0x3E5B                  6. (16-bit CRC received) goto state 1
> 
> Now of course, each state transistion can send a code back to the sender 
> telling whether things are proceeding smoothly.  Another state 
> transition path to add might be:
> 
> data:
> 
> <none yet>              1. wait for start code
> 0x5678                  7. (request for ACK received) send ACK, goto
>                            state 1
> 
> That way if the receiver is stuck waiting for data and the sender 
> doesn't know what state receiver is in, the sender can request ACK until 
> the data receive counter transitions things back to state 1.
> 
> Is there anything more needed than this (aside from additional fields in 
> the protocol)?
> 
> Since there is a specific code to tell the receiver that a data package 
> has begun, when the receiver is in state 1, any sort of commands can 
> also be issued by using any codes other than the 0x1234 example.
> 
> Any comments would be welcome regarding my budding protocol idea.

X-modem CRC.  Then you only have to write it for
the target.

On Mar 3, 4:36 am, CC <cr...@BOGUS.sbcglobal.net> wrote:

> I wish to transfer binary data in the simplest way possible from a PC to
> either an AVR or TMS320F2812 embedded target via USB.

You may be interested in RFC 1549 "PPP in HDLC framing", see
http://www.ietf.org/rfc/rfc1549.txt

The frame format is simple and robust, and has fairly low overhead.

Le Sun, 02 Mar 2008 19:36:51 -0800, CC a écrit:

> Greetings,
> 
> I wish to transfer binary data in the simplest way possible from a PC to 
> either an AVR or TMS320F2812 embedded target via USB.
> 
> The PC will be running a Python/wx.Python program and use PySerial to 
> send data out through the virtual COMM port produced by using a FTDI 
> FT245BM USB-to-FIFO parallel interface chip on the target.  Ideally this 
> program will be useable on Linux as well as Windoze machines.
> 
> I estimate that I need up to 120kB/s (without overhead) of data 
> throughput to transfer 30000 points per second of 32-bit words 
> containing vector graphic X,Y,RGB points for a laser show project.
> 
> The FTDI chip says it can do 300kB/s with the virtual COMM driver, so 
> this should work unless they are fibbing.
> 
> I have gained some experience recently by writing a set of low-level 
> serial driver functions for the F2812.  Then I wrote a command parser 
> for it so the F2812 can respond to text commands and send back various 
> reports.  This is all via text terminal.  That was my first foray into 
> communication interface programming.  It was fun and not too difficult.
> 
> Now I need to transfer binary data.  The project here won't need any 
> human readable command line.  I also don't want to increase overhead by 
> having to reserve the 8th bit for example to distinguish control codes 
> from data, or use some text rendition of the data, etc.  Just straight 
> binary.
> 
> Thus, I suspect the simplest way for the target to behave is to 
> implement a state machine something like this:
> 
> data:                   state:
> 
> <none yet>              1. wait for start code
> 0x1234                  2. (start code received) wait for size field
> 0x0123                  3. (16-bit size field received, specifying 291
>                             words to follow) wait for data
> 0xdddddddd              4. (data incoming) count data words until 291
> 0xdddddddd                  received
> 0xdddddddd
> ...
> 0xdddddddd              5. (291st word received) wait for CRC state
> 0x3E5B                  6. (16-bit CRC received) goto state 1
> 
> Now of course, each state transistion can send a code back to the sender 
> telling whether things are proceeding smoothly.  Another state 
> transition path to add might be:
> 
> data:
> 
> <none yet>              1. wait for start code
> 0x5678                  7. (request for ACK received) send ACK, goto
>                             state 1
> 
> That way if the receiver is stuck waiting for data and the sender 
> doesn't know what state receiver is in, the sender can request ACK until 
> the data receive counter transitions things back to state 1.
> 
> Is there anything more needed than this (aside from additional fields in 
> the protocol)?
> 
> Since there is a specific code to tell the receiver that a data package 
> has begun, when the receiver is in state 1, any sort of commands can 
> also be issued by using any codes other than the 0x1234 example.
> 
> Any comments would be welcome regarding my budding protocol idea.
> 
> 
> Good day!
> 
> 
> 
>

I agree with Jim and Arlet although Jim solution is easier to implement

-- 
HBV

On 3 Mar, 04:36, CC <cr...@BOGUS.sbcglobal.net> wrote:
> Greetings,
>
> I wish to transfer binary data in the simplest way possible from a PC to
> either an AVR or TMS320F2812 embedded target via USB.
>
> The PC will be running a Python/wx.Python program and use PySerial to
> send data out through the virtual COMM port produced by using a FTDI
> FT245BM USB-to-FIFO parallel interface chip on the target. =A0Ideally this=

> program will be useable on Linux as well as Windoze machines.
>
> I estimate that I need up to 120kB/s (without overhead) of data
> throughput to transfer 30000 points per second of 32-bit words
> containing vector graphic X,Y,RGB points for a laser show project.
>
> The FTDI chip says it can do 300kB/s with the virtual COMM driver, so
> this should work unless they are fibbing.
>
> I have gained some experience recently by writing a set of low-level
> serial driver functions for the F2812. =A0Then I wrote a command parser
> for it so the F2812 can respond to text commands and send back various
> reports. =A0This is all via text terminal. =A0That was my first foray into=

> communication interface programming. =A0It was fun and not too difficult.
>
> Now I need to transfer binary data. =A0The project here won't need any
> human readable command line. =A0I also don't want to increase overhead by
> having to reserve the 8th bit for example to distinguish control codes
> from data, or use some text rendition of the data, etc. =A0Just straight
> binary.
>
> Thus, I suspect the simplest way for the target to behave is to
> implement a state machine something like this:
>
> data: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 state:
>
> <none yet> =A0 =A0 =A0 =A0 =A0 =A0 =A01. wait for start code
> 0x1234 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A02. (start code received) wait fo=
r size field
> 0x0123 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A03. (16-bit size field received, =
specifying 291
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 words to follow) w=
ait for data
> 0xdddddddd =A0 =A0 =A0 =A0 =A0 =A0 =A04. (data incoming) count data words =
until 291
> 0xdddddddd =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0received
> 0xdddddddd
> ...
> 0xdddddddd =A0 =A0 =A0 =A0 =A0 =A0 =A05. (291st word received) wait for CR=
C state
> 0x3E5B =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A06. (16-bit CRC received) goto st=
ate 1
>
> Now of course, each state transistion can send a code back to the sender
> telling whether things are proceeding smoothly. =A0Another state
> transition path to add might be:
>
> data:
>
> <none yet> =A0 =A0 =A0 =A0 =A0 =A0 =A01. wait for start code
> 0x5678 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A07. (request for ACK received) se=
nd ACK, goto
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 state 1
>
> That way if the receiver is stuck waiting for data and the sender
> doesn't know what state receiver is in, the sender can request ACK until
> the data receive counter transitions things back to state 1.
>
> Is there anything more needed than this (aside from additional fields in
> the protocol)?
>
> Since there is a specific code to tell the receiver that a data package
> has begun, when the receiver is in state 1, any sort of commands can
> also be issued by using any codes other than the 0x1234 example.
>
> Any comments would be welcome regarding my budding protocol idea.
>
> Good day!
>
> --
> _____________________
> Christopher R. Carlen
> cr...@bogus-remove-me.sbcglobal.net
> SuSE 9.1 Linux 2.6.5

Are you not reinventing a wheel that you already have installed? USB
already handles ack/nak and CRC protection of your data. Or is there
some peculiarity of the FTDI chip here?

Are you sending real-time data? As in "if the sent data is corrupt, we
do not have time to ask for a resend, so we will just skip it"? If so,
maybe you should skip CRCs altoghether and use USB isochronous mode.
If not, you will need some way for the receiving end to tell the
transmitter that something went wrong (faulty CRC, not enough bytes
received, faulty header data, etc.) and request a retransmission.

On Sun, 02 Mar 2008 19:36:51 -0800, CC <crobc@BOGUS.sbcglobal.net>
wrote:

>Greetings,
>
>I wish to transfer binary data in the simplest way possible from a PC to 
>either an AVR or TMS320F2812 embedded target via USB.
>
>The PC will be running a Python/wx.Python program and use PySerial to 
>send data out through the virtual COMM port produced by using a FTDI 
>FT245BM USB-to-FIFO parallel interface chip on the target.  Ideally this 
>program will be useable on Linux as well as Windoze machines.
>
>I estimate that I need up to 120kB/s (without overhead) of data 
>throughput to transfer 30000 points per second of 32-bit words 
>containing vector graphic X,Y,RGB points for a laser show project.
>
>The FTDI chip says it can do 300kB/s with the virtual COMM driver, so 
>this should work unless they are fibbing.
>
>I have gained some experience recently by writing a set of low-level 
>serial driver functions for the F2812.  Then I wrote a command parser 
>for it so the F2812 can respond to text commands and send back various 
>reports.  This is all via text terminal.  That was my first foray into 
>communication interface programming.  It was fun and not too difficult.
>
>Now I need to transfer binary data.  The project here won't need any 
>human readable command line.  I also don't want to increase overhead by 
>having to reserve the 8th bit for example to distinguish control codes 
>from data, or use some text rendition of the data, etc.  Just straight 
>binary.
>
>Thus, I suspect the simplest way for the target to behave is to 
>implement a state machine something like this:
>
>data:                   state:
>
><none yet>              1. wait for start code
>0x1234                  2. (start code received) wait for size field
>0x0123                  3. (16-bit size field received, specifying 291
>                            words to follow) wait for data
>0xdddddddd              4. (data incoming) count data words until 291
>0xdddddddd                  received
>0xdddddddd
>...
>0xdddddddd              5. (291st word received) wait for CRC state
>0x3E5B                  6. (16-bit CRC received) goto state 1
>
>Now of course, each state transistion can send a code back to the sender 
>telling whether things are proceeding smoothly.  Another state 
>transition path to add might be:
>
>data:
>
><none yet>              1. wait for start code
>0x5678                  7. (request for ACK received) send ACK, goto
>                            state 1
>
>That way if the receiver is stuck waiting for data and the sender 
>doesn't know what state receiver is in, the sender can request ACK until 
>the data receive counter transitions things back to state 1.
>
>Is there anything more needed than this (aside from additional fields in 
>the protocol)?
>
>Since there is a specific code to tell the receiver that a data package 
>has begun, when the receiver is in state 1, any sort of commands can 
>also be issued by using any codes other than the 0x1234 example.
>
>Any comments would be welcome regarding my budding protocol idea.
>
>
>Good day!


For a first time protocol all i can say is dont use straight binary,
use ascii (start sentinals and end sentials etc) that way you dont
have to worry about escape sequences etc. 

"CC" <crobc@BOGUS.sbcglobal.net> a &#4294967295;crit dans le message news:
fqfrok0bl5@news5.newsguy.com...
> Greetings,
>
> I wish to transfer binary data in the simplest way possible from a PC to
> either an AVR or TMS320F2812 embedded target via USB.
>
> The PC will be running a Python/wx.Python program and use PySerial to
> send data out through the virtual COMM port produced by using a FTDI
> FT245BM USB-to-FIFO parallel interface chip on the target.  Ideally this
> program will be useable on Linux as well as Windoze machines.
>
> I estimate that I need up to 120kB/s (without overhead) of data
> throughput to transfer 30000 points per second of 32-bit words
> containing vector graphic X,Y,RGB points for a laser show project.
>
> The FTDI chip says it can do 300kB/s with the virtual COMM driver, so
> this should work unless they are fibbing.
>
> I have gained some experience recently by writing a set of low-level
> serial driver functions for the F2812.  Then I wrote a command parser
> for it so the F2812 can respond to text commands and send back various
> reports.  This is all via text terminal.  That was my first foray into
> communication interface programming.  It was fun and not too difficult.
>
> Now I need to transfer binary data.  The project here won't need any
> human readable command line.  I also don't want to increase overhead by
> having to reserve the 8th bit for example to distinguish control codes
> from data, or use some text rendition of the data, etc.  Just straight
> binary.
>
> Thus, I suspect the simplest way for the target to behave is to
> implement a state machine something like this:
>
> data:                   state:
>
> <none yet>              1. wait for start code
> 0x1234                  2. (start code received) wait for size field
> 0x0123                  3. (16-bit size field received, specifying 291
>                             words to follow) wait for data
> 0xdddddddd              4. (data incoming) count data words until 291
> 0xdddddddd                  received
> 0xdddddddd
> ...
> 0xdddddddd              5. (291st word received) wait for CRC state
> 0x3E5B                  6. (16-bit CRC received) goto state 1
>
> Now of course, each state transistion can send a code back to the sender
> telling whether things are proceeding smoothly.  Another state
> transition path to add might be:
>
> data:
>
> <none yet>              1. wait for start code
> 0x5678                  7. (request for ACK received) send ACK, goto
>                             state 1
>
> That way if the receiver is stuck waiting for data and the sender
> doesn't know what state receiver is in, the sender can request ACK until
> the data receive counter transitions things back to state 1.
>
> Is there anything more needed than this (aside from additional fields in
> the protocol)?
>
> Since there is a specific code to tell the receiver that a data package
> has begun, when the receiver is in state 1, any sort of commands can
> also be issued by using any codes other than the 0x1234 example.
>
> Any comments would be welcome regarding my budding protocol idea.
>

Some ideas

You can implement the protocol you describe but it is rather
involved and you will have to validate it. There are tools that
can be used for the simulation and validation of protocols.
You will need in particular to check all possible circumstances.

Alternatively, you can implement a very simple protocol with:
- A fixed size frame : you know in advance the amount
  of data you should receive. You can use a single timeout.
  You will have only to pad the last frame.
- A single frame structure. Fixed size header + fixed size data +
   fixed size CRC.
- You can compress your data before transmission which will
   increase throughput.

The Real Andy wrote:
> For a first time protocol all i can say is dont use straight binary,
> use ascii (start sentinals and end sentials etc) that way you dont
> have to worry about escape sequences etc. 


Thanks for the input.

Since I have total control over the receiving end driver code and can 
design it to provide a getc() that doesn't process any escape sequences, 
then as long at the outgoing bytes on from the PC are also not processed 
by the PC, there is no problem with binary.

As far as I have seen so far, it is the case that outgoing bytes on the 
PC are emitted directly.

The problem with ASCII is that it will at least double the physical 
layer throughput needed to acheive a certain high-level data throughput. 
  Ie, if I took each binary byte and sent it as two ASCII text chars, 
one for each nibble, which is a common way of transmitting binary data 
in ASCII form, while allowing control chars to be treated as control chars.


-- 
Good day!

____________________________________
CRC
crobcREMOVETHIS@BOGUSsbcglobal.net
NOTE, delete texts: "REMOVETHIS" and
"BOGUS" from email address to reply.

Lanarcam wrote:
> You can implement the protocol you describe but it is rather
> involved and you will have to validate it. There are tools that
> can be used for the simulation and validation of protocols.
> You will need in particular to check all possible circumstances.

Thanks for the input.

Validate it using tools?  This isn't a very critical application nor a 
complex protocol.  I would think I could validate it quite easily for 
the anticipated manners in which things could go right/wrong by hand 
with creative use of the debugging tools available.  Python is 
interpreted, so it should be very easy to hand send binary data using 
PySerial and the Python command interpreter.  Or using a terminal 
capable of binary interaction.  At the target side, the 'F2812 debugging 
tools would make this very easy.

Thus, I don't see any need to simulate the protocol using other than the 
actual hardware and software in this case.

> Alternatively, you can implement a very simple protocol with:
> - A fixed size frame : you know in advance the amount
>   of data you should receive. You can use a single timeout.
>   You will have only to pad the last frame.
> - A single frame structure. Fixed size header + fixed size data +
>    fixed size CRC.

Now the idea of a fixed frame size is interesting.  The data is a vector 
graphic frame.  So either it must include a count of the points to 
render so that the target knows not to attempt to render null data 
points (because the graphic frames can have variable point counts up to 
a limit of 1250 points/frame), or it must include a count of the points 
so that the protocol knows when the end of the data has been reached.

Either way it has to have a count!  I will have to think about this more.

> - You can compress your data before transmission which will
>    increase throughput.

That's probably not needed in this case.  Remember: simple!



-- 
Good day!

____________________________________
CRC
crobcREMOVETHIS@BOGUSsbcglobal.net
NOTE, delete texts: "REMOVETHIS" and
"BOGUS" from email address to reply.

Hans Odeberg wrote:
> Are you not reinventing a wheel that you already have installed? USB
> already handles ack/nak and CRC protection of your data. Or is there
> some peculiarity of the FTDI chip here?

Thanks for your input.

Perhaps you are right.  I suppose if a byte is available to read from 
the FTDI chip, then it is certain to be valid.

But that is at the lowest level of single bytes, since the USB driver in 
the PC will be emulating a serial COMM port.  Thus, I still need some 
higher level handshaking for the target/PC to even know that one another 
exists and that they are communicating effectively.

> Are you sending real-time data? As in "if the sent data is corrupt, we
> do not have time to ask for a resend, so we will just skip it"? If so,
> maybe you should skip CRCs altoghether and use USB isochronous mode.
> If not, you will need some way for the receiving end to tell the
> transmitter that something went wrong (faulty CRC, not enough bytes
> received, faulty header data, etc.) and request a retransmission.

It is real-time in aggregate, but not per byte.  The plan is that a full 
new graphic frame will be sent to the target before the present one has 
finished rendering.  Then the target will simply switch to the next 
frame when it completes the present one.  If a frame fails to transmit 
correctly, then yes it would be sensible for the target to just repeat 
rendering the existing frame.  A minor visual glitch in the animation 
would result.

The likelyhood of this sort of trouble is probably extremely low, however.

It does seem that CRC might be unneeded.  I will have to get more in 
depth with the FTDI chip to confirm this.  By not doing CRC, I will very 
quickly get a statistical sense of data transmission reliability since 
it will be visual!  A flipped bit somewhere (particularly in the X,Y 
coordinates) will cause a very visible glitch in the vector graphic.

I am certain though that I will not have any control over the USB mode 
if I am using the FTDI virtual COMM port emulation driver.  So 
isochronous mode won't be an option.  And I don't want to go there 
anyway, since that would be more complex.  Since the data throughput 
needs are met in the simplest mode, then that's what I want to use.



-- 
Good day!

____________________________________
CRC
crobcREMOVETHIS@BOGUSsbcglobal.net
NOTE, delete texts: "REMOVETHIS" and
"BOGUS" from email address to reply.