Forums

Designing a protocol for the first time

Started by CC March 2, 2008
Greetings,

I wish to transfer binary data in the simplest way possible from a PC to 
either an AVR or TMS320F2812 embedded target via USB.

The PC will be running a Python/wx.Python program and use PySerial to 
send data out through the virtual COMM port produced by using a FTDI 
FT245BM USB-to-FIFO parallel interface chip on the target.  Ideally this 
program will be useable on Linux as well as Windoze machines.

I estimate that I need up to 120kB/s (without overhead) of data 
throughput to transfer 30000 points per second of 32-bit words 
containing vector graphic X,Y,RGB points for a laser show project.

The FTDI chip says it can do 300kB/s with the virtual COMM driver, so 
this should work unless they are fibbing.

I have gained some experience recently by writing a set of low-level 
serial driver functions for the F2812.  Then I wrote a command parser 
for it so the F2812 can respond to text commands and send back various 
reports.  This is all via text terminal.  That was my first foray into 
communication interface programming.  It was fun and not too difficult.

Now I need to transfer binary data.  The project here won't need any 
human readable command line.  I also don't want to increase overhead by 
having to reserve the 8th bit for example to distinguish control codes 
from data, or use some text rendition of the data, etc.  Just straight 
binary.

Thus, I suspect the simplest way for the target to behave is to 
implement a state machine something like this:

data:                   state:

<none yet>              1. wait for start code
0x1234                  2. (start code received) wait for size field
0x0123                  3. (16-bit size field received, specifying 291
                            words to follow) wait for data
0xdddddddd              4. (data incoming) count data words until 291
0xdddddddd                  received
0xdddddddd
...
0xdddddddd              5. (291st word received) wait for CRC state
0x3E5B                  6. (16-bit CRC received) goto state 1

Now of course, each state transistion can send a code back to the sender 
telling whether things are proceeding smoothly.  Another state 
transition path to add might be:

data:

<none yet>              1. wait for start code
0x5678                  7. (request for ACK received) send ACK, goto
                            state 1

That way if the receiver is stuck waiting for data and the sender 
doesn't know what state receiver is in, the sender can request ACK until 
the data receive counter transitions things back to state 1.

Is there anything more needed than this (aside from additional fields in 
the protocol)?

Since there is a specific code to tell the receiver that a data package 
has begun, when the receiver is in state 1, any sort of commands can 
also be issued by using any codes other than the 0x1234 example.

Any comments would be welcome regarding my budding protocol idea.


Good day!




-- 
_____________________
Christopher R. Carlen
crobc@bogus-remove-me.sbcglobal.net
SuSE 9.1 Linux 2.6.5
CC wrote:
> Greetings, > > I wish to transfer binary data in the simplest way possible from a PC to > either an AVR or TMS320F2812 embedded target via USB. > > The PC will be running a Python/wx.Python program and use PySerial to > send data out through the virtual COMM port produced by using a FTDI > FT245BM USB-to-FIFO parallel interface chip on the target. Ideally this > program will be useable on Linux as well as Windoze machines. > > I estimate that I need up to 120kB/s (without overhead) of data > throughput to transfer 30000 points per second of 32-bit words > containing vector graphic X,Y,RGB points for a laser show project. > > The FTDI chip says it can do 300kB/s with the virtual COMM driver, so > this should work unless they are fibbing. > > I have gained some experience recently by writing a set of low-level > serial driver functions for the F2812. Then I wrote a command parser > for it so the F2812 can respond to text commands and send back various > reports. This is all via text terminal. That was my first foray into > communication interface programming. It was fun and not too difficult. > > Now I need to transfer binary data. The project here won't need any > human readable command line. I also don't want to increase overhead by > having to reserve the 8th bit for example to distinguish control codes > from data, or use some text rendition of the data, etc. Just straight > binary. > > Thus, I suspect the simplest way for the target to behave is to > implement a state machine something like this: > > data: state: > > <none yet> 1. wait for start code > 0x1234 2. (start code received) wait for size field > 0x0123 3. (16-bit size field received, specifying 291 > words to follow) wait for data > 0xdddddddd 4. (data incoming) count data words until 291 > 0xdddddddd received > 0xdddddddd > ... > 0xdddddddd 5. (291st word received) wait for CRC state > 0x3E5B 6. (16-bit CRC received) goto state 1 > > Now of course, each state transistion can send a code back to the sender > telling whether things are proceeding smoothly. Another state > transition path to add might be: > > data: > > <none yet> 1. wait for start code > 0x5678 7. (request for ACK received) send ACK, goto > state 1 > > That way if the receiver is stuck waiting for data and the sender > doesn't know what state receiver is in, the sender can request ACK until > the data receive counter transitions things back to state 1. > > Is there anything more needed than this (aside from additional fields in > the protocol)? > > Since there is a specific code to tell the receiver that a data package > has begun, when the receiver is in state 1, any sort of commands can > also be issued by using any codes other than the 0x1234 example. > > Any comments would be welcome regarding my budding protocol idea.
X-modem CRC. Then you only have to write it for the target.
On Mar 3, 4:36 am, CC <cr...@BOGUS.sbcglobal.net> wrote:

> I wish to transfer binary data in the simplest way possible from a PC to > either an AVR or TMS320F2812 embedded target via USB.
You may be interested in RFC 1549 "PPP in HDLC framing", see http://www.ietf.org/rfc/rfc1549.txt The frame format is simple and robust, and has fairly low overhead.
Le Sun, 02 Mar 2008 19:36:51 -0800, CC a &eacute;crit:

> Greetings, > > I wish to transfer binary data in the simplest way possible from a PC to > either an AVR or TMS320F2812 embedded target via USB. > > The PC will be running a Python/wx.Python program and use PySerial to > send data out through the virtual COMM port produced by using a FTDI > FT245BM USB-to-FIFO parallel interface chip on the target. Ideally this > program will be useable on Linux as well as Windoze machines. > > I estimate that I need up to 120kB/s (without overhead) of data > throughput to transfer 30000 points per second of 32-bit words > containing vector graphic X,Y,RGB points for a laser show project. > > The FTDI chip says it can do 300kB/s with the virtual COMM driver, so > this should work unless they are fibbing. > > I have gained some experience recently by writing a set of low-level > serial driver functions for the F2812. Then I wrote a command parser > for it so the F2812 can respond to text commands and send back various > reports. This is all via text terminal. That was my first foray into > communication interface programming. It was fun and not too difficult. > > Now I need to transfer binary data. The project here won't need any > human readable command line. I also don't want to increase overhead by > having to reserve the 8th bit for example to distinguish control codes > from data, or use some text rendition of the data, etc. Just straight > binary. > > Thus, I suspect the simplest way for the target to behave is to > implement a state machine something like this: > > data: state: > > <none yet> 1. wait for start code > 0x1234 2. (start code received) wait for size field > 0x0123 3. (16-bit size field received, specifying 291 > words to follow) wait for data > 0xdddddddd 4. (data incoming) count data words until 291 > 0xdddddddd received > 0xdddddddd > ... > 0xdddddddd 5. (291st word received) wait for CRC state > 0x3E5B 6. (16-bit CRC received) goto state 1 > > Now of course, each state transistion can send a code back to the sender > telling whether things are proceeding smoothly. Another state > transition path to add might be: > > data: > > <none yet> 1. wait for start code > 0x5678 7. (request for ACK received) send ACK, goto > state 1 > > That way if the receiver is stuck waiting for data and the sender > doesn't know what state receiver is in, the sender can request ACK until > the data receive counter transitions things back to state 1. > > Is there anything more needed than this (aside from additional fields in > the protocol)? > > Since there is a specific code to tell the receiver that a data package > has begun, when the receiver is in state 1, any sort of commands can > also be issued by using any codes other than the 0x1234 example. > > Any comments would be welcome regarding my budding protocol idea. > > > Good day! > > > >
I agree with Jim and Arlet although Jim solution is easier to implement -- HBV
On 3 Mar, 04:36, CC <cr...@BOGUS.sbcglobal.net> wrote:
> Greetings, > > I wish to transfer binary data in the simplest way possible from a PC to > either an AVR or TMS320F2812 embedded target via USB. > > The PC will be running a Python/wx.Python program and use PySerial to > send data out through the virtual COMM port produced by using a FTDI > FT245BM USB-to-FIFO parallel interface chip on the target. =A0Ideally this=
> program will be useable on Linux as well as Windoze machines. > > I estimate that I need up to 120kB/s (without overhead) of data > throughput to transfer 30000 points per second of 32-bit words > containing vector graphic X,Y,RGB points for a laser show project. > > The FTDI chip says it can do 300kB/s with the virtual COMM driver, so > this should work unless they are fibbing. > > I have gained some experience recently by writing a set of low-level > serial driver functions for the F2812. =A0Then I wrote a command parser > for it so the F2812 can respond to text commands and send back various > reports. =A0This is all via text terminal. =A0That was my first foray into=
> communication interface programming. =A0It was fun and not too difficult. > > Now I need to transfer binary data. =A0The project here won't need any > human readable command line. =A0I also don't want to increase overhead by > having to reserve the 8th bit for example to distinguish control codes > from data, or use some text rendition of the data, etc. =A0Just straight > binary. > > Thus, I suspect the simplest way for the target to behave is to > implement a state machine something like this: > > data: =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 state: > > <none yet> =A0 =A0 =A0 =A0 =A0 =A0 =A01. wait for start code > 0x1234 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A02. (start code received) wait fo=
r size field
> 0x0123 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A03. (16-bit size field received, =
specifying 291
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 words to follow) w=
ait for data
> 0xdddddddd =A0 =A0 =A0 =A0 =A0 =A0 =A04. (data incoming) count data words =
until 291
> 0xdddddddd =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0received > 0xdddddddd > ... > 0xdddddddd =A0 =A0 =A0 =A0 =A0 =A0 =A05. (291st word received) wait for CR=
C state
> 0x3E5B =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A06. (16-bit CRC received) goto st=
ate 1
> > Now of course, each state transistion can send a code back to the sender > telling whether things are proceeding smoothly. =A0Another state > transition path to add might be: > > data: > > <none yet> =A0 =A0 =A0 =A0 =A0 =A0 =A01. wait for start code > 0x5678 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A07. (request for ACK received) se=
nd ACK, goto
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 state 1 > > That way if the receiver is stuck waiting for data and the sender > doesn't know what state receiver is in, the sender can request ACK until > the data receive counter transitions things back to state 1. > > Is there anything more needed than this (aside from additional fields in > the protocol)? > > Since there is a specific code to tell the receiver that a data package > has begun, when the receiver is in state 1, any sort of commands can > also be issued by using any codes other than the 0x1234 example. > > Any comments would be welcome regarding my budding protocol idea. > > Good day! > > -- > _____________________ > Christopher R. Carlen > cr...@bogus-remove-me.sbcglobal.net > SuSE 9.1 Linux 2.6.5
Are you not reinventing a wheel that you already have installed? USB already handles ack/nak and CRC protection of your data. Or is there some peculiarity of the FTDI chip here? Are you sending real-time data? As in "if the sent data is corrupt, we do not have time to ask for a resend, so we will just skip it"? If so, maybe you should skip CRCs altoghether and use USB isochronous mode. If not, you will need some way for the receiving end to tell the transmitter that something went wrong (faulty CRC, not enough bytes received, faulty header data, etc.) and request a retransmission.
On Sun, 02 Mar 2008 19:36:51 -0800, CC <crobc@BOGUS.sbcglobal.net>
wrote:

>Greetings, > >I wish to transfer binary data in the simplest way possible from a PC to >either an AVR or TMS320F2812 embedded target via USB. > >The PC will be running a Python/wx.Python program and use PySerial to >send data out through the virtual COMM port produced by using a FTDI >FT245BM USB-to-FIFO parallel interface chip on the target. Ideally this >program will be useable on Linux as well as Windoze machines. > >I estimate that I need up to 120kB/s (without overhead) of data >throughput to transfer 30000 points per second of 32-bit words >containing vector graphic X,Y,RGB points for a laser show project. > >The FTDI chip says it can do 300kB/s with the virtual COMM driver, so >this should work unless they are fibbing. > >I have gained some experience recently by writing a set of low-level >serial driver functions for the F2812. Then I wrote a command parser >for it so the F2812 can respond to text commands and send back various >reports. This is all via text terminal. That was my first foray into >communication interface programming. It was fun and not too difficult. > >Now I need to transfer binary data. The project here won't need any >human readable command line. I also don't want to increase overhead by >having to reserve the 8th bit for example to distinguish control codes >from data, or use some text rendition of the data, etc. Just straight >binary. > >Thus, I suspect the simplest way for the target to behave is to >implement a state machine something like this: > >data: state: > ><none yet> 1. wait for start code >0x1234 2. (start code received) wait for size field >0x0123 3. (16-bit size field received, specifying 291 > words to follow) wait for data >0xdddddddd 4. (data incoming) count data words until 291 >0xdddddddd received >0xdddddddd >... >0xdddddddd 5. (291st word received) wait for CRC state >0x3E5B 6. (16-bit CRC received) goto state 1 > >Now of course, each state transistion can send a code back to the sender >telling whether things are proceeding smoothly. Another state >transition path to add might be: > >data: > ><none yet> 1. wait for start code >0x5678 7. (request for ACK received) send ACK, goto > state 1 > >That way if the receiver is stuck waiting for data and the sender >doesn't know what state receiver is in, the sender can request ACK until >the data receive counter transitions things back to state 1. > >Is there anything more needed than this (aside from additional fields in >the protocol)? > >Since there is a specific code to tell the receiver that a data package >has begun, when the receiver is in state 1, any sort of commands can >also be issued by using any codes other than the 0x1234 example. > >Any comments would be welcome regarding my budding protocol idea. > > >Good day!
For a first time protocol all i can say is dont use straight binary, use ascii (start sentinals and end sentials etc) that way you dont have to worry about escape sequences etc.
"CC" <crobc@BOGUS.sbcglobal.net> a &#2013265929;crit dans le message news:
fqfrok0bl5@news5.newsguy.com...
> Greetings, > > I wish to transfer binary data in the simplest way possible from a PC to > either an AVR or TMS320F2812 embedded target via USB. > > The PC will be running a Python/wx.Python program and use PySerial to > send data out through the virtual COMM port produced by using a FTDI > FT245BM USB-to-FIFO parallel interface chip on the target. Ideally this > program will be useable on Linux as well as Windoze machines. > > I estimate that I need up to 120kB/s (without overhead) of data > throughput to transfer 30000 points per second of 32-bit words > containing vector graphic X,Y,RGB points for a laser show project. > > The FTDI chip says it can do 300kB/s with the virtual COMM driver, so > this should work unless they are fibbing. > > I have gained some experience recently by writing a set of low-level > serial driver functions for the F2812. Then I wrote a command parser > for it so the F2812 can respond to text commands and send back various > reports. This is all via text terminal. That was my first foray into > communication interface programming. It was fun and not too difficult. > > Now I need to transfer binary data. The project here won't need any > human readable command line. I also don't want to increase overhead by > having to reserve the 8th bit for example to distinguish control codes > from data, or use some text rendition of the data, etc. Just straight > binary. > > Thus, I suspect the simplest way for the target to behave is to > implement a state machine something like this: > > data: state: > > <none yet> 1. wait for start code > 0x1234 2. (start code received) wait for size field > 0x0123 3. (16-bit size field received, specifying 291 > words to follow) wait for data > 0xdddddddd 4. (data incoming) count data words until 291 > 0xdddddddd received > 0xdddddddd > ... > 0xdddddddd 5. (291st word received) wait for CRC state > 0x3E5B 6. (16-bit CRC received) goto state 1 > > Now of course, each state transistion can send a code back to the sender > telling whether things are proceeding smoothly. Another state > transition path to add might be: > > data: > > <none yet> 1. wait for start code > 0x5678 7. (request for ACK received) send ACK, goto > state 1 > > That way if the receiver is stuck waiting for data and the sender > doesn't know what state receiver is in, the sender can request ACK until > the data receive counter transitions things back to state 1. > > Is there anything more needed than this (aside from additional fields in > the protocol)? > > Since there is a specific code to tell the receiver that a data package > has begun, when the receiver is in state 1, any sort of commands can > also be issued by using any codes other than the 0x1234 example. > > Any comments would be welcome regarding my budding protocol idea. >
Some ideas You can implement the protocol you describe but it is rather involved and you will have to validate it. There are tools that can be used for the simulation and validation of protocols. You will need in particular to check all possible circumstances. Alternatively, you can implement a very simple protocol with: - A fixed size frame : you know in advance the amount of data you should receive. You can use a single timeout. You will have only to pad the last frame. - A single frame structure. Fixed size header + fixed size data + fixed size CRC. - You can compress your data before transmission which will increase throughput.
The Real Andy wrote:
> For a first time protocol all i can say is dont use straight binary, > use ascii (start sentinals and end sentials etc) that way you dont > have to worry about escape sequences etc.
Thanks for the input. Since I have total control over the receiving end driver code and can design it to provide a getc() that doesn't process any escape sequences, then as long at the outgoing bytes on from the PC are also not processed by the PC, there is no problem with binary. As far as I have seen so far, it is the case that outgoing bytes on the PC are emitted directly. The problem with ASCII is that it will at least double the physical layer throughput needed to acheive a certain high-level data throughput. Ie, if I took each binary byte and sent it as two ASCII text chars, one for each nibble, which is a common way of transmitting binary data in ASCII form, while allowing control chars to be treated as control chars. -- Good day! ____________________________________ CRC crobcREMOVETHIS@BOGUSsbcglobal.net NOTE, delete texts: "REMOVETHIS" and "BOGUS" from email address to reply.
Lanarcam wrote:
> You can implement the protocol you describe but it is rather > involved and you will have to validate it. There are tools that > can be used for the simulation and validation of protocols. > You will need in particular to check all possible circumstances.
Thanks for the input. Validate it using tools? This isn't a very critical application nor a complex protocol. I would think I could validate it quite easily for the anticipated manners in which things could go right/wrong by hand with creative use of the debugging tools available. Python is interpreted, so it should be very easy to hand send binary data using PySerial and the Python command interpreter. Or using a terminal capable of binary interaction. At the target side, the 'F2812 debugging tools would make this very easy. Thus, I don't see any need to simulate the protocol using other than the actual hardware and software in this case.
> Alternatively, you can implement a very simple protocol with: > - A fixed size frame : you know in advance the amount > of data you should receive. You can use a single timeout. > You will have only to pad the last frame. > - A single frame structure. Fixed size header + fixed size data + > fixed size CRC.
Now the idea of a fixed frame size is interesting. The data is a vector graphic frame. So either it must include a count of the points to render so that the target knows not to attempt to render null data points (because the graphic frames can have variable point counts up to a limit of 1250 points/frame), or it must include a count of the points so that the protocol knows when the end of the data has been reached. Either way it has to have a count! I will have to think about this more.
> - You can compress your data before transmission which will > increase throughput.
That's probably not needed in this case. Remember: simple! -- Good day! ____________________________________ CRC crobcREMOVETHIS@BOGUSsbcglobal.net NOTE, delete texts: "REMOVETHIS" and "BOGUS" from email address to reply.
Hans Odeberg wrote:
> Are you not reinventing a wheel that you already have installed? USB > already handles ack/nak and CRC protection of your data. Or is there > some peculiarity of the FTDI chip here?
Thanks for your input. Perhaps you are right. I suppose if a byte is available to read from the FTDI chip, then it is certain to be valid. But that is at the lowest level of single bytes, since the USB driver in the PC will be emulating a serial COMM port. Thus, I still need some higher level handshaking for the target/PC to even know that one another exists and that they are communicating effectively.
> Are you sending real-time data? As in "if the sent data is corrupt, we > do not have time to ask for a resend, so we will just skip it"? If so, > maybe you should skip CRCs altoghether and use USB isochronous mode. > If not, you will need some way for the receiving end to tell the > transmitter that something went wrong (faulty CRC, not enough bytes > received, faulty header data, etc.) and request a retransmission.
It is real-time in aggregate, but not per byte. The plan is that a full new graphic frame will be sent to the target before the present one has finished rendering. Then the target will simply switch to the next frame when it completes the present one. If a frame fails to transmit correctly, then yes it would be sensible for the target to just repeat rendering the existing frame. A minor visual glitch in the animation would result. The likelyhood of this sort of trouble is probably extremely low, however. It does seem that CRC might be unneeded. I will have to get more in depth with the FTDI chip to confirm this. By not doing CRC, I will very quickly get a statistical sense of data transmission reliability since it will be visual! A flipped bit somewhere (particularly in the X,Y coordinates) will cause a very visible glitch in the vector graphic. I am certain though that I will not have any control over the USB mode if I am using the FTDI virtual COMM port emulation driver. So isochronous mode won't be an option. And I don't want to go there anyway, since that would be more complex. Since the data throughput needs are met in the simplest mode, then that's what I want to use. -- Good day! ____________________________________ CRC crobcREMOVETHIS@BOGUSsbcglobal.net NOTE, delete texts: "REMOVETHIS" and "BOGUS" from email address to reply.