Here's some sample code for the sending and receiving sides. It should
illustrate the "escaping" method, as well as how trivial it is. Much of
the ~200 lines of code is either comment or whitespace.
Of course, if you can afford the big overhead for ASCII encoding, that
might be better. Keep in mind that the overhead is both transmission
time (could be up to double) and decode time at the receiver.
The below is straight C. I don't recall how it might be handled by the
Z-World compiler (e.g. enums and initialized auto variables ring some
bells, for some reason).
#define ESC 0x1B
#define STX 0x02
#define ETX 0x03
/********************************************************************
SENDER SIDE
*******************************************************************/
/* a function that sends a single byte over the communications
medium */
extern void SendByte( unsigned char byte );
/* This function sends the packet, inserting ESC characters as
required. The caller must have already performed the encryption
and CRC calculations. */
void SendPacket(
unsigned char *data, unsigned short int dataLen,
unsigned char *CRC, unsigned short int CRCLen )
{
/* Initial STX */
SendByte( STX );
/* send , inserting ESC as required */
while ( dataLen-- )
{
unsigned char byte = *data++;
if ( byte == ESC || byte == STX || byte == ETX )
SendByte( ESC );
SendByte( byte );
}
/* terminating ETX */
SendByte( ETX );
/* send the fixed-length CRC (no escaping) */
while ( CRCLen-- )
SendByte( *CRC++ );
}
/********************************************************************
RECEIVER SIDE
*******************************************************************/
unsigned char RxDataAvailable( void )
{
/* Return 0 if no data is available. Otherwise,
return non-zero */
return 0;
}
unsigned char GetRxData( void )
{
/* Return next available received byte. Caller
must ensure that data is available (via above
function) before calling this */
return 0;
}
#define MAX_DATA_LENGTH 1024
unsigned char RxData[ MAX_DATA_LENGTH ];
unsigned short int RxDataLen;
#define CRC_LEN 16
unsigned char CRC[ CRC_LEN ];
unsigned short int CRCLen;
unsigned char Escaped;
enum
{
_RS_NewPacket,
_RS_WaitSTX,
_RS_GetData,
_RS_GetCRC
} RxState;
/* Initialize rx state machine. Call this before the first call
to RxHandler() */
void RxInit( void )
{
RxState = _RS_NewPacket;
}
/* A function that verifies the CRC */
extern unsigned char VerifyCRC(
unsigned char *data, unsigned short int dataLen,
unsigned char *crc, unsigned short int crcLen );
/* A function that decrypts the received data. Decryption is done
"in place" (replacing raw data).
Function returns count of decrypted bytes */
extern unsigned short int DecryptRxData(
unsigned char *data, unsigned short int len );
/* A function that processes the decrypted data */
extern void ProcessRxData(
unsigned char *data, unsigned short int len );
/* Call this function periodically to process received bytes */
void RxHandler( void )
{
/* stay here as long as there is received data */
while ( RxDataAvailable() )
{
/* go get the next received byte */
unsigned char byte = GetRxData();
/* handle it */
switch ( RxState )
{
default:
case _RS_NewPacket:
Escaped = 0;
RxState = _RS_WaitSTX;
/* intentional fall-through */
case _RS_WaitSTX:
if ( Escaped )
Escaped = 0;
else
if ( byte == ESC )
Escaped = 1;
else
if ( byte == STX )
{
RxDataLen = 0;
RxState = _RS_GetData;
}
break;
case _RS_GetData:
if ( Escaped )
{
/* last rx byte was ESC, so this byte is handled
literally */
if ( RxDataLen < MAX_DATA_LENGTH )
RxData[ RxDataLen++ ] = byte;
Escaped = 0;
}
else
if ( byte == ESC )
/* next byte is to be handled literally */
Escaped = 1;
else
if ( byte == STX )
/* unexpected un-escaped STX... must be start of
packet, so reset data counter */
RxDataLen = 0;
else
if ( byte == ETX )
{
/* end-of-packet indicator - go get CRC */
CRCLen = 0;
RxState = _RS_GetCRC;
}
else
/* not escaped and not a special character, so just
store it */
{
if ( RxDataLen < MAX_DATA_LENGTH )
RxData[ RxDataLen++ ] = byte;
}
break;
case _RS_GetCRC:
/* Get CRC. Presumes fixed-length CRC, data not escaped. */
CRC[ CRCLen++ ] = byte;
if ( CRCLen == CRC_LEN )
{
/* got all expected CRC bytes. go check the CRC */
if ( VerifyCRC( RxData, RxDataLen, CRC, CRCLen ) )
{
/* decrypt it, retrieving the decrypted length */
unsigned short int decLen = DecryptRxData( RxData, RxDatLen );
/* and process the decrypted data */
ProcessRxData( RxData, decLen );
}
/* and prepare for next packet */
RxState = _RS_NewPacket;
}
break;
}
}
}
PARSING DATA
Started by ●January 6, 2012
Reply by ●January 6, 20122012-01-06
Reply by ●January 6, 20122012-01-06
--->>
Simple, send all DATA as a char and only allow (0x30 thru 0x39) & (0x41 thru 0x0x5a) & (0x61 thru 0x7a) values through as data, then all of your ASCII control characters will be available for control.
This also makes Parsing easier.
@Jim, it is true we can do that if the data that we send are not in a encrypted condition. If the data is encrypted, we can not guarantee that the results encrypt there is no "ASCII control character".
---
My suggestion is to convert your encrypted data to ASCII hex characters. The STX/ETX method only works if the data are ASCII characters.
@Seec, I've attempted to do this and succeed, the problem is the data length to be 2x if convert to ASCII HEX characters. Data 16 bytes ASCII characters will converted to 32 bytes ASCII HEX character. Performance will decrease if send the data very long.
---
ASCII encoding is an automatic overhead penalty (up to 100% of data, depending on implementation). That is, representing each byte of data as two ASCII bytes (i.e. 0x02 becomes `0' `2'), automatically doubles the amount of data transferred. This may not be acceptable to the O.P.
@don, you are right, That's the reason why I choose to transfer in the form ASCII character because the data can be half shorter compared with ASCII HEX character and that will affect in the poll data performace.
---
@ALL, Thank you for the advice. From your input, I have an idea to keep sending data in ASCII character(1 byte), but the separator using the ASCII HEX character(2 bytes). Look below... I know this is not standard, but I've tried and succeeded and get better performance. Very fast with long data. ALL THANK YOU... :)
'02' 0x2 0x3 0x4 0x5 0x6... '03' 0x3 0x4 0x3...
ascii hex .... ascii char ...... ascii hex ..ascii char..
(2 bytes) (n bytes) (2bytes) (16 bytes)
--- In r..., "Don Starr" wrote:
>
> Of course, TCP/IP by itself won't help. Since it's just a stream of
> bytes, the sender and receiver still need to coordinate
> "packetization". In the O.P.'s case, he's doing it with STX/ETX. But,
> since STX and ETX can appear in his data, he needs to somehow tell the
> receiver that the embedded STX/ETX are raw data and not packet
> delimiters.
>
>
>
> UDP might work, since it's a message-based protocol. "Packetization"
> is inherent.
>
>
>
> Neither TCP nor UDP is appropriate if the O.P. doesn't want to incur
> that code and/or transmission overhead, or if he can't incorporate
> them into his project(s) (e.g. if a stack isn't available for either
> the sender or receiver. we don't know that both are Rabbits).
>
>
>
> ASCII encoding is an automatic overhead penalty (up to 100% of data,
> depending on implementation). That is, representing each byte of data
> as two ASCII bytes (i.e. 0x02 becomes '0' '2'), automatically doubles
> the amount of data transferred. This may not be acceptable to the O.P.
>
>
>
> "Escaping" random data as I've suggested would be a 1.2% overhead
> penalty (on average, presuming that the 3 "special characters" occur
> at the same frequency as all other byte values). The maximum overhead
> penalty is 100%, if all bytes sent are one of the 3 special character
> values; this seems very unlikely, if the O.P.'s encryption is working.
> Finally, the implementation requirements for this method are trivial.
> On the sending side, it's one C statement. On the receiving side, it's
> a new flag plus a couple of tests. Much easier/faster than
> ASCII-encoding everything or bringing in a new protocol stack.
>
>
>
> -Don
>
>
>
>
>
> _____
>
> From: r... [mailto:r...]
> On Behalf Of Jim Ashby
> Sent: Friday, 06 January, 2012 10:39
> To: r...
> Subject: RE: [rabbit-semi] Re: PARSING DATA
>
>
>
>
> Don, this is understood and understood well.
>
>
>
> It is a standard method for data compression, used along with a
> special char to demark a sequence of repeating characters which is
> followed by the number of repeating chars and the repeating char.
>
>
>
> So if this method is chosen, you will need to add a demark char for
> each control code needed, and also the method of allowing the
> detection and processing of the control characters.
>
>
>
> About 20 years ago I wrote that type of code, but it involved wrapping
> the data block with a structured format, which allowed me to determine
> if the spacial char was for data or control.
>
>
>
> That is why they use the structure of TCP/IP wrappers around data
> blocks, there is no need for special chars when the entire structure
> of the data block is wrapped.
>
> With the wrapped format you also have the ability to include variable
> data block length, forward error correction, etc...
>
>
>
> Unless an engineer is ready to create a new custom wrapper, it would
> be better to just use a known and proven TCP/IP, UDP, or MODBUSS,
> style transfer protocol.
>
>
>
> All are available from ZWorld (Rabbit).
>
>
>
> My 25 cents worth.
>
>
>
> Jim Ashby
>
>
>
>
>
>
>
> -------Original Message-------
>
>
>
> From: Don Starr Date: 1/6/2012 10:08:58 AM
>
> To: r...
>
> Subject: RE: [rabbit-semi] Re: PARSING DATA
>
>
>
>
>
> If the encrypted data has an escaped pair, each of those bytes is
> escaped. For example, if the encrypted data (before sending) had 0x7E
> 0x02, then the actual sent data would be 0x7E 0x7E 0x7E 0x02
> (presuming the escape character was 0x7E). The receiver would discard
> the first and third 0x7E, resulting in 0x7E 0x02.
>
>
>
> unsigned char dstData[ MAX_DATA_LEN ];
>
> int dstLen;
>
> unsigned char esc;
>
> #define ESC 0x7E
>
> #define STX 0x02
>
> #define ETX 0x03
>
>
>
> unsigned char RxDataAvailable( void );
>
> unsigned char GetRxByte( void );
>
>
>
> esc = 0;
>
> dstLen = 0;
>
> while ( 1 )
>
> {
>
> if ( RxDataAvailable() )
>
> {
>
> unsigned char c = GetRxByte();
>
> if ( esc )
>
> {
>
> if ( dstLen < MAX_DATA_LEN )
>
> dstData[ dstLen++ ] = c;
>
> esc = 0;
>
> }
>
> else
>
> if ( c == ESC )
>
> esc = 1;
>
> else
>
> if ( c == STX )
>
> dstLen = 0;
>
> else
>
> if ( c == ETX )
>
> ; /* go get CRC and process packet */
>
> else
>
> if ( dstLen < MAX_DATA_LEN )
>
> dstData[ dstLen++ ] = c;
>
> }
>
> }
>
>
>
>
>
> _____
>
> From: r... [mailto:r...]
> On Behalf Of seecwriter
> Sent: Friday, 06 January, 2012 9:56
> To: r...
> Subject: [rabbit-semi] Re: PARSING DATA
>
>
>
>
>
> A problem with trying to escape special characters is that you can't
> guarantee that your encrypted data will not generate an escaped
> special character pair.
> My suggestion is to convert your encrypted data to ASCII hex
> characters. The STX/ETX method only works if the data are ASCII
> characters. Otherwise you will need to add a byte-count to the
> message.
>
> --- In r...
> , "siph4@"
> wrote:
> >
> > I had a problem with parsing the data. I have one packet data as
> bellow:
> >
> > |---------------------|.......> 1 packet data
> > STX DATA ETX CRC
> > |---|
> > |..............> Data will be encrypt with AES CBC/CFB
> > with KEY bytes and Initial Vector bytes
> >
> > |--------|
> > |...............> CRC 16 bytes with MD5 HASH
> > Calcula te Data and ETX.
> >
> > STX : 0x02 (1 bytes)
> > DATA (encrypted): 0x33 0xF3 0x06 0x02 0xF8 0x98 0x03 ... (16 bytes)
> > ETX : 0x03 (1 bytes)
> > CRC : 0x45 0x88 0x7E 0x23 0x56 0x93 0x03 ... (16 bytes)
> >
> >
> > I parsing the DATA PER PACKET based STX (0x02) separator.
> >
> > I just separate the DATA with the CRC based ETX (0x03) separator.
> >
> > I calculate the CRC with MD5 HASH to ensure that data is not
> corrupted in the network.
> >
> > Like the example above, there is a character with ascii 0x02 in the
> encrypted data. Possible values range are 0x00-0xFF in
> data, which exists between the value of some are used as separators of
> data packets(STX and ETX).
> >
> > My question, how can I distinguish 0x02 is STX, NOT PART OF THE DATA
> and also 0x03 is ETX, NOT PART OF THE DATA ?
> >
> >
> > Thank You.
> > Siph4
>
Simple, send all DATA as a char and only allow (0x30 thru 0x39) & (0x41 thru 0x0x5a) & (0x61 thru 0x7a) values through as data, then all of your ASCII control characters will be available for control.
This also makes Parsing easier.
@Jim, it is true we can do that if the data that we send are not in a encrypted condition. If the data is encrypted, we can not guarantee that the results encrypt there is no "ASCII control character".
---
My suggestion is to convert your encrypted data to ASCII hex characters. The STX/ETX method only works if the data are ASCII characters.
@Seec, I've attempted to do this and succeed, the problem is the data length to be 2x if convert to ASCII HEX characters. Data 16 bytes ASCII characters will converted to 32 bytes ASCII HEX character. Performance will decrease if send the data very long.
---
ASCII encoding is an automatic overhead penalty (up to 100% of data, depending on implementation). That is, representing each byte of data as two ASCII bytes (i.e. 0x02 becomes `0' `2'), automatically doubles the amount of data transferred. This may not be acceptable to the O.P.
@don, you are right, That's the reason why I choose to transfer in the form ASCII character because the data can be half shorter compared with ASCII HEX character and that will affect in the poll data performace.
---
@ALL, Thank you for the advice. From your input, I have an idea to keep sending data in ASCII character(1 byte), but the separator using the ASCII HEX character(2 bytes). Look below... I know this is not standard, but I've tried and succeeded and get better performance. Very fast with long data. ALL THANK YOU... :)
'02' 0x2 0x3 0x4 0x5 0x6... '03' 0x3 0x4 0x3...
ascii hex .... ascii char ...... ascii hex ..ascii char..
(2 bytes) (n bytes) (2bytes) (16 bytes)
--- In r..., "Don Starr" wrote:
>
> Of course, TCP/IP by itself won't help. Since it's just a stream of
> bytes, the sender and receiver still need to coordinate
> "packetization". In the O.P.'s case, he's doing it with STX/ETX. But,
> since STX and ETX can appear in his data, he needs to somehow tell the
> receiver that the embedded STX/ETX are raw data and not packet
> delimiters.
>
>
>
> UDP might work, since it's a message-based protocol. "Packetization"
> is inherent.
>
>
>
> Neither TCP nor UDP is appropriate if the O.P. doesn't want to incur
> that code and/or transmission overhead, or if he can't incorporate
> them into his project(s) (e.g. if a stack isn't available for either
> the sender or receiver. we don't know that both are Rabbits).
>
>
>
> ASCII encoding is an automatic overhead penalty (up to 100% of data,
> depending on implementation). That is, representing each byte of data
> as two ASCII bytes (i.e. 0x02 becomes '0' '2'), automatically doubles
> the amount of data transferred. This may not be acceptable to the O.P.
>
>
>
> "Escaping" random data as I've suggested would be a 1.2% overhead
> penalty (on average, presuming that the 3 "special characters" occur
> at the same frequency as all other byte values). The maximum overhead
> penalty is 100%, if all bytes sent are one of the 3 special character
> values; this seems very unlikely, if the O.P.'s encryption is working.
> Finally, the implementation requirements for this method are trivial.
> On the sending side, it's one C statement. On the receiving side, it's
> a new flag plus a couple of tests. Much easier/faster than
> ASCII-encoding everything or bringing in a new protocol stack.
>
>
>
> -Don
>
>
>
>
>
> _____
>
> From: r... [mailto:r...]
> On Behalf Of Jim Ashby
> Sent: Friday, 06 January, 2012 10:39
> To: r...
> Subject: RE: [rabbit-semi] Re: PARSING DATA
>
>
>
>
> Don, this is understood and understood well.
>
>
>
> It is a standard method for data compression, used along with a
> special char to demark a sequence of repeating characters which is
> followed by the number of repeating chars and the repeating char.
>
>
>
> So if this method is chosen, you will need to add a demark char for
> each control code needed, and also the method of allowing the
> detection and processing of the control characters.
>
>
>
> About 20 years ago I wrote that type of code, but it involved wrapping
> the data block with a structured format, which allowed me to determine
> if the spacial char was for data or control.
>
>
>
> That is why they use the structure of TCP/IP wrappers around data
> blocks, there is no need for special chars when the entire structure
> of the data block is wrapped.
>
> With the wrapped format you also have the ability to include variable
> data block length, forward error correction, etc...
>
>
>
> Unless an engineer is ready to create a new custom wrapper, it would
> be better to just use a known and proven TCP/IP, UDP, or MODBUSS,
> style transfer protocol.
>
>
>
> All are available from ZWorld (Rabbit).
>
>
>
> My 25 cents worth.
>
>
>
> Jim Ashby
>
>
>
>
>
>
>
> -------Original Message-------
>
>
>
> From: Don Starr Date: 1/6/2012 10:08:58 AM
>
> To: r...
>
> Subject: RE: [rabbit-semi] Re: PARSING DATA
>
>
>
>
>
> If the encrypted data has an escaped pair, each of those bytes is
> escaped. For example, if the encrypted data (before sending) had 0x7E
> 0x02, then the actual sent data would be 0x7E 0x7E 0x7E 0x02
> (presuming the escape character was 0x7E). The receiver would discard
> the first and third 0x7E, resulting in 0x7E 0x02.
>
>
>
> unsigned char dstData[ MAX_DATA_LEN ];
>
> int dstLen;
>
> unsigned char esc;
>
> #define ESC 0x7E
>
> #define STX 0x02
>
> #define ETX 0x03
>
>
>
> unsigned char RxDataAvailable( void );
>
> unsigned char GetRxByte( void );
>
>
>
> esc = 0;
>
> dstLen = 0;
>
> while ( 1 )
>
> {
>
> if ( RxDataAvailable() )
>
> {
>
> unsigned char c = GetRxByte();
>
> if ( esc )
>
> {
>
> if ( dstLen < MAX_DATA_LEN )
>
> dstData[ dstLen++ ] = c;
>
> esc = 0;
>
> }
>
> else
>
> if ( c == ESC )
>
> esc = 1;
>
> else
>
> if ( c == STX )
>
> dstLen = 0;
>
> else
>
> if ( c == ETX )
>
> ; /* go get CRC and process packet */
>
> else
>
> if ( dstLen < MAX_DATA_LEN )
>
> dstData[ dstLen++ ] = c;
>
> }
>
> }
>
>
>
>
>
> _____
>
> From: r... [mailto:r...]
> On Behalf Of seecwriter
> Sent: Friday, 06 January, 2012 9:56
> To: r...
> Subject: [rabbit-semi] Re: PARSING DATA
>
>
>
>
>
> A problem with trying to escape special characters is that you can't
> guarantee that your encrypted data will not generate an escaped
> special character pair.
> My suggestion is to convert your encrypted data to ASCII hex
> characters. The STX/ETX method only works if the data are ASCII
> characters. Otherwise you will need to add a byte-count to the
> message.
>
> --- In r...
> , "siph4@"
> wrote:
> >
> > I had a problem with parsing the data. I have one packet data as
> bellow:
> >
> > |---------------------|.......> 1 packet data
> > STX DATA ETX CRC
> > |---|
> > |..............> Data will be encrypt with AES CBC/CFB
> > with KEY bytes and Initial Vector bytes
> >
> > |--------|
> > |...............> CRC 16 bytes with MD5 HASH
> > Calcula te Data and ETX.
> >
> > STX : 0x02 (1 bytes)
> > DATA (encrypted): 0x33 0xF3 0x06 0x02 0xF8 0x98 0x03 ... (16 bytes)
> > ETX : 0x03 (1 bytes)
> > CRC : 0x45 0x88 0x7E 0x23 0x56 0x93 0x03 ... (16 bytes)
> >
> >
> > I parsing the DATA PER PACKET based STX (0x02) separator.
> >
> > I just separate the DATA with the CRC based ETX (0x03) separator.
> >
> > I calculate the CRC with MD5 HASH to ensure that data is not
> corrupted in the network.
> >
> > Like the example above, there is a character with ascii 0x02 in the
> encrypted data. Possible values range are 0x00-0xFF in
> data, which exists between the value of some are used as separators of
> data packets(STX and ETX).
> >
> > My question, how can I distinguish 0x02 is STX, NOT PART OF THE DATA
> and also 0x03 is ETX, NOT PART OF THE DATA ?
> >
> >
> > Thank You.
> > Siph4
>
Reply by ●January 6, 20122012-01-06
--- In r..., "siph4@..." wrote:
>
> @ALL, Thank you for the advice. From your input, I have an idea to keep sending data in ASCII character(1 byte), but the separator using the ASCII HEX character(2 bytes). Look below... I know this is not standard, but I've tried and succeeded and get better performance. Very fast with long data. ALL THANK YOU... :)
>
> '02' 0x2 0x3 0x4 0x5 0x6... '03' 0x3 0x4 0x3...
> ascii hex .... ascii char ...... ascii hex ..ascii char..
> (2 bytes) (n bytes) (2bytes) (16 bytes)
>
What if your data contains the 2-byte sequence 0x30 0x32 or 0x30 0x33 ? To the receiver, that will look like '02' or '03'.
>
> @ALL, Thank you for the advice. From your input, I have an idea to keep sending data in ASCII character(1 byte), but the separator using the ASCII HEX character(2 bytes). Look below... I know this is not standard, but I've tried and succeeded and get better performance. Very fast with long data. ALL THANK YOU... :)
>
> '02' 0x2 0x3 0x4 0x5 0x6... '03' 0x3 0x4 0x3...
> ascii hex .... ascii char ...... ascii hex ..ascii char..
> (2 bytes) (n bytes) (2bytes) (16 bytes)
>
What if your data contains the 2-byte sequence 0x30 0x32 or 0x30 0x33 ? To the receiver, that will look like '02' or '03'.
Reply by ●January 7, 20122012-01-07
@Don, Yes, data contains the 2-byte sequence 0x30 0x32 or 0x30 0x33. But
sometimes there are characters in encrypted the data :(
Actually if I want use the separator have ASCII a greater than 0xFF because the data coverage is only 0x00-0xFF.
In VB.NET I can use ChrW(Number), with Number > 0xFF
Eg:
STX = ChrW(&H100) = ChrW(0x100) = ChrW(256)
ETX = ChrW(&H102) = ChrW(0x101) = ChrW(257)
What is function in the dynamic c that can replace the task ChrW() to convert ASCII value more than 255 into the character?
There are two options, using a separator have ASCII value > 255 or use your suggestions, give a sign if there is a separator character in the encrypted data. Thanks.
Siph4
--- In r..., "donstarr" wrote:
>
> --- In r..., "siph4@" wrote:
> >
> > @ALL, Thank you for the advice. From your input, I have an idea to keep sending data in ASCII character(1 byte), but the separator using the ASCII HEX character(2 bytes). Look below... I know this is not standard, but I've tried and succeeded and get better performance. Very fast with long data. ALL THANK YOU... :)
> >
> >
> >
> > '02' 0x2 0x3 0x4 0x5 0x6... '03' 0x3 0x4 0x3...
> > ascii hex .... ascii char ...... ascii hex ..ascii char..
> > (2 bytes) (n bytes) (2bytes) (16 bytes)
> >
> What if your data contains the 2-byte sequence 0x30 0x32 or 0x30 0x33 ? To the receiver, that will look like '02' or '03'.
>
Actually if I want use the separator have ASCII a greater than 0xFF because the data coverage is only 0x00-0xFF.
In VB.NET I can use ChrW(Number), with Number > 0xFF
Eg:
STX = ChrW(&H100) = ChrW(0x100) = ChrW(256)
ETX = ChrW(&H102) = ChrW(0x101) = ChrW(257)
What is function in the dynamic c that can replace the task ChrW() to convert ASCII value more than 255 into the character?
There are two options, using a separator have ASCII value > 255 or use your suggestions, give a sign if there is a separator character in the encrypted data. Thanks.
Siph4
--- In r..., "donstarr" wrote:
>
> --- In r..., "siph4@" wrote:
> >
> > @ALL, Thank you for the advice. From your input, I have an idea to keep sending data in ASCII character(1 byte), but the separator using the ASCII HEX character(2 bytes). Look below... I know this is not standard, but I've tried and succeeded and get better performance. Very fast with long data. ALL THANK YOU... :)
> >
> >
> >
> > '02' 0x2 0x3 0x4 0x5 0x6... '03' 0x3 0x4 0x3...
> > ascii hex .... ascii char ...... ascii hex ..ascii char..
> > (2 bytes) (n bytes) (2bytes) (16 bytes)
> >
> What if your data contains the 2-byte sequence 0x30 0x32 or 0x30 0x33 ? To the receiver, that will look like '02' or '03'.
>
Reply by ●January 7, 20122012-01-07
After the stx why not send a 2-byte length in binary and not escape anything? If
this is a running stream without a known starting point then escapes won't
work either.
Bill
--- In r..., "siph4@..." wrote:
>
> @Don, Yes, data contains the 2-byte sequence 0x30 0x32 or 0x30 0x33. But sometimes there are characters in encrypted the data :(
>
> Actually if I want use the separator have ASCII a greater than 0xFF because the data coverage is only 0x00-0xFF.
>
> In VB.NET I can use ChrW(Number), with Number > 0xFF
>
> Eg:
> STX = ChrW(&H100) = ChrW(0x100) = ChrW(256)
> ETX = ChrW(&H102) = ChrW(0x101) = ChrW(257)
>
> What is function in the dynamic c that can replace the task ChrW() to convert ASCII value more than 255 into the character?
>
> There are two options, using a separator have ASCII value > 255 or use your suggestions, give a sign if there is a separator character in the encrypted data. Thanks.
> Siph4
>
> --- In r..., "donstarr" wrote:
> >
> > --- In r..., "siph4@" wrote:
> > >
> > > @ALL, Thank you for the advice. From your input, I have an idea to keep sending data in ASCII character(1 byte), but the separator using the ASCII HEX character(2 bytes). Look below... I know this is not standard, but I've tried and succeeded and get better performance. Very fast with long data. ALL THANK YOU... :)
> > >
> > >
> > >
> > > '02' 0x2 0x3 0x4 0x5 0x6... '03' 0x3 0x4 0x3...
> > > ascii hex .... ascii char ...... ascii hex ..ascii char..
> > > (2 bytes) (n bytes) (2bytes) (16 bytes)
> > >
> >
> >
> > What if your data contains the 2-byte sequence 0x30 0x32 or 0x30 0x33 ? To the receiver, that will look like '02' or '03'.
>
Bill
--- In r..., "siph4@..." wrote:
>
> @Don, Yes, data contains the 2-byte sequence 0x30 0x32 or 0x30 0x33. But sometimes there are characters in encrypted the data :(
>
> Actually if I want use the separator have ASCII a greater than 0xFF because the data coverage is only 0x00-0xFF.
>
> In VB.NET I can use ChrW(Number), with Number > 0xFF
>
> Eg:
> STX = ChrW(&H100) = ChrW(0x100) = ChrW(256)
> ETX = ChrW(&H102) = ChrW(0x101) = ChrW(257)
>
> What is function in the dynamic c that can replace the task ChrW() to convert ASCII value more than 255 into the character?
>
> There are two options, using a separator have ASCII value > 255 or use your suggestions, give a sign if there is a separator character in the encrypted data. Thanks.
> Siph4
>
> --- In r..., "donstarr" wrote:
> >
> > --- In r..., "siph4@" wrote:
> > >
> > > @ALL, Thank you for the advice. From your input, I have an idea to keep sending data in ASCII character(1 byte), but the separator using the ASCII HEX character(2 bytes). Look below... I know this is not standard, but I've tried and succeeded and get better performance. Very fast with long data. ALL THANK YOU... :)
> > >
> > >
> > >
> > > '02' 0x2 0x3 0x4 0x5 0x6... '03' 0x3 0x4 0x3...
> > > ascii hex .... ascii char ...... ascii hex ..ascii char..
> > > (2 bytes) (n bytes) (2bytes) (16 bytes)
> > >
> >
> >
> > What if your data contains the 2-byte sequence 0x30 0x32 or 0x30 0x33 ? To the receiver, that will look like '02' or '03'.
>
Reply by ●January 9, 20122012-01-09
There is a big advantage in sending command and data bytes as ascii hex
characters..you can debug with hyperterminal or some other terminal emulator
since the characters are all valid printable characters. There is another
way to separate command and data bytes which will allow you to send and
identify which bytes are command bytes, and which bytes are data bytes.
Simply use any byte 0 - 7F as a command, convert it to ASCII HEX. When
formatting your transmitted DATA bytes, always set the high bit after you
convert it to ASCII HEX. In this way command bytes will have their highest
bit cleared, and data bytes will have their highest bits set. This allows
you to "resynchronize" to the command bytes when you get them.
The sequence would go something like this. Into your transmit buffer, start
with your ASCII Hex Command Byte or Bytes (with the High Bit CLEARED)
followed by your converted data ASCII HEX data Bytes all with the High Bit
Set. Perhaps at the end of the transmit buffer you would append a check-sum
so that you can verify the validity of the packet you send.
In the receiver, you wait for a valid "command" character i.e. one with the
high bit cleared. All ASCII HEX bytes following the command byte are data
if the HIGH BIT is SET. Depending on the command, the data length could be
any size.
Later,
Rich
From: r... [mailto:r...] On
Behalf Of s...@ymail.com
Sent: Friday, January 06, 2012 1:48 PM
To: r...
Subject: [rabbit-semi] Re: PARSING DATA
--->>
Simple, send all DATA as a char and only allow (0x30 thru 0x39) & (0x41 thru
0x0x5a) & (0x61 thru 0x7a) values through as data, then all of your ASCII
control characters will be available for control.
This also makes Parsing easier.
@Jim, it is true we can do that if the data that we send are not in a
encrypted condition. If the data is encrypted, we can not guarantee that the
results encrypt there is no "ASCII control character".
---
My suggestion is to convert your encrypted data to ASCII hex characters. The
STX/ETX method only works if the data are ASCII characters.
@Seec, I've attempted to do this and succeed, the problem is the data length
to be 2x if convert to ASCII HEX characters. Data 16 bytes ASCII characters
will converted to 32 bytes ASCII HEX character. Performance will decrease if
send the data very long.
---
ASCII encoding is an automatic overhead penalty (up to 100% of data,
depending on implementation). That is, representing each byte of data as two
ASCII bytes (i.e. 0x02 becomes `0' `2'), automatically doubles the amount of
data transferred. This may not be acceptable to the O.P.
@don, you are right, That's the reason why I choose to transfer in the form
ASCII character because the data can be half shorter compared with ASCII HEX
character and that will affect in the poll data performace.
---
@ALL, Thank you for the advice. From your input, I have an idea to keep
sending data in ASCII character(1 byte), but the separator using the ASCII
HEX character(2 bytes). Look below... I know this is not standard, but I've
tried and succeeded and get better performance. Very fast with long data.
ALL THANK YOU... :)
'02' 0x2 0x3 0x4 0x5 0x6... '03' 0x3 0x4 0x3...
ascii hex .... ascii char ...... ascii hex ..ascii char..
(2 bytes) (n bytes) (2bytes) (16 bytes)
--- In r... ,
"Don Starr" wrote:
>
> Of course, TCP/IP by itself won't help. Since it's just a stream of
> bytes, the sender and receiver still need to coordinate
> "packetization". In the O.P.'s case, he's doing it with STX/ETX. But,
> since STX and ETX can appear in his data, he needs to somehow tell the
> receiver that the embedded STX/ETX are raw data and not packet
> delimiters.
>
> UDP might work, since it's a message-based protocol. "Packetization"
> is inherent.
>
> Neither TCP nor UDP is appropriate if the O.P. doesn't want to incur
> that code and/or transmission overhead, or if he can't incorporate
> them into his project(s) (e.g. if a stack isn't available for either
> the sender or receiver. we don't know that both are Rabbits).
>
> ASCII encoding is an automatic overhead penalty (up to 100% of data,
> depending on implementation). That is, representing each byte of data
> as two ASCII bytes (i.e. 0x02 becomes '0' '2'), automatically doubles
> the amount of data transferred. This may not be acceptable to the O.P.
>
> "Escaping" random data as I've suggested would be a 1.2% overhead
> penalty (on average, presuming that the 3 "special characters" occur
> at the same frequency as all other byte values). The maximum overhead
> penalty is 100%, if all bytes sent are one of the 3 special character
> values; this seems very unlikely, if the O.P.'s encryption is working.
> Finally, the implementation requirements for this method are trivial.
> On the sending side, it's one C statement. On the receiving side, it's
> a new flag plus a couple of tests. Much easier/faster than
> ASCII-encoding everything or bringing in a new protocol stack.
>
> -Don
>
> _____
>
> From: r...
[mailto:r... ]
> On Behalf Of Jim Ashby
> Sent: Friday, 06 January, 2012 10:39
> To: r...
> Subject: RE: [rabbit-semi] Re: PARSING DATA
> Don, this is understood and understood well.
>
> It is a standard method for data compression, used along with a
> special char to demark a sequence of repeating characters which is
> followed by the number of repeating chars and the repeating char.
>
> So if this method is chosen, you will need to add a demark char for
> each control code needed, and also the method of allowing the
> detection and processing of the control characters.
>
> About 20 years ago I wrote that type of code, but it involved wrapping
> the data block with a structured format, which allowed me to determine
> if the spacial char was for data or control.
>
> That is why they use the structure of TCP/IP wrappers around data
> blocks, there is no need for special chars when the entire structure
> of the data block is wrapped.
>
> With the wrapped format you also have the ability to include variable
> data block length, forward error correction, etc...
>
> Unless an engineer is ready to create a new custom wrapper, it would
> be better to just use a known and proven TCP/IP, UDP, or MODBUSS,
> style transfer protocol.
>
> All are available from ZWorld (Rabbit).
>
> My 25 cents worth.
>
> Jim Ashby
>
> -------Original Message-------
>
> From: Don Starr Date: 1/6/2012 10:08:58 AM
>
> To: r... Subject: RE: [rabbit-semi] Re: PARSING DATA
>
> If the encrypted data has an escaped pair, each of those bytes is
> escaped. For example, if the encrypted data (before sending) had 0x7E
> 0x02, then the actual sent data would be 0x7E 0x7E 0x7E 0x02
> (presuming the escape character was 0x7E). The receiver would discard
> the first and third 0x7E, resulting in 0x7E 0x02.
>
> unsigned char dstData[ MAX_DATA_LEN ];
>
> int dstLen;
>
> unsigned char esc;
>
> #define ESC 0x7E
>
> #define STX 0x02
>
> #define ETX 0x03
>
> unsigned char RxDataAvailable( void );
>
> unsigned char GetRxByte( void );
>
> esc = 0;
>
> dstLen = 0;
>
> while ( 1 )
>
> {
>
> if ( RxDataAvailable() )
>
> {
>
> unsigned char c = GetRxByte();
>
> if ( esc )
>
> {
>
> if ( dstLen < MAX_DATA_LEN )
>
> dstData[ dstLen++ ] = c;
>
> esc = 0;
>
> }
>
> else
>
> if ( c == ESC )
>
> esc = 1;
>
> else
>
> if ( c == STX )
>
> dstLen = 0;
>
> else
>
> if ( c == ETX )
>
> ; /* go get CRC and process packet */
>
> else
>
> if ( dstLen < MAX_DATA_LEN )
>
> dstData[ dstLen++ ] = c;
>
> }
>
> }
>
> _____
>
> From: r...
[mailto:r... ]
> On Behalf Of seecwriter
> Sent: Friday, 06 January, 2012 9:56
> To: r...
> Subject: [rabbit-semi] Re: PARSING DATA
>
> A problem with trying to escape special characters is that you can't
> guarantee that your encrypted data will not generate an escaped
> special character pair.
> My suggestion is to convert your encrypted data to ASCII hex
> characters. The STX/ETX method only works if the data are ASCII
> characters. Otherwise you will need to add a byte-count to the
> message.
>
> --- In r...
> , "siph4@"
> wrote:
> >
> > I had a problem with parsing the data. I have one packet data as
> bellow:
> >
> > |---------------------|.......> 1 packet data
> > STX DATA ETX CRC
> > |---|
> > |..............> Data will be encrypt with AES CBC/CFB
> > with KEY bytes and Initial Vector bytes
> >
> > |--------|
> > |...............> CRC 16 bytes with MD5 HASH
> > Calcula te Data and ETX.
> >
> > STX : 0x02 (1 bytes)
> > DATA (encrypted): 0x33 0xF3 0x06 0x02 0xF8 0x98 0x03 ... (16 bytes)
> > ETX : 0x03 (1 bytes)
> > CRC : 0x45 0x88 0x7E 0x23 0x56 0x93 0x03 ... (16 bytes)
> >
> >
> > I parsing the DATA PER PACKET based STX (0x02) separator.
> >
> > I just separate the DATA with the CRC based ETX (0x03) separator.
> >
> > I calculate the CRC with MD5 HASH to ensure that data is not
> corrupted in the network.
> >
> > Like the example above, there is a character with ascii 0x02 in the
> encrypted data. Possible values range are 0x00-0xFF in
> data, which exists between the value of some are used as separators of
> data packets(STX and ETX).
> >
> > My question, how can I distinguish 0x02 is STX, NOT PART OF THE DATA
> and also 0x03 is ETX, NOT PART OF THE DATA ?
> >
> >
> > Thank You.
> > Siph4
>
characters..you can debug with hyperterminal or some other terminal emulator
since the characters are all valid printable characters. There is another
way to separate command and data bytes which will allow you to send and
identify which bytes are command bytes, and which bytes are data bytes.
Simply use any byte 0 - 7F as a command, convert it to ASCII HEX. When
formatting your transmitted DATA bytes, always set the high bit after you
convert it to ASCII HEX. In this way command bytes will have their highest
bit cleared, and data bytes will have their highest bits set. This allows
you to "resynchronize" to the command bytes when you get them.
The sequence would go something like this. Into your transmit buffer, start
with your ASCII Hex Command Byte or Bytes (with the High Bit CLEARED)
followed by your converted data ASCII HEX data Bytes all with the High Bit
Set. Perhaps at the end of the transmit buffer you would append a check-sum
so that you can verify the validity of the packet you send.
In the receiver, you wait for a valid "command" character i.e. one with the
high bit cleared. All ASCII HEX bytes following the command byte are data
if the HIGH BIT is SET. Depending on the command, the data length could be
any size.
Later,
Rich
From: r... [mailto:r...] On
Behalf Of s...@ymail.com
Sent: Friday, January 06, 2012 1:48 PM
To: r...
Subject: [rabbit-semi] Re: PARSING DATA
--->>
Simple, send all DATA as a char and only allow (0x30 thru 0x39) & (0x41 thru
0x0x5a) & (0x61 thru 0x7a) values through as data, then all of your ASCII
control characters will be available for control.
This also makes Parsing easier.
@Jim, it is true we can do that if the data that we send are not in a
encrypted condition. If the data is encrypted, we can not guarantee that the
results encrypt there is no "ASCII control character".
---
My suggestion is to convert your encrypted data to ASCII hex characters. The
STX/ETX method only works if the data are ASCII characters.
@Seec, I've attempted to do this and succeed, the problem is the data length
to be 2x if convert to ASCII HEX characters. Data 16 bytes ASCII characters
will converted to 32 bytes ASCII HEX character. Performance will decrease if
send the data very long.
---
ASCII encoding is an automatic overhead penalty (up to 100% of data,
depending on implementation). That is, representing each byte of data as two
ASCII bytes (i.e. 0x02 becomes `0' `2'), automatically doubles the amount of
data transferred. This may not be acceptable to the O.P.
@don, you are right, That's the reason why I choose to transfer in the form
ASCII character because the data can be half shorter compared with ASCII HEX
character and that will affect in the poll data performace.
---
@ALL, Thank you for the advice. From your input, I have an idea to keep
sending data in ASCII character(1 byte), but the separator using the ASCII
HEX character(2 bytes). Look below... I know this is not standard, but I've
tried and succeeded and get better performance. Very fast with long data.
ALL THANK YOU... :)
'02' 0x2 0x3 0x4 0x5 0x6... '03' 0x3 0x4 0x3...
ascii hex .... ascii char ...... ascii hex ..ascii char..
(2 bytes) (n bytes) (2bytes) (16 bytes)
--- In r... ,
"Don Starr" wrote:
>
> Of course, TCP/IP by itself won't help. Since it's just a stream of
> bytes, the sender and receiver still need to coordinate
> "packetization". In the O.P.'s case, he's doing it with STX/ETX. But,
> since STX and ETX can appear in his data, he needs to somehow tell the
> receiver that the embedded STX/ETX are raw data and not packet
> delimiters.
>
> UDP might work, since it's a message-based protocol. "Packetization"
> is inherent.
>
> Neither TCP nor UDP is appropriate if the O.P. doesn't want to incur
> that code and/or transmission overhead, or if he can't incorporate
> them into his project(s) (e.g. if a stack isn't available for either
> the sender or receiver. we don't know that both are Rabbits).
>
> ASCII encoding is an automatic overhead penalty (up to 100% of data,
> depending on implementation). That is, representing each byte of data
> as two ASCII bytes (i.e. 0x02 becomes '0' '2'), automatically doubles
> the amount of data transferred. This may not be acceptable to the O.P.
>
> "Escaping" random data as I've suggested would be a 1.2% overhead
> penalty (on average, presuming that the 3 "special characters" occur
> at the same frequency as all other byte values). The maximum overhead
> penalty is 100%, if all bytes sent are one of the 3 special character
> values; this seems very unlikely, if the O.P.'s encryption is working.
> Finally, the implementation requirements for this method are trivial.
> On the sending side, it's one C statement. On the receiving side, it's
> a new flag plus a couple of tests. Much easier/faster than
> ASCII-encoding everything or bringing in a new protocol stack.
>
> -Don
>
> _____
>
> From: r...
[mailto:r... ]
> On Behalf Of Jim Ashby
> Sent: Friday, 06 January, 2012 10:39
> To: r...
> Subject: RE: [rabbit-semi] Re: PARSING DATA
> Don, this is understood and understood well.
>
> It is a standard method for data compression, used along with a
> special char to demark a sequence of repeating characters which is
> followed by the number of repeating chars and the repeating char.
>
> So if this method is chosen, you will need to add a demark char for
> each control code needed, and also the method of allowing the
> detection and processing of the control characters.
>
> About 20 years ago I wrote that type of code, but it involved wrapping
> the data block with a structured format, which allowed me to determine
> if the spacial char was for data or control.
>
> That is why they use the structure of TCP/IP wrappers around data
> blocks, there is no need for special chars when the entire structure
> of the data block is wrapped.
>
> With the wrapped format you also have the ability to include variable
> data block length, forward error correction, etc...
>
> Unless an engineer is ready to create a new custom wrapper, it would
> be better to just use a known and proven TCP/IP, UDP, or MODBUSS,
> style transfer protocol.
>
> All are available from ZWorld (Rabbit).
>
> My 25 cents worth.
>
> Jim Ashby
>
> -------Original Message-------
>
> From: Don Starr Date: 1/6/2012 10:08:58 AM
>
> To: r... Subject: RE: [rabbit-semi] Re: PARSING DATA
>
> If the encrypted data has an escaped pair, each of those bytes is
> escaped. For example, if the encrypted data (before sending) had 0x7E
> 0x02, then the actual sent data would be 0x7E 0x7E 0x7E 0x02
> (presuming the escape character was 0x7E). The receiver would discard
> the first and third 0x7E, resulting in 0x7E 0x02.
>
> unsigned char dstData[ MAX_DATA_LEN ];
>
> int dstLen;
>
> unsigned char esc;
>
> #define ESC 0x7E
>
> #define STX 0x02
>
> #define ETX 0x03
>
> unsigned char RxDataAvailable( void );
>
> unsigned char GetRxByte( void );
>
> esc = 0;
>
> dstLen = 0;
>
> while ( 1 )
>
> {
>
> if ( RxDataAvailable() )
>
> {
>
> unsigned char c = GetRxByte();
>
> if ( esc )
>
> {
>
> if ( dstLen < MAX_DATA_LEN )
>
> dstData[ dstLen++ ] = c;
>
> esc = 0;
>
> }
>
> else
>
> if ( c == ESC )
>
> esc = 1;
>
> else
>
> if ( c == STX )
>
> dstLen = 0;
>
> else
>
> if ( c == ETX )
>
> ; /* go get CRC and process packet */
>
> else
>
> if ( dstLen < MAX_DATA_LEN )
>
> dstData[ dstLen++ ] = c;
>
> }
>
> }
>
> _____
>
> From: r...
[mailto:r... ]
> On Behalf Of seecwriter
> Sent: Friday, 06 January, 2012 9:56
> To: r...
> Subject: [rabbit-semi] Re: PARSING DATA
>
> A problem with trying to escape special characters is that you can't
> guarantee that your encrypted data will not generate an escaped
> special character pair.
> My suggestion is to convert your encrypted data to ASCII hex
> characters. The STX/ETX method only works if the data are ASCII
> characters. Otherwise you will need to add a byte-count to the
> message.
>
> --- In r...
> , "siph4@"
> wrote:
> >
> > I had a problem with parsing the data. I have one packet data as
> bellow:
> >
> > |---------------------|.......> 1 packet data
> > STX DATA ETX CRC
> > |---|
> > |..............> Data will be encrypt with AES CBC/CFB
> > with KEY bytes and Initial Vector bytes
> >
> > |--------|
> > |...............> CRC 16 bytes with MD5 HASH
> > Calcula te Data and ETX.
> >
> > STX : 0x02 (1 bytes)
> > DATA (encrypted): 0x33 0xF3 0x06 0x02 0xF8 0x98 0x03 ... (16 bytes)
> > ETX : 0x03 (1 bytes)
> > CRC : 0x45 0x88 0x7E 0x23 0x56 0x93 0x03 ... (16 bytes)
> >
> >
> > I parsing the DATA PER PACKET based STX (0x02) separator.
> >
> > I just separate the DATA with the CRC based ETX (0x03) separator.
> >
> > I calculate the CRC with MD5 HASH to ensure that data is not
> corrupted in the network.
> >
> > Like the example above, there is a character with ascii 0x02 in the
> encrypted data. Possible values range are 0x00-0xFF in
> data, which exists between the value of some are used as separators of
> data packets(STX and ETX).
> >
> > My question, how can I distinguish 0x02 is STX, NOT PART OF THE DATA
> and also 0x03 is ETX, NOT PART OF THE DATA ?
> >
> >
> > Thank You.
> > Siph4
>
