error detection rate with crc-16 CCITT

Hi Shane,

On 3/27/2011 4:39 AM, Shane williams wrote:
> On Mar 27, 11:53 pm, Michael Karas<mka...@carousel-design.com>  wrote:

[8<]

>> Are you getting some of the errors in your transmission path
>> due to distortion of the RS485 waveform due to non-equal propagation
>> delays through your logic on the "0"-->"1" transition versus the
>> one from "1"-->"0"? Common problem with certain optocouplers. ;-)

And some devices degrade with age.

> Thanks. I'm trying to figure out whether it's possible/ viable to
> dynamically determine the fastest baud rate we can use by checking the
> error rate.  The cable lengths and types of wire used when our systems
> are installed varies and I was hoping we could automatically work out
> what speed a particular connection can run at.  The spec for the
> MOC5007 Optocoupler seems a bit vague so I was trying to find a better
> one.

<frown>  You might, instead, want to think of this from the
"engineering" standpoint -- what are the likely/expected
*sources* of your errors?  I.e., how is the channel typically [1]
going to be corrupted.

First, think of the medium by itself.  With a given type of
cable (including "crap" that someone might fabricate on-the-spot),
how will your system likely behave (waveform distortions,
sampling skew in the receiver, component aging, etc.).

Then, think of the likely noise sources that might interfere
with your signal.  Is there some synchronous source nearby that
will periodically be bouncing your grounds or coupling directly
to your signals (i.e., will your cable be routed alongside
something noisey)?  [this assumes you have identified any
sources of "noise" that your system imposes on *itself*!  e.g.,
each time *you* command the VFD to engage the 10HP motor you
might notice glitches in your data...]

Then, think of what aperiodic/transient/"random" disturbances
are likely to be encountered in your environment.

In each case, think of the impact on the data stream AT ALL
THE DATA RATES YOU *MIGHT* BE LIKELY TO HAVE IN USE.  Are
you likely to see lots of dispersed single bit errors?  How
far apart (temporally) are they likely to be (far enough
that two different code words can cover them?)  Or, will
you encounter a burst of consecutive errors?  (if so, how
wide?)

Finally, regarding your hinted algorithm:  note that the
time constant you use in determining when/if to change rates
has to take into consideration these observations on the likely
environment.  E.g., if errors are likely to creep in "slowly"
(beginning with low probability, low error rate), then you
can "notice" the errors and start anticipating more (?) and
back off on your data rate -- hopefully, quick enough that the
error rate doesn't grow to exceed your *continued* ability
for your CRC to remain effective.

OTOH, if the error rate ever "grows" (instantaneously) faster
than your CRC is able to detect the increased error rate,
you run the risk of accepting bad data "as good".  And, sitting
"fat, happy and glorious" all the while you are doing so!
(i.e., sort of like a PLL locking on a harmonic outside the
intended capture range).

Can you, instead, figure out how to *ensure* a reliable channel?

--------------------
[1] and *atypically*!

On Mar 28, 6:51=A0am, Tim Wescott <t...@seemywebsite.com> wrote:
> On 03/27/2011 07:21 AM, Vladimir Vassilevsky wrote:
>
>
>
> > Shane williams wrote:
>
> >> Thanks. I'm trying to figure out whether it's possible/ viable to
> >> dynamically determine the fastest baud rate we can use by checking the
> >> error rate.
>
> > Yes. But:
>
> > 1) It is easier, faster and more reliable to evaluate the channel by
> > transmitting a known pseudo-random test pattern rather then the actual
> > data.
>
> I've done this -- and it is.
>
> > 2) If the baud rate is changed dynamically, how would the receivers kno=
w
> > the baud rate of the transmitters?
>
> There's ways. =A0Any good embedded programmer should be able to figure ou=
t
> half a dozen before they even put pen to napkin.
>
> > 3) Since the system is intended to be operable even at the lowest baud,
> > why not always use the lowest baud?
>
> If it's like ones that I've worked with, the data over the link is a
> combination of high-priority "gotta haves" like operational data, and
> lower-priority "dang this would be nice" things like diagnostics, faster
> status updates, and that sort of thing.
>
> So the advantages of going up in speed are obvious. =A0For that matter,
> there may be advantages to being able to tell the a maintenance guy what
> not-quite-fast-enough speed can be achieved, so he can make an informed
> choice about what faults to look for.
>

Didn't think about that.

You're exactly right about the need for speed.  Background data is
fine at the slower rate but when an operator is doing something on the
system we want the response to be faster than the slowest rate gives
us.

Switching rates seems fairly easy to me.  One end tells the other what
rate they're switching to, the other acknowledges, if no ack then
retry a couple of times.  If one end switches and the other doesn't,
after one second or so of no communication, they both switch back to
the slowest rate.

On Mar 28, 8:29=A0am, D Yuniskis <not.going.to...@seen.com> wrote:
> Hi Shane,
>
> On 3/27/2011 4:39 AM, Shane williams wrote:
>
> > On Mar 27, 11:53 pm, Michael Karas<mka...@carousel-design.com> =A0wrote=
:
>
> [8<]
>
> >> Are you getting some of the errors in your transmission path
> >> due to distortion of the RS485 waveform due to non-equal propagation
> >> delays through your logic on the "0"-->"1" transition versus the
> >> one from "1"-->"0"? Common problem with certain optocouplers. ;-)
>
> And some devices degrade with age.
>
> > Thanks. I'm trying to figure out whether it's possible/ viable to
> > dynamically determine the fastest baud rate we can use by checking the
> > error rate. =A0The cable lengths and types of wire used when our system=
s
> > are installed varies and I was hoping we could automatically work out
> > what speed a particular connection can run at. =A0The spec for the
> > MOC5007 Optocoupler seems a bit vague so I was trying to find a better
> > one.
>
> <frown> =A0You might, instead, want to think of this from the
> "engineering" standpoint -- what are the likely/expected
> *sources* of your errors? =A0I.e., how is the channel typically [1]
> going to be corrupted.
>
> First, think of the medium by itself. =A0With a given type of
> cable (including "crap" that someone might fabricate on-the-spot),
> how will your system likely behave (waveform distortions,
> sampling skew in the receiver, component aging, etc.).
>
> Then, think of the likely noise sources that might interfere
> with your signal. =A0Is there some synchronous source nearby that
> will periodically be bouncing your grounds or coupling directly
> to your signals (i.e., will your cable be routed alongside
> something noisey)? =A0[this assumes you have identified any
> sources of "noise" that your system imposes on *itself*! =A0e.g.,
> each time *you* command the VFD to engage the 10HP motor you
> might notice glitches in your data...]
>
> Then, think of what aperiodic/transient/"random" disturbances
> are likely to be encountered in your environment.
>
> In each case, think of the impact on the data stream AT ALL
> THE DATA RATES YOU *MIGHT* BE LIKELY TO HAVE IN USE. =A0Are
> you likely to see lots of dispersed single bit errors? =A0How
> far apart (temporally) are they likely to be (far enough
> that two different code words can cover them?) =A0Or, will
> you encounter a burst of consecutive errors? =A0(if so, how
> wide?)
>
> Finally, regarding your hinted algorithm: =A0note that the
> time constant you use in determining when/if to change rates
> has to take into consideration these observations on the likely
> environment. =A0E.g., if errors are likely to creep in "slowly"
> (beginning with low probability, low error rate), then you
> can "notice" the errors and start anticipating more (?) and
> back off on your data rate -- hopefully, quick enough that the
> error rate doesn't grow to exceed your *continued* ability
> for your CRC to remain effective.
>
> OTOH, if the error rate ever "grows" (instantaneously) faster
> than your CRC is able to detect the increased error rate,
> you run the risk of accepting bad data "as good". =A0And, sitting
> "fat, happy and glorious" all the while you are doing so!
> (i.e., sort of like a PLL locking on a harmonic outside the
> intended capture range).
>
> Can you, instead, figure out how to *ensure* a reliable channel?
>
> --------------------
> [1] and *atypically*!

Interesting points, thanks.  The environment can be just about
anything.  I suspect we'll back off the baud rate fairly quickly once
errors start occurring.  I'm also thinking we could raise the security
for some of the critical messages, like double transmissions perhaps.

On Mar 28, 3:46=A0am, Rafael Deliano <Rafael_DelianoENTFER...@t-
online.de> wrote:
> > I'm trying to figure out whether it's possible/ viable to
> > dynamically determine the fastest baud rate we can use by checking the
> > error rate.
>
> Packet length for a 16 bit CRCs should be limited to 4kbyte.
> =A0 =A0The CRC doesn t know at which baud rate the packets are coming.
> Your assumption ( which may well be true ) is that the error-pattern
> shifts from singlebit to bursts and more errors will go undetected.
> But the detected error rate would go way up too. By counting
> retransmissions now and later at the higher baud rate one could
> easily see if that has happened and switch to a 24 bit or 32 bit CRC.
>
> MfG JRD

Packet length is max 270 bytes / 2700 bits or so but critical messages
are more like about 50 bytes / 500 bits.

On 03/27/2011 11:36 AM, Jim Stewart wrote:
> Tim Wescott wrote:
>
>> It isn't that simple. CRC-16 will be able to detect _all_ 1, 2 and 3 bit
>> errors, and some 4-bit errors.
>
> I've often wondered about that statement. Suppose
> you get a 1 bit error in the message and an error
> in the crc remainder that results in a "good" message?
>
> Is there an implicit guarantee in the algorithm
> that it will take more than 3 bits to "fix" the
> remainder.
>
> My apologies if this is covered in the Webb article,
> running late today and don't have time to read it.

One bit error in the message and one in the CRC counts as two bit 
errors.  It's the number of bit errors in _both_ the CRC _and_ the 
message that you need to count.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Do you need to implement control loops in software?
"Applied Control Theory for Embedded Systems" was written for you.
See details at http://www.wescottdesign.com/actfes/actfes.html

Hi Shane,

On 3/27/2011 3:31 PM, Shane williams wrote:

> Interesting points, thanks.  The environment can be just about
> anything.  I suspect we'll back off the baud rate fairly quickly once
> errors start occurring.  I'm also thinking we could raise the security
> for some of the critical messages, like double transmissions perhaps.

Consider carefully what sort of "encoding" you use.  E.g.,
"double transmissions" might add lots of overhead for very
little gain in "reliability".

You can [1] also consider dynamically varying the data rate in
a TDM sort of scheme -- so, in this timeslot, you run at a slow,
reliable rate transfering critical messages; then, in this other
timeslot, you run "flat out" pushing data that would be "nice to
have" but not critical to proper operation.

Again, you really need to look hard at what you are likely to
encounter "in the field" before you can come to any expectations
regarding likely performance.  I've seen (and have been guilty,
myself!) some pretty mangled patches to deployed systems "just
to get by until the FedEx replacement parts delivery arrives".
If you *might* be running on the bleeding edge in some configuration,
the last thing you want is a guy in the field to *think* things
are OK when, in fact, they are not.

[e.g., you might want to add a switch that forces communications
to stay in the "degraded/secure" mode if you suspect you are not
catching all the communication errors in a particular installation...
because the tech made a cable out of "bell wire"]

----------------------------

[1] Depends on what is on the other end of the link, of course.
But, if you can autobaud dynamically, then that suggests you have
some control over both ends of the link!

In article <14a46afd-a5a4-4d6b-be24-de552c289027
@l14g2000pre.googlegroups.com>, shane.2471958@gmail.com says...
> Subject: Re: error detection rate with crc-16 CCITT
> Date: Sun, 27 Mar 2011 15:01:53 -0700 (PDT)
> From: Shane williams <shane.2471958@gmail.com>
> Newsgroups: comp.arch.embedded
> 
> On Mar 28, 6:51&#4294967295;am, Tim Wescott <t...@seemywebsite.com> wrote:
> > On 03/27/2011 07:21 AM, Vladimir Vassilevsky wrote:
> >
> >
> >
> > > Shane williams wrote:
> >
> > >> Thanks. I'm trying to figure out whether it's possible/ viable to
> > >> dynamically determine the fastest baud rate we can use by checking the
> > >> error rate.
> >
> > > Yes. But:
> >
> > > 1) It is easier, faster and more reliable to evaluate the channel by
> > > transmitting a known pseudo-random test pattern rather then the actual
> > > data.
> >
> > I've done this -- and it is.
> >
> > > 2) If the baud rate is changed dynamically, how would the receivers know
> > > the baud rate of the transmitters?
> >
> > There's ways. &#4294967295;Any good embedded programmer should be able to figure out
> > half a dozen before they even put pen to napkin.
> >
> > > 3) Since the system is intended to be operable even at the lowest baud,
> > > why not always use the lowest baud?
> >
> > If it's like ones that I've worked with, the data over the link is a
> > combination of high-priority "gotta haves" like operational data, and
> > lower-priority "dang this would be nice" things like diagnostics, faster
> > status updates, and that sort of thing.
> >
> > So the advantages of going up in speed are obvious. &#4294967295;For that matter,
> > there may be advantages to being able to tell the a maintenance guy what
> > not-quite-fast-enough speed can be achieved, so he can make an informed
> > choice about what faults to look for.
> >
> 
> Didn't think about that.
> 
> You're exactly right about the need for speed.  Background data is
> fine at the slower rate but when an operator is doing something on the
> system we want the response to be faster than the slowest rate gives
> us.

> Switching rates seems fairly easy to me.  One end tells the other what
> rate they're switching to, the other acknowledges, if no ack then
> retry a couple of times.  If one end switches and the other doesn't,
> after one second or so of no communication, they both switch back to
> the slowest rate.

Have you thought about about simple heartbeat loopback data packets?

If you get to the situation where too many error bits cannot be detected
how will you know everything is alright.

Every once in a while send a small varying pseudo-random data packet at 
highest speed to various nodes, which will just echo the packet back if 
decoded correctly. Once received check every bit is correct.

This way you are less likely to have false-positives about data being
correct when it is not.

You can change speeds and retry on failures. If you don't see an echo 
back, you have more problems to resolve.

Sending larger data packets at higher speeds helps to thoroughly check
data integrity and more chnce of more data switching frequencies that
may or may not be affected.

-- 
Paul Carpenter          | paul@pcserviceselectronics.co.uk
<http://www.pcserviceselectronics.co.uk/>    PC Services
<http://www.pcserviceselectronics.co.uk/fonts/> Timing Diagram Font
<http://www.gnuh8.org.uk/>  GNU H8 - compiler & Renesas H8/H8S/H8 Tiny
<http://www.badweb.org.uk/> For those web sites you hate

On Mar 28, 12:22=A0pm, D Yuniskis <not.going.to...@seen.com> wrote:
> Hi Shane,
>
> On 3/27/2011 3:31 PM, Shane williams wrote:
>
> > Interesting points, thanks. =A0The environment can be just about
> > anything. =A0I suspect we'll back off the baud rate fairly quickly once
> > errors start occurring. =A0I'm also thinking we could raise the securit=
y
> > for some of the critical messages, like double transmissions perhaps.
>
> Consider carefully what sort of "encoding" you use. =A0E.g.,
> "double transmissions" might add lots of overhead for very
> little gain in "reliability".
>
> You can [1] also consider dynamically varying the data rate in
> a TDM sort of scheme -- so, in this timeslot, you run at a slow,
> reliable rate transfering critical messages; then, in this other
> timeslot, you run "flat out" pushing data that would be "nice to
> have" but not critical to proper operation.
>
> Again, you really need to look hard at what you are likely to
> encounter "in the field" before you can come to any expectations
> regarding likely performance. =A0I've seen (and have been guilty,
> myself!) some pretty mangled patches to deployed systems "just
> to get by until the FedEx replacement parts delivery arrives".
> If you *might* be running on the bleeding edge in some configuration,
> the last thing you want is a guy in the field to *think* things
> are OK when, in fact, they are not.
>
> [e.g., you might want to add a switch that forces communications
> to stay in the "degraded/secure" mode if you suspect you are not
> catching all the communication errors in a particular installation...
> because the tech made a cable out of "bell wire"]
>
> ----------------------------
>
> [1] Depends on what is on the other end of the link, of course.
> But, if you can autobaud dynamically, then that suggests you have
> some control over both ends of the link!

Yep, it's the same device at both ends.

Regarding double transmissions, what do you mean by "encoding".  We
could complement all bits in the second transmission I guess.

TDM might not be viable and probably too much hassle I suspect.  The
baud rate behavior will be user configurable with probably a system
wide switch to allow the faster baud rate.

Thanks

On Mar 28, 12:22=A0pm, Paul <p...@pcserviceselectronics.co.uk> wrote:
> In article <14a46afd-a5a4-4d6b-be24-de552c289027
> @l14g2000pre.googlegroups.com>, shane.2471...@gmail.com says...
>
>
> Have you thought about about simple heartbeat loopback data packets?
>
> If you get to the situation where too many error bits cannot be detected
> how will you know everything is alright.
>
> Every once in a while send a small varying pseudo-random data packet at
> highest speed to various nodes, which will just echo the packet back if
> decoded correctly. Once received check every bit is correct.
>
> This way you are less likely to have false-positives about data being
> correct when it is not.
>
> You can change speeds and retry on failures. If you don't see an echo
> back, you have more problems to resolve.
>
> Sending larger data packets at higher speeds helps to thoroughly check
> data integrity and more chnce of more data switching frequencies that
> may or may not be affected.

Thanks for the idea about loop-back data packets.  That sounds useful.

The system is a ring of devices with each connection point to point
with one device at each end.

On Sun, 27 Mar 2011 17:36:45 -0700 (PDT), Shane williams
<shane.2471958@gmail.com> wrote:

<snippety snip>
>Regarding double transmissions, what do you mean by "encoding".  We
>could complement all bits in the second transmission I guess.

One approach that I've used in the past is to require an ack/nak for
each message sent. If the ack includes the CRC portion of the message
that's being acknowledged, then a simple match by the originator against
the CRC that it sent gives pretty good confidence that the receiver got
a correct message.

The returned CRC is, of course, part of the message body that the remote
unit sends which is in its turn used to build that message's CRC.

-- 
Rich Webb     Norfolk, VA