Binary protocol design: TLV, LTV, or else?

Hi John,

On 1/11/2014 12:45 PM, John Devereux wrote:
> Grant Edwards<invalid@invalid.invalid>  writes:
>> On 2014-01-11, upsidedown@downunder.com<upsidedown@downunder.com>  wrote:
>>
>>> The cost of the vampire tap transceivers were so high that the first
>>> "Ethernet" network I designed and built was a network in a computer
>>> room between computers, using AUI cabling (with 15 bit connectors)
>>> through a DELNI "hub", later adding some long AUI cables for terminal
>>> servers at the other end of the office. Thus no coaxial cable used.
>>
>> The first place I worked where "Ethernet" was widely used, there was a
>> thick Ethernet backbone, but the vast majority of the wiring was AUI
>> cables and hubs.

"Orange hose" was a misnomer.  SHould have been called "orange ROD"!
I suspect the bend radius on that stuff was something on the order
of a quarter of a mile!  :<  If it wasn;t for AUI taps, you could never
have run the cable to all the nodes that wanted to connect to it!

> We had "cheapernet"(?) where, when a single one of the dodgy homemade
> BNCs went flakey, the whole network went down.

I *loved* 10Base2!  It made cable routing (for me) *so* much easier!
Just string *one* cable from A to B to C to...

By contrast, (physical) stars tend to have *bundles* of cables
re-traversing the same ground as they make the trek out to the
"box next to" the one you just wired.

I had more problems with bad terminators and connections *to* NIC's
failing (i.e., the T or F put too much stress on the connection
as it "hung" off the back of the NIC).

Of course, adding/removing nodes was a chore.  But, all the nodes
were mine (or, were within a single "product") so it was easy to
"administer" such changes.

> Fun times.

Hi Aleksander,

On 1/10/2014 12:04 PM, Aleksandar Kuktin wrote:
>> Can you,
>> instead, create a single packet format whose contents are indicated by a
>> "packet type" specified in the header?  Even if this means leaving space
>> for values/parameters that might not be required in every packet type?
>> For example:
>>      <header>  <field1>  <field2>  <field3>  <field4>
>> Where certain fields may not be used in certain packet types (their
>> contents then being "don't care").
>>
>> Alternatively, a packet type that implicitly *defines* the format of the
>> balance of the packet.  For example:
>>      type1:<header1>  <fieldA>  <fieldB>
>>      type2:<header2>  <fieldA>
>>      type3:<header3>  <fieldA>  <fieldB>  <fieldC>  <fieldD>
>> (where the format of each field may vary significantly between message
>> types)
>
> This is explicitly what I don't want. That way, I would need to send
> many, many packets to transmit my message across.

You could always incluide provisions that allow two or more of these
to be "bundled" together.  I.e., as the parser finishes, it returns
to its start and examines the balance of the "packet" to see if another
"message type" is present.

>>> But I would like to hear other (read: opposing) opinions. Particularly
>>> so since I am self-taught so there may be considerations obvious to
>>> graduated engineers that I am oblivious to.
>>>
>>> BTW, the periphery that is on the other end is autonomous and rather
>>> intelligent, but very resource constrained. Really, resource
>>> constrainment of the periphery is my main problem here.
>>
>> So, the less "thinking" (i.e., handling of variations) that the remote
>> device has to handle, the better.
>
> Hmmm... Not really. Avability of CPU cycles depends on other details of
> the device, but if need be I can make the device drown in its own CPU
> cycles. Memory, on the other hand, is constrained.

What is "a little" to some may be "a lot" to others.  :>
You'll have to decide what you can spare and where to apply it.
Much will depend on your coding style and the framework you
code within.

E.g., (memory of another reply fresh in my mind) when faced with a
missing (MAC,IP) binding to transmit a message (or reply), you could
send out an ARP request and explicitly wait for its reply; or,
consider your work "done" and wait for the ARP reply to return and
identify itself as such and handle that information *without*
remembering that you had explictly requested it!  ("Ah, I can send
this message because I have all the information I need to create
the reply datagram")

In one case, you have a separate process/thread/task (with all
that overhead) blocked awaiting the reply.  In the other case,
your single thread once again notices that it doesn't have the
(MAC,IP) binding that it needs -- but, examines a timer that
tells it NOT to issue an ARP request, yet ("Gee, I wonder why?")

[Sorry, this would hve been easier to explain with pseudo-code)

On Wed, 8 Jan 2014 21:30:09 +0000 (UTC), Aleksandar Kuktin
<akuktin@gmail.com> wrote:

>Hi all.
>
>I'm making a protocol for communication between a PC and a peripheral 
>device. The protocol is expected to, at first, run on raw Ethernet but I 
>am also supposed to not make any blunders that would make it impossible 
>to later use the exact same protocol on things like IP and friends.
>
>Since I saw these kinds of things in many Internet protocols (DNS, DHCP, 
>TCP options, off the top of my head - but note that these may have a 
>different order of fields), I have decided to make it an array of type-
>length-value triplets encapsulated in the packet frame (no header). The 
>commands would fill the "type" field, "length" would specify the length 
>of data ("value") following the length field, and "value" would contain 
>the data for the command.
>
>But I would like to hear other (read: opposing) opinions. Particularly so 
>since I am self-taught so there may be considerations obvious to 
>graduated engineers that I am oblivious to.
>
>BTW, the periphery that is on the other end is autonomous and rather 
>intelligent, but very resource constrained. Really, resource 
>constrainment of the periphery is my main problem here.
>
>
>Some interesting questions:
>Is ommiting a packet header a good idea? In the long run?
>
>If I put a packet header, what do I put in it? Since addressing and error 
>detection and "recovery" is supposed to be done by underlying protocols, 
>the only thing I can think of putting into the header is the total-length 
>field, and maybe, maybe, maybe a packet-id or transaction-id field. But I 
>really don't need any of these.
>
>My reasoning with packet-id and transaction-id (and protocol-version, 
>really) is that I don't need them now, so I can omit them, and if I ever 
>do need them, I can just add a command which implements them. In doing 
>this, am I setting myself up for a very nasty problem in the future?
>
>Is using flexible packets like this one (opposed the the contents of, 
>say, IP header which has strictly defined fields) a good idea, or am I 
>better off rigidifying my packets?
>
>Is there a special prefference or reason as to why some protocols do TLV 
>and others do LTV? (Note that I am not trying to ignite a holy war, I'm 
>just asking.)
>
>Is it good practice to require aligning the beggining of a TLV with a 
>boundary, say 16-bit word boundary?

Have you looked at ASN.1, is there something that makes it unusable
for your application ?

On 2014-01-11, Don Y <This.is@not.Me> wrote:
> Hi John,
>
> On 1/11/2014 12:45 PM, John Devereux wrote:
>
>> We had "cheapernet"(?) where, when a single one of the dodgy homemade
>> BNCs went flakey, the whole network went down.
>
> I *loved* 10Base2!  It made cable routing (for me) *so* much easier!
> Just string *one* cable from A to B to C to...
>

But then you had to deal with users deciding to unscrew a BNC connector...

Personally, I never had any problems with users doing that, but I read
somewhere one site fixed this problem by putting "Danger - high voltage!"
stickers on the BNC connectors. :-)

Simon.

PS: I only "love" 10Base2 in the sense that I "love" not having to deal
with it anymore... :-)

-- 
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
[Note: email address not currently working as the system is physically moving]
Microsoft: Bringing you 1980s technology to a 21st century world

Hi Simon,

On 1/11/2014 3:00 PM, Simon Clubley wrote:
> On 2014-01-11, Don Y<This.is@not.Me>  wrote:
>> On 1/11/2014 12:45 PM, John Devereux wrote:
>>
>>> We had "cheapernet"(?) where, when a single one of the dodgy homemade
>>> BNCs went flakey, the whole network went down.
>>
>> I *loved* 10Base2!  It made cable routing (for me) *so* much easier!
>> Just string *one* cable from A to B to C to...
>
> But then you had to deal with users deciding to unscrew a BNC connector...

Only if that is an upstream/downstream connector.

As I said, it was for *me* -- all the machines were mine (so no other
"users" that I had to coordinate my activities with).  At one time,
I had three SPARC LX's sitting side by side on a desktop with short
lengths of coax connecting them.  Had that been a physical star
technology, I would have had to run three separate cables to them
(or, one cable and a small switch -- along with its power supply).
Currently, I have 16 nodes in my office -- over a span of 24 feet.
Using a switch there is a real PITA (especially if I want to move a
node and now have to replace a wire all the way back to the switch
with a slightly longer one!).

[I recently upgraded the switch and moved it.  So, *every* wire
had to be replaced as none of their lengths were appropriate
any longer]

And, in the few products that I deployed with 10Base2, there's no
users *inside* those products, either!  (i.e., do you worry that some
"user" is going to perturb the CAN bus in your automobile and render it
inoperable as you are driving down the road?  Or, pull a PCI card
out of your PC while it is running?)

> Personally, I never had any problems with users doing that, but I read
> somewhere one site fixed this problem by putting "Danger - high voltage!"
> stickers on the BNC connectors. :-)
>
> Simon.
>
> PS: I only "love" 10Base2 in the sense that I "love" not having to deal
> with it anymore... :-)

On 2014-01-11, Don Y <this@isnotme.com> wrote:
>
> And, in the few products that I deployed with 10Base2, there's no
> users *inside* those products, either!  (i.e., do you worry that some
> "user" is going to perturb the CAN bus in your automobile and render it
> inoperable as you are driving down the road?  Or, pull a PCI card
> out of your PC while it is running?)
>

If either of those were accessible in the way a 10Base2 cable could be,
then the answer is probably yes. :-)

Simon.

-- 
Simon Clubley, clubley@remove_me.eisner.decus.org-Earth.UFP
[Note: email address not currently working as the system is physically moving]
Microsoft: Bringing you 1980s technology to a 21st century world

On Sat, 11 Jan 2014 19:45:36 +0000, John Devereux
<john@devereux.me.uk> wrote:

>Grant Edwards <invalid@invalid.invalid> writes:
>
>> On 2014-01-11, upsidedown@downunder.com <upsidedown@downunder.com> wrote:
>>
>>> The cost of the vampire tap transceivers were so high that the first
>>> "Ethernet" network I designed and built was a network in a computer
>>> room between computers, using AUI cabling (with 15 bit connectors)
>>> through a DELNI "hub", later adding some long AUI cables for terminal
>>> servers at the other end of the office. Thus no coaxial cable used.
>>
>> The first place I worked where "Ethernet" was widely used, there was a
>> thick Ethernet backbone, but the vast majority of the wiring was AUI
>> cables and hubs.
>
>We had "cheapernet"(?) where, when a single one of the dodgy homemade
>BNCs went flakey, the whole network went down.
>
>Fun times.


I've made the comment before that the horror of thinnet
(cheapernet/10base2) is what gave other technologies (like
Token-Ring*) room to exist at all, and nearly killed Ethernet.  For
all its faults, 10base5 was at least reliable.  10baseT pretty much
solved all the problems, and thus killed everything else.


*Token-Ring was in many ways a PITA to work with, and not really all
that reliable, but it made 10base2 look like a complete joke in terms
of reliability.

On Sat, 11 Jan 2014 22:31:33 +0200, upsidedown@downunder.com wrote:

>On Wed, 8 Jan 2014 21:30:09 +0000 (UTC), Aleksandar Kuktin
><akuktin@gmail.com> wrote:
>
>>Hi all.
>>
>>I'm making a protocol for communication between a PC and a peripheral 
>>device. The protocol is expected to, at first, run on raw Ethernet but I 
>>am also supposed to not make any blunders that would make it impossible 
>>to later use the exact same protocol on things like IP and friends.
>>
>>Since I saw these kinds of things in many Internet protocols (DNS, DHCP, 
>>TCP options, off the top of my head - but note that these may have a 
>>different order of fields), I have decided to make it an array of type-
>>length-value triplets encapsulated in the packet frame (no header). The 
>>commands would fill the "type" field, "length" would specify the length 
>>of data ("value") following the length field, and "value" would contain 
>>the data for the command.
>>
>>But I would like to hear other (read: opposing) opinions. Particularly so 
>>since I am self-taught so there may be considerations obvious to 
>>graduated engineers that I am oblivious to.
>>
>>BTW, the periphery that is on the other end is autonomous and rather 
>>intelligent, but very resource constrained. Really, resource 
>>constrainment of the periphery is my main problem here.
>>
>>
>>Some interesting questions:
>>Is ommiting a packet header a good idea? In the long run?
>>
>>If I put a packet header, what do I put in it? Since addressing and error 
>>detection and "recovery" is supposed to be done by underlying protocols, 
>>the only thing I can think of putting into the header is the total-length 
>>field, and maybe, maybe, maybe a packet-id or transaction-id field. But I 
>>really don't need any of these.
>>
>>My reasoning with packet-id and transaction-id (and protocol-version, 
>>really) is that I don't need them now, so I can omit them, and if I ever 
>>do need them, I can just add a command which implements them. In doing 
>>this, am I setting myself up for a very nasty problem in the future?
>>
>>Is using flexible packets like this one (opposed the the contents of, 
>>say, IP header which has strictly defined fields) a good idea, or am I 
>>better off rigidifying my packets?
>>
>>Is there a special prefference or reason as to why some protocols do TLV 
>>and others do LTV? (Note that I am not trying to ignite a holy war, I'm 
>>just asking.)
>>
>>Is it good practice to require aligning the beggining of a TLV with a 
>>boundary, say 16-bit word boundary?
>
>Have you looked at ASN.1, is there something that makes it unusable
>for your application ?


I take it you've never written a parser for ASN.1?  The OP seems to
have resource constrained systems in mind.  Generating the stuff is
not so bad, but decoding it is a huge PITA.

On 1/12/2014 3:55 AM, Simon Clubley wrote:
> On 2014-01-11, Don Y<this@isnotme.com>  wrote:
>>
>> And, in the few products that I deployed with 10Base2, there's no
>> users *inside* those products, either!  (i.e., do you worry that some
>> "user" is going to perturb the CAN bus in your automobile and render it
>> inoperable as you are driving down the road?  Or, pull a PCI card
>> out of your PC while it is running?)
>
> If either of those were accessible in the way a 10Base2 cable could be,
> then the answer is probably yes. :-)

Well, I suppose you could take out a 15 foot ladder and climb up
onto a deployed device *IN USE* and start tugging on cables.
Of course, if you did so, "disrupting the network" would be
the least of your concerns (I'd worry more about breaking your
neck from the fall or getting electrocuted as you climb over a
device that isn't intended to be "walked on"  :> )

On 1/11/2014 1:31 PM, upsidedown@downunder.com wrote:
> On Wed, 8 Jan 2014 21:30:09 +0000 (UTC), Aleksandar Kuktin
> <akuktin@gmail.com>  wrote:

>> Is it good practice to require aligning the beggining of a TLV with a
>> boundary, say 16-bit word boundary?
>
> Have you looked at ASN.1, is there something that makes it unusable
> for your application ?

Blech!  You *really* don't want to go there...

Are there existing protocols used by similar devices that the
OP can "borrow" (or, learn from)?  Even some existing protocol
ported to run *over* IP, etc.?