SMB/CIFS for uTasker/Coldfire?

David Brown <david.brown@hesbynett.no> writes:
> [...]
> Sometimes clients /do/ try to demand the impossible, which can be
> inconvenient.

Ha ha ha! Yeah, impossible is slightly inconvenient...

> TFTP is quite easy to implement, especially if you stick to the simplest
> parts (such as fixed 512 byte blocks).  There are servers and clients
> for Windows easily available, and *nix systems will have them as
> standard.  I use it for software updates - with the embedded system
> being the server side.

Thanks for the feedback. Sounds like TFTP is the right solution for this
task.
-- 
Randy Yates, DSP/Embedded Firmware Developer
Digital Signal Labs
http://www.digitalsignallabs.com

On 9/8/2016 8:59 AM, Randy Yates wrote:
> Don Y <blockedofcourse@foo.invalid> writes:
>
>> On 9/7/2016 7:10 PM, Randy Yates wrote:
>>> Thanks David and Dimiter. TFTP does sound like a great solution and is
>>> probably the way I'll go. There was a miscommunication in the
>>> requirements and I thought we HAD to implement a SMB/CIFS client.
>>> Apparently that is not the case.
>>
>> You've not indicated the nature of the "files" to be transfered,
>> nor their expected sizes.  (Just because your target has limited
>> resources doesn't mean it won't try to transfer objects exceeding
>> those!  E.g., imagine using a mapped file as a LOG -- which could
>> grow indefinitely!)
>>
>> You can either err on the side of being compatible with the most
>> recent standards and extensions to the original protocol (packet
>> size negotiation, window size negotiation, timeouts, etc.) *or*
>> try to aim for the "least imposing" set of requirements (512B blocks,
>> 32767-1 blocks) to accommodate potentially buggy server/client
>> implementations.  In effect, this limits transfers to 16MB.
>>
>> All the caveats of UDP-based protocols also apply.
>
> Hi Don,
>
> The files are 4500 bytes each. They consist of binary data. The overall
> transfer rate is low, on the order of 1 KB/s or less.

Assuming you are reasonably "network close" to the source, you'll have
no problems.  If you and the server/client on the other end can
support packet size negotiation, you can even negotiate a 4500 octet
packet and transfer the whole object in one block!

(safer bet is to only expect to be able to transfer 512B blocks)

> What "caveats of UDP-based protocols" are you referring to? Are you
> referring to things such as out-of-order packets and no error-checking?

Out of order packets, dropped packets, no OVERALL checksum, etc.
You also have to consider the nature of the fabric between
you and it:  will the traffic be routed?  If routed, will its
presence in a potentially wider stream cause it to get
dropped somewhere along the line?  E.g., most UDP-based protocols
spend a fair effort on retries, timeouts, verifying the packet
they received is the packet they expected (and not a stale
packet that got stuck in the fabric in the suburbs of Houston),
etc.

I'll assume there's no concern of an adversary spoofing you
or the other party involved (you can hijack a TFTP "connection"
if you want to; most clients/servers won't notice that they've
received two ACKs for a packet, etc.

You'll also have to decide how you identify/locate the other
party.  Will the user specify a *name* that you'll have to
resolve?  (DNS/YP/WINS)  Or, an IP address (and port number,
if you don't use the "well-known" TFTP port)?

Note that Windows hosts use a different discovery protocol
so you'll have to decide how much the network will do for you
(how will you get YOUR IP address?  etc.)

> I am not really knowledgable on the lower layers of TCP/IP.

UDP is pretty simple.  And, as a result, does very little
*for* you; anything that you might "need" will have to be
provided by you on top of that protocol.

[So, if you have a say over the format of the files, you might
want to include any features IN BAND to make the transfers more
robust.  Like a checksum IN the file at a known location so
you and the other party can agree that the file is intact]

On Thu, 08 Sep 2016 11:59:37 -0400, Randy Yates
<yates@digitalsignallabs.com> wrote:

>Don Y <blockedofcourse@foo.invalid> writes:
>>
>> You can either err on the side of being compatible with the most
>> recent standards and extensions to the original protocol (packet
>> size negotiation, window size negotiation, timeouts, etc.) *or*
>> try to aim for the "least imposing" set of requirements (512B blocks,
>> 32767-1 blocks) to accommodate potentially buggy server/client
>> implementations.  In effect, this limits transfers to 16MB.
>>
>> All the caveats of UDP-based protocols also apply.
>
>Hi Don,
>
>The files are 4500 bytes each. They consist of binary data. The overall
>transfer rate is low, on the order of 1 KB/s or less.
>
>What "caveats of UDP-based protocols" are you referring to? Are you
>referring to things such as out-of-order packets and no error-checking?
>
>I am not really knowledgable on the lower layers of TCP/IP.

Don is super, super cautious.  [Sometimes rightly so 8-)]  
His preference for really tiny packets is based on fear of
fragmentation in the network fabric.

For common networks, packets up to 1KB won't be fragmented.  You need
to worry about smaller if you are using modem/telephone (SLiP, PPP,
etc.).  I haven't followed the whole thread, so I may have missed what
network hardware you expect to be using.


UDP packets have a 16-bit checksum.  In the [ancient] past, the
checksum protected only the header information, but in modern
implementations it protects the entire packet.

http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm


Many desktop/server implementations limit UDP packets to match their
OS VMM page size - typically 4KB.  The UDP packet size technically
allows for packets up to 64KB, and some implementations allow large
packets, but you should limit to 4KB maximum for portability.


Hope this helps,
George

Hi George,

On 9/8/2016 10:23 AM, George Neuner wrote:
> Don is super, super cautious.  [Sometimes rightly so 8-)]
> His preference for really tiny packets is based on fear of
> fragmentation in the network fabric.

It's also a factor in the TFTP implementation.  Negotiating
packet sizes was not present in the original specification.
And, even if your client/server (depends on which end Randy is
NOT implementing) *can* negotiate a packet size > 512B,
there is no guarantee that it *will* negotiate the packet size
you desire

(RY's files are tiny so it's not really an issue.  OTOH, if you
are wanting to transfer BIG files -- or, use a file oriented
protocol to transfer a stream in the guise of a "big file" -- then
relying on the ability to negotiate larger packets, *hoping*
the "other side" hasn't used an "int" for the packet counter
AND hoping it can support "block count rollover" can be
important in your implementation decision)

> For common networks, packets up to 1KB won't be fragmented.  You need
> to worry about smaller if you are using modem/telephone (SLiP, PPP,
> etc.).  I haven't followed the whole thread, so I may have missed what
> network hardware you expect to be using.
>
> UDP packets have a 16-bit checksum.  In the [ancient] past, the
> checksum protected only the header information, but in modern
> implementations it protects the entire packet.
>
> http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
> http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm

But it doesn't protect the entire "file".  I.e., if you don't
explicitly verify the correct number of blocks have been
received but simply rely on size of the last block being
"less than full" and INFER that the file is intact "because
all the checksums were correct" you can be lulled into thinking
you have the "correct" file contents when you may only have some
of it (assuming no adversaries).

If you allow for the file on the server to potentially be
*changed* while the transfer is underway, all sorts of interesting
problems can manifest -- depending on the TFTPd implementation
(e.g., getting packets A-J from fileA and packets K-Z from the
*revised* file A)

> Many desktop/server implementations limit UDP packets to match their
> OS VMM page size - typically 4KB.  The UDP packet size technically
> allows for packets up to 64KB, and some implementations allow large
> packets, but you should limit to 4KB maximum for portability.

I see nothing wrong with sticking to the smaller packets.  It's
"safe".  And, the simplex/HDX nature of TFTP (REQ/ACK) isn't going
to "cost" that much for 9 packets -- not worth the effort to negotiate
a larger packet size or a larger window.

KISS.

When I PXE boot (which, in my case, relies on TFTP), I want
larger packets cuz I don't want the network to sit idle while
ACK's move up and down the network stacks -- "give me another
packet, and make it a big one, please!"

The point is to be aware of the sorts of things that *can* go
wrong so your code can be prepared for those "can't happens"
when they actually *do* happen!  (WTF?  I already ACK'd
packet #4 -- why am I seeing it AGAIN??  Should I *assume*
that this is a repeat copy of the previous one that I already
"verified" and ACK'd?  As I happen to have a copy of that
earlier packet, should I, perhaps, verify that this new
one does, in fact, agree with it?  It *should*, right??)

E.g., what do you do when some other host (misconfigured
or an adversary) starts issuing ACKs for those packets
cuz it thinks they are intended for *it*?  Or, claims
the same IP?  Or (gasp) the same MAC??!

"Cautious".  Yeah, I guess so!  :>  Much easier to tell the
user that I'm seeing things that "can't happen" and explain
them than have to have him rely on being able to sniff
network traffic to figure out why data isn't going where
it should...

On 08/09/16 19:23, George Neuner wrote:
> On Thu, 08 Sep 2016 11:59:37 -0400, Randy Yates
> <yates@digitalsignallabs.com> wrote:
> 
>> Don Y <blockedofcourse@foo.invalid> writes:
>>>
>>> You can either err on the side of being compatible with the most
>>> recent standards and extensions to the original protocol (packet
>>> size negotiation, window size negotiation, timeouts, etc.) *or*
>>> try to aim for the "least imposing" set of requirements (512B blocks,
>>> 32767-1 blocks) to accommodate potentially buggy server/client
>>> implementations.  In effect, this limits transfers to 16MB.
>>>
>>> All the caveats of UDP-based protocols also apply.
>>
>> Hi Don,
>>
>> The files are 4500 bytes each. They consist of binary data. The overall
>> transfer rate is low, on the order of 1 KB/s or less.
>>
>> What "caveats of UDP-based protocols" are you referring to? Are you
>> referring to things such as out-of-order packets and no error-checking?
>>
>> I am not really knowledgable on the lower layers of TCP/IP.
> 
> Don is super, super cautious.  [Sometimes rightly so 8-)]  
> His preference for really tiny packets is based on fear of
> fragmentation in the network fabric.
> 
> For common networks, packets up to 1KB won't be fragmented.  You need
> to worry about smaller if you are using modem/telephone (SLiP, PPP,
> etc.).  I haven't followed the whole thread, so I may have missed what
> network hardware you expect to be using.
> 
> 
> UDP packets have a 16-bit checksum.  In the [ancient] past, the
> checksum protected only the header information, but in modern
> implementations it protects the entire packet.
> 
> http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
> http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm
> 
> 
> Many desktop/server implementations limit UDP packets to match their
> OS VMM page size - typically 4KB.  The UDP packet size technically
> allows for packets up to 64KB, and some implementations allow large
> packets, but you should limit to 4KB maximum for portability.
> 

For TFTP, it is simpler if you stick to 512 byte data blocks (plus the 4
byte TFTP header).  If you are looking for optimal transfer speed of
large files, it is worth negotiating bigger block sizes (assuming both
sides support it), but for 4.5 KB it is easier just to send 9 packets.
You also don't want to deal with big UDP packets when your
microcontroller has little ram - why mess about trying to put together
separate Ethernet frames into a single UDP packet when you can have the
packet entirely within one frame, and use it directly without any copying?

With TFTP, you don't have to worry about some common UDP concerns such
as out-of-order packets (you don't send the next packet until the
previous one is acknowledged, and you retry on timeouts), and if you are
one a single Ethernet network you have no worries about UDP ports (which
can be a pain through firewalls).

If you can make the embedded side the server, then you also don't have
an issue with naming, DNS, addressing, etc. - or rather, it is the PC
client that has this concern rather than the embedded system.

Just try to avoid having more than one TFTP connection to or from the
same embedded system at the same time - it's easier if you don't have to
keep track.

On 9/9/2016 12:03 AM, David Brown wrote:
> On 08/09/16 19:23, George Neuner wrote:
>> On Thu, 08 Sep 2016 11:59:37 -0400, Randy Yates
>> <yates@digitalsignallabs.com> wrote:
>>
>>> Don Y <blockedofcourse@foo.invalid> writes:
>>>>
>>>> You can either err on the side of being compatible with the most
>>>> recent standards and extensions to the original protocol (packet
>>>> size negotiation, window size negotiation, timeouts, etc.) *or*
>>>> try to aim for the "least imposing" set of requirements (512B blocks,
>>>> 32767-1 blocks) to accommodate potentially buggy server/client
>>>> implementations.  In effect, this limits transfers to 16MB.
>>>>
>>>> All the caveats of UDP-based protocols also apply.
>>>
>>> Hi Don,
>>>
>>> The files are 4500 bytes each. They consist of binary data. The overall
>>> transfer rate is low, on the order of 1 KB/s or less.
>>>
>>> What "caveats of UDP-based protocols" are you referring to? Are you
>>> referring to things such as out-of-order packets and no error-checking?
>>>
>>> I am not really knowledgable on the lower layers of TCP/IP.
>>
>> Don is super, super cautious.  [Sometimes rightly so 8-)]
>> His preference for really tiny packets is based on fear of
>> fragmentation in the network fabric.
>>
>> For common networks, packets up to 1KB won't be fragmented.  You need
>> to worry about smaller if you are using modem/telephone (SLiP, PPP,
>> etc.).  I haven't followed the whole thread, so I may have missed what
>> network hardware you expect to be using.
>>
>>
>> UDP packets have a 16-bit checksum.  In the [ancient] past, the
>> checksum protected only the header information, but in modern
>> implementations it protects the entire packet.
>>
>> http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
>> http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm
>>
>>
>> Many desktop/server implementations limit UDP packets to match their
>> OS VMM page size - typically 4KB.  The UDP packet size technically
>> allows for packets up to 64KB, and some implementations allow large
>> packets, but you should limit to 4KB maximum for portability.
>
> For TFTP, it is simpler if you stick to 512 byte data blocks (plus the 4
> byte TFTP header).  If you are looking for optimal transfer speed of
> large files, it is worth negotiating bigger block sizes (assuming both
> sides support it), but for 4.5 KB it is easier just to send 9 packets.
> You also don't want to deal with big UDP packets when your
> microcontroller has little ram - why mess about trying to put together
> separate Ethernet frames into a single UDP packet when you can have the
> packet entirely within one frame, and use it directly without any copying?

A smarter move is to negotiate a larger window size.  So, the "next"
transfer can begin before your acknowledgement of the earlier one
has been received.  But, you have to 'remember more' to implement this
(on server and client).

But, for 4K files, its a moot point.

The problem (potentially) comes when you later decide to leverage
this existing code to address some OTHER need -- that doesn't
suffer from the same assumptions.

> With TFTP, you don't have to worry about some common UDP concerns such
> as out-of-order packets (you don't send the next packet until the
> previous one is acknowledged, and you retry on timeouts), and if you are
> one a single Ethernet network you have no worries about UDP ports (which
> can be a pain through firewalls).

You still have the problem of a packet getting delayed in the fabric.
If your client and server are on the same network segment, then
this isn't a real issue.  But, if a packet can get caught up
in a router, etc. then the potential for re-requesting a packet
that you (mistakenly) thought was "lost" can lead to problems.

I.e., you want to keep the timeout as short as possible (so you
don't wait "a long time" before rerequesting a packet).  The
shorter the timeout, the easier it is for a "delayed packet"
to appear to have been LOST.  So, you use a dynamic/adaptive
timeout

You rerequest (re-ACK) and end up with *two* packets -- each claiming
to be the original packet (the original that was delayed and the
repeated packet -- that, presumably, arrives AFTER the first
delayed packet).

It also means you MUST examine the block number in the packet
and not just ASSUME that the next packet that you SEE is the
next packet that you *need*.

[Folks implementing the protocol to the most recent RFC's
won't have a problem.  Folks dealing with legacy implementations
*can* have a problem!]

> If you can make the embedded side the server, then you also don't have
> an issue with naming, DNS, addressing, etc. - or rather, it is the PC
> client that has this concern rather than the embedded system.
>
> Just try to avoid having more than one TFTP connection to or from the
> same embedded system at the same time - it's easier if you don't have to
> keep track.

The problem with all network protocols is similar to the problem
that comes with "power cords" in homes/businesses.  People tend to
think "if I can plug into the outlet, everything will magically
work"!  (i.e., plug 4 hair dryers into a single branch circuit and
WONDER why the fuse blows!).

An RJ45 "here" and an RJ45 "there" don't magically imply that
two devices plugged into "here" and "there" can communicate
without regard for protocol concerns, etc.

On Fri, 09 Sep 2016 09:03:48 +0200, David Brown
<david.brown@hesbynett.no> wrote:

>On 08/09/16 19:23, George Neuner wrote:
>> On Thu, 08 Sep 2016 11:59:37 -0400, Randy Yates
>> <yates@digitalsignallabs.com> wrote:
>> 
>>> Don Y <blockedofcourse@foo.invalid> writes:
>>>>
>>>> You can either err on the side of being compatible with the most
>>>> recent standards and extensions to the original protocol (packet
>>>> size negotiation, window size negotiation, timeouts, etc.) *or*
>>>> try to aim for the "least imposing" set of requirements (512B blocks,
>>>> 32767-1 blocks) to accommodate potentially buggy server/client
>>>> implementations.  In effect, this limits transfers to 16MB.
>>>>
>>>> All the caveats of UDP-based protocols also apply.
>>>
>>> Hi Don,
>>>
>>> The files are 4500 bytes each. They consist of binary data. The overall
>>> transfer rate is low, on the order of 1 KB/s or less.
>>>
>>> What "caveats of UDP-based protocols" are you referring to? Are you
>>> referring to things such as out-of-order packets and no error-checking?
>>>
>>> I am not really knowledgable on the lower layers of TCP/IP.
>> 
>> Don is super, super cautious.  [Sometimes rightly so 8-)]  
>> His preference for really tiny packets is based on fear of
>> fragmentation in the network fabric.
>> 
>> For common networks, packets up to 1KB won't be fragmented.  You need
>> to worry about smaller if you are using modem/telephone (SLiP, PPP,
>> etc.).  I haven't followed the whole thread, so I may have missed what
>> network hardware you expect to be using.
>> 
>> 
>> UDP packets have a 16-bit checksum.  In the [ancient] past, the
>> checksum protected only the header information, but in modern
>> implementations it protects the entire packet.
>> 
>> http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
>> http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm
>> 
>> 
>> Many desktop/server implementations limit UDP packets to match their
>> OS VMM page size - typically 4KB.  The UDP packet size technically
>> allows for packets up to 64KB, and some implementations allow large
>> packets, but you should limit to 4KB maximum for portability.
>> 
>
>For TFTP, it is simpler if you stick to 512 byte data blocks (plus the 4
>byte TFTP header).  If you are looking for optimal transfer speed of
>large files, it is worth negotiating bigger block sizes (assuming both
>sides support it), but for 4.5 KB it is easier just to send 9 packets.
>You also don't want to deal with big UDP packets when your
>microcontroller has little ram - why mess about trying to put together
>separate Ethernet frames into a single UDP packet when you can have the
>packet entirely within one frame, and use it directly without any copying?
>
>With TFTP, you don't have to worry about some common UDP concerns such
>as out-of-order packets (you don't send the next packet until the
>previous one is acknowledged, and you retry on timeouts), and if you are
>one a single Ethernet network you have no worries about UDP ports (which
>can be a pain through firewalls).
>
>If you can make the embedded side the server, then you also don't have
>an issue with naming, DNS, addressing, etc. - or rather, it is the PC
>client that has this concern rather than the embedded system.
>
>Just try to avoid having more than one TFTP connection to or from the
>same embedded system at the same time - it's easier if you don't have to
>keep track.

If I understood correctly, the OP has a microcontroller capable of
Ethernet, so the maximum frame size is 1500 bytes, so there is not
much point of transferring blocks larger than that.

In practical applications, the net data block size would be 1024 bytes
(assuming power of 2 block sizes), but that 512  byte block size
wouldn't be too bad.  

For strictly half duplex (request/response) protocols, the Tx/Rx/Tx
latency is very critical. A direct point to point connection can have
a quite reasonable throughput.  

With full duplex protocols, the main transfer direction channel can be
kept fully occupied, if the reverse channel acknowledgements can be
overlapped.

On 09/09/16 10:05, Don Y wrote:
> On 9/9/2016 12:03 AM, David Brown wrote:
>> On 08/09/16 19:23, George Neuner wrote:
>>> On Thu, 08 Sep 2016 11:59:37 -0400, Randy Yates
>>> <yates@digitalsignallabs.com> wrote:
>>>
>>>> Don Y <blockedofcourse@foo.invalid> writes:
>>>>>
>>>>> You can either err on the side of being compatible with the most
>>>>> recent standards and extensions to the original protocol (packet
>>>>> size negotiation, window size negotiation, timeouts, etc.) *or*
>>>>> try to aim for the "least imposing" set of requirements (512B blocks,
>>>>> 32767-1 blocks) to accommodate potentially buggy server/client
>>>>> implementations.  In effect, this limits transfers to 16MB.
>>>>>
>>>>> All the caveats of UDP-based protocols also apply.
>>>>
>>>> Hi Don,
>>>>
>>>> The files are 4500 bytes each. They consist of binary data. The overall
>>>> transfer rate is low, on the order of 1 KB/s or less.
>>>>
>>>> What "caveats of UDP-based protocols" are you referring to? Are you
>>>> referring to things such as out-of-order packets and no error-checking?
>>>>
>>>> I am not really knowledgable on the lower layers of TCP/IP.
>>>
>>> Don is super, super cautious.  [Sometimes rightly so 8-)]
>>> His preference for really tiny packets is based on fear of
>>> fragmentation in the network fabric.
>>>
>>> For common networks, packets up to 1KB won't be fragmented.  You need
>>> to worry about smaller if you are using modem/telephone (SLiP, PPP,
>>> etc.).  I haven't followed the whole thread, so I may have missed what
>>> network hardware you expect to be using.
>>>
>>>
>>> UDP packets have a 16-bit checksum.  In the [ancient] past, the
>>> checksum protected only the header information, but in modern
>>> implementations it protects the entire packet.
>>>
>>> http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
>>> http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm
>>>
>>>
>>>
>>> Many desktop/server implementations limit UDP packets to match their
>>> OS VMM page size - typically 4KB.  The UDP packet size technically
>>> allows for packets up to 64KB, and some implementations allow large
>>> packets, but you should limit to 4KB maximum for portability.
>>
>> For TFTP, it is simpler if you stick to 512 byte data blocks (plus the 4
>> byte TFTP header).  If you are looking for optimal transfer speed of
>> large files, it is worth negotiating bigger block sizes (assuming both
>> sides support it), but for 4.5 KB it is easier just to send 9 packets.
>> You also don't want to deal with big UDP packets when your
>> microcontroller has little ram - why mess about trying to put together
>> separate Ethernet frames into a single UDP packet when you can have the
>> packet entirely within one frame, and use it directly without any
>> copying?
> 
> A smarter move is to negotiate a larger window size.  So, the "next"
> transfer can begin before your acknowledgement of the earlier one
> has been received.  But, you have to 'remember more' to implement this
> (on server and client).
> 

UDP does not have a "window size" - it has a packet reassembly buffer
size.  And with TFTP, your next transfer /cannot/ begin before the
acknowledge of the previous packet is sent - that is fundamental to the
TFTP protocol.  It is a simple query-response protocol.

> But, for 4K files, its a moot point.

Agreed.

> 
> The problem (potentially) comes when you later decide to leverage
> this existing code to address some OTHER need -- that doesn't
> suffer from the same assumptions.

That may be true.  TFTP is designed to be simple and low overhead, but
not high speed, flexible, or suitable for larger networks.

If the requirements change, you have to look at something else.

> 
>> With TFTP, you don't have to worry about some common UDP concerns such
>> as out-of-order packets (you don't send the next packet until the
>> previous one is acknowledged, and you retry on timeouts), and if you are
>> one a single Ethernet network you have no worries about UDP ports (which
>> can be a pain through firewalls).
> 
> You still have the problem of a packet getting delayed in the fabric.
> If your client and server are on the same network segment, then
> this isn't a real issue.  But, if a packet can get caught up
> in a router, etc. then the potential for re-requesting a packet
> that you (mistakenly) thought was "lost" can lead to problems.

Yes.  With TFTP you have a simple timeout - precisely because it is
mainly for simple systems on a single network segment that is usually
entirely reliable.  If that's not the case, then you may need something
more complex than TFTP.

> 
> I.e., you want to keep the timeout as short as possible (so you
> don't wait "a long time" before rerequesting a packet).  The
> shorter the timeout, the easier it is for a "delayed packet"
> to appear to have been LOST.  So, you use a dynamic/adaptive
> timeout
> 
> You rerequest (re-ACK) and end up with *two* packets -- each claiming
> to be the original packet (the original that was delayed and the
> repeated packet -- that, presumably, arrives AFTER the first
> delayed packet).
> 
> It also means you MUST examine the block number in the packet
> and not just ASSUME that the next packet that you SEE is the
> next packet that you *need*.
> 
> [Folks implementing the protocol to the most recent RFC's
> won't have a problem.  Folks dealing with legacy implementations
> *can* have a problem!]
> 
>> If you can make the embedded side the server, then you also don't have
>> an issue with naming, DNS, addressing, etc. - or rather, it is the PC
>> client that has this concern rather than the embedded system.
>>
>> Just try to avoid having more than one TFTP connection to or from the
>> same embedded system at the same time - it's easier if you don't have to
>> keep track.
> 
> The problem with all network protocols is similar to the problem
> that comes with "power cords" in homes/businesses.  People tend to
> think "if I can plug into the outlet, everything will magically
> work"!  (i.e., plug 4 hair dryers into a single branch circuit and
> WONDER why the fuse blows!).

Good analogy.

> 
> An RJ45 "here" and an RJ45 "there" don't magically imply that
> two devices plugged into "here" and "there" can communicate
> without regard for protocol concerns, etc.

On 9/9/2016 3:20 AM, David Brown wrote:
>>> For TFTP, it is simpler if you stick to 512 byte data blocks (plus the 4
>>> byte TFTP header).  If you are looking for optimal transfer speed of
>>> large files, it is worth negotiating bigger block sizes (assuming both
>>> sides support it), but for 4.5 KB it is easier just to send 9 packets.
>>> You also don't want to deal with big UDP packets when your
>>> microcontroller has little ram - why mess about trying to put together
>>> separate Ethernet frames into a single UDP packet when you can have the
>>> packet entirely within one frame, and use it directly without any
>>> copying?
>>
>> A smarter move is to negotiate a larger window size.  So, the "next"
>> transfer can begin before your acknowledgement of the earlier one
>> has been received.  But, you have to 'remember more' to implement this
>> (on server and client).
>
> UDP does not have a "window size" - it has a packet reassembly buffer
> size.  And with TFTP, your next transfer /cannot/ begin before the
> acknowledge of the previous packet is sent - that is fundamental to the
> TFTP protocol.  It is a simple query-response protocol.

No.  Later RFC's enhanced the protocol to allow for the sender and
receiver to negotiate a number of outstanding yet-to-be-ACKed
packets.

So, if you specify a window size of 8 (packets), the sender can
send up to 8 packets (the size of each can also separately be
negotiated) before requiring an ACK from the receiver.  An ACk
indicates the block number that is being acknowledged.  It
*implies* acknowledgement of all consecutive block numbers
up to and including the block in question.

By way of example, a sender can push packets 1, then 2, then 3, then
4, then 5, then 6, 7, 8 onto the wire BEFORE waiting for an
acknowledgement (in the case of a window size of 8).  If it receives
an "ACK(8)", then it knows packets 1-8 have been successfully received
and acknowledged.

It then starts sending the next set of (up to 8) packets.  If
(for example), the 11th packet is corrupted, the receiver will
send an "ACK(10)" -- whenever it comes to this conclusion!
This will be some time after packet 10 has been received but
can be any time up to and including the receipt of the 16th
packet (from the original ACK(8) + the window size of 8 more!).

*WHEN* the sender sees the ACK(10), it knows that all of the
packets AFTER that packet have been sent in vain.  And, it
is foolish to continue sending packets 12, 13, 14, 15, 16
(the implication is that 11 *has* been sent -- and found to be
in error).

Instead, the sender resends the packet AFTER the most recently ACK'd
(in this case, 11) and continues with up to 8 more (11 - 18).

Hopefully, it receives an ACK(18) after the 18th packet.  But,
if it receives an ACK(n) -- where 10 < n <= 18 -- then it
knows the receiver is essentially NAK'ing the packets that
follow 'n'.

This *effectively* increases the packet size -- while also
allowing a "partial effectively increased packet size"
to be acknowledged.

[If you negotiate a packet size of 4K -- 8 * 512 -- each ACK
indicates ALL of that has been received.  But, each "NAK" implies
NONE OF IT has been received!!  The window size allows you to
claim 3584 bytes have been received correctly and just the
last 512 bytes (packet!) need to be resent]

>> But, for 4K files, its a moot point.
>
> Agreed.
>
>> The problem (potentially) comes when you later decide to leverage
>> this existing code to address some OTHER need -- that doesn't
>> suffer from the same assumptions.
>
> That may be true.  TFTP is designed to be simple and low overhead, but
> not high speed, flexible, or suitable for larger networks.
>
> If the requirements change, you have to look at something else.

Or, understand the impact on the protocol.  As I said (snipped),
just because you plug device X and Y into *a* network doesn't mean
the protocol will behave the same as if plugged into the same
*switch*, etc.

Note that you should actively shape UDP traffic to "be a good
network citizen".   Just because you *can* issue requests/packets/ACKs
at a particular rate doesn't mean you *should* do so!

And, while it is probably NOT the case in RY's application, you
have to remember that the receiver and transmitter may have
other uses WHILE you are running this protocol.

On 09/09/16 13:05, Don Y wrote:
> On 9/9/2016 3:20 AM, David Brown wrote:
>>>> For TFTP, it is simpler if you stick to 512 byte data blocks (plus
>>>> the 4
>>>> byte TFTP header).  If you are looking for optimal transfer speed of
>>>> large files, it is worth negotiating bigger block sizes (assuming both
>>>> sides support it), but for 4.5 KB it is easier just to send 9 packets.
>>>> You also don't want to deal with big UDP packets when your
>>>> microcontroller has little ram - why mess about trying to put together
>>>> separate Ethernet frames into a single UDP packet when you can have the
>>>> packet entirely within one frame, and use it directly without any
>>>> copying?
>>>
>>> A smarter move is to negotiate a larger window size.  So, the "next"
>>> transfer can begin before your acknowledgement of the earlier one
>>> has been received.  But, you have to 'remember more' to implement this
>>> (on server and client).
>>
>> UDP does not have a "window size" - it has a packet reassembly buffer
>> size.  And with TFTP, your next transfer /cannot/ begin before the
>> acknowledge of the previous packet is sent - that is fundamental to the
>> TFTP protocol.  It is a simple query-response protocol.
> 
> No.  Later RFC's enhanced the protocol to allow for the sender and
> receiver to negotiate a number of outstanding yet-to-be-ACKed
> packets.
> 

Ah, okay.  I have only used TFTP in its original /trivial/ version.  I
know you can negotiate certain aspects, such as block size (and
apparently the number of outstanding blocks), but I did not bother with
such "features".  That would completely miss the point, as far as I can
see.  And when an 200 KB update takes a second or so, including flash
erasing and programming, there's little need to complicate things.

> So, if you specify a window size of 8 (packets), the sender can
> send up to 8 packets (the size of each can also separately be
> negotiated) before requiring an ACK from the receiver.  An ACk
> indicates the block number that is being acknowledged.  It
> *implies* acknowledgement of all consecutive block numbers
> up to and including the block in question.
> 
> By way of example, a sender can push packets 1, then 2, then 3, then
> 4, then 5, then 6, 7, 8 onto the wire BEFORE waiting for an
> acknowledgement (in the case of a window size of 8).  If it receives
> an "ACK(8)", then it knows packets 1-8 have been successfully received
> and acknowledged.

For high throughput, especially over higher latency routes, I can see
this making sense.  But if I needed something like that I would be
tempted to make my own simple protocol over TCP and let the TCP stack
handle all the details here (as well as making firewall traversal much
easier).
> 
>>> But, for 4K files, its a moot point.
>>
>> Agreed.
>>
>>> The problem (potentially) comes when you later decide to leverage
>>> this existing code to address some OTHER need -- that doesn't
>>> suffer from the same assumptions.
>>
>> That may be true.  TFTP is designed to be simple and low overhead, but
>> not high speed, flexible, or suitable for larger networks.
>>
>> If the requirements change, you have to look at something else.
> 
> Or, understand the impact on the protocol.  As I said (snipped),
> just because you plug device X and Y into *a* network doesn't mean
> the protocol will behave the same as if plugged into the same
> *switch*, etc.
> 
> Note that you should actively shape UDP traffic to "be a good
> network citizen".   Just because you *can* issue requests/packets/ACKs
> at a particular rate doesn't mean you *should* do so!
> 
> And, while it is probably NOT the case in RY's application, you
> have to remember that the receiver and transmitter may have
> other uses WHILE you are running this protocol.
>