Reply by mjbcswitzerland●October 3, 20162016-10-03
Hi
I didn't understand the exact transmission requirements (apart from that
SMB/CIFS is not required and the processor is a MCF52235).
Based on the opening uTasker reference, I should point out that there
should besically be no problems with UDP or TCP based transfer protocols
on the device since the uTasker TCP/IP stack is optimised for small memory
footprints and RAM and can achieve high speed transfers of large data.
It includes TFTP, but FTP is usually preferred, where it included as both
a client and a server and can achieve must higher data rates:
http://www.utasker.com/docs/uTasker/uTaskerFTP_client.pdfhttp://www.utasker.com/docs/uTasker/uTaskerFTP.PDF
(the server has however been upgraded since the referenced server document
to support multi-connection clients)
File transfer are also often done with HTTP where I once did a M52235
based development for ambulance services where there is a voice recorder
built into the medics' emergency box which helps post-analysis of
incidents. We had the sound recordings saved to high reliability SPI Flash
(several hundred MBYTES) and simply used an internal web server to allow
access via an Ethernet cable.
Also Ethernet/IP has been used on the M52233, together with FTP and web
server with total 32k RAM.
My other remark is that today it is very uncommon to start new work with
V2 Coldfires since they are legacy devices. Freescale/NXP has effectively
replaced then with their Kinetis parts (check out the K64) which have many
advantages:
- much more memory
- quite a lot faster
- smaller code size (about half the size compared to Coldfire)
- lower power
- cheaper
- more and better peripherals
- Ethernet includes checksum offloading and so UDP is around 4x faster on
a K70 that on an M52235:
http://www.utasker.com/docs/uTasker/uTaskerBenchmarks.PDF
Regards
Mark
http://www.uTasker.com
---------------------------------------
Posted through http://www.EmbeddedRelated.com
Reply by Don Y●September 9, 20162016-09-09
On 9/9/2016 5:01 AM, David Brown wrote:
> On 09/09/16 13:05, Don Y wrote:
>> On 9/9/2016 3:20 AM, David Brown wrote:
>>>>> For TFTP, it is simpler if you stick to 512 byte data blocks (plus
>>>>> the 4
>>>>> byte TFTP header). If you are looking for optimal transfer speed of
>>>>> large files, it is worth negotiating bigger block sizes (assuming both
>>>>> sides support it), but for 4.5 KB it is easier just to send 9 packets.
>>>>> You also don't want to deal with big UDP packets when your
>>>>> microcontroller has little ram - why mess about trying to put together
>>>>> separate Ethernet frames into a single UDP packet when you can have the
>>>>> packet entirely within one frame, and use it directly without any
>>>>> copying?
>>>>
>>>> A smarter move is to negotiate a larger window size. So, the "next"
>>>> transfer can begin before your acknowledgement of the earlier one
>>>> has been received. But, you have to 'remember more' to implement this
>>>> (on server and client).
>>>
>>> UDP does not have a "window size" - it has a packet reassembly buffer
>>> size. And with TFTP, your next transfer /cannot/ begin before the
>>> acknowledge of the previous packet is sent - that is fundamental to the
>>> TFTP protocol. It is a simple query-response protocol.
>>
>> No. Later RFC's enhanced the protocol to allow for the sender and
>> receiver to negotiate a number of outstanding yet-to-be-ACKed
>> packets.
>
> Ah, okay. I have only used TFTP in its original /trivial/ version. I
> know you can negotiate certain aspects, such as block size (and
> apparently the number of outstanding blocks), but I did not bother with
> such "features". That would completely miss the point, as far as I can
> see. And when an 200 KB update takes a second or so, including flash
> erasing and programming, there's little need to complicate things.
It's value is in moving many files and big files without all the
ping-pong activity of (traditional) TFTP. Negotiate a large
packet size AND a large window size and you will approach/exceed
the throughput of some of the SMB/NFS implementations with very
little overhead on either side.
[The bulk of the overhead is in parsing the messages. The rest is
pretty simple/obvious. The downside is you may get NOTHING that
you desire -- if the other party isn't willing to honor your
"enhanced" request(s)]
>> So, if you specify a window size of 8 (packets), the sender can
>> send up to 8 packets (the size of each can also separately be
>> negotiated) before requiring an ACK from the receiver. An ACk
>> indicates the block number that is being acknowledged. It
>> *implies* acknowledgement of all consecutive block numbers
>> up to and including the block in question.
>>
>> By way of example, a sender can push packets 1, then 2, then 3, then
>> 4, then 5, then 6, 7, 8 onto the wire BEFORE waiting for an
>> acknowledgement (in the case of a window size of 8). If it receives
>> an "ACK(8)", then it knows packets 1-8 have been successfully received
>> and acknowledged.
>
> For high throughput, especially over higher latency routes, I can see
> this making sense. But if I needed something like that I would be
> tempted to make my own simple protocol over TCP and let the TCP stack
> handle all the details here (as well as making firewall traversal much
> easier).
The advantage to the TFTP extensions is that they are a "standard"
instead of a "custom implementation".
I've used X terminals and diskless appliances rather extensively
over the years so have had to deal with various "low profile"
protocols for bootstrapping a system. For "connected devices",
I now much prefer it to having application-specific resident
firmware; there's never a need to push an upgrade as it is
automatically delivered as an inherent part of the next IPL!
(if the incoming image agrees with the image present in
your FLASH, no need to write "that block")
For implementations with lots of RAM where you can
afford to execute out of that RAM, then there's not even
a need to consider reflashing!
Reply by David Brown●September 9, 20162016-09-09
On 09/09/16 13:05, Don Y wrote:
> On 9/9/2016 3:20 AM, David Brown wrote:
>>>> For TFTP, it is simpler if you stick to 512 byte data blocks (plus
>>>> the 4
>>>> byte TFTP header). If you are looking for optimal transfer speed of
>>>> large files, it is worth negotiating bigger block sizes (assuming both
>>>> sides support it), but for 4.5 KB it is easier just to send 9 packets.
>>>> You also don't want to deal with big UDP packets when your
>>>> microcontroller has little ram - why mess about trying to put together
>>>> separate Ethernet frames into a single UDP packet when you can have the
>>>> packet entirely within one frame, and use it directly without any
>>>> copying?
>>>
>>> A smarter move is to negotiate a larger window size. So, the "next"
>>> transfer can begin before your acknowledgement of the earlier one
>>> has been received. But, you have to 'remember more' to implement this
>>> (on server and client).
>>
>> UDP does not have a "window size" - it has a packet reassembly buffer
>> size. And with TFTP, your next transfer /cannot/ begin before the
>> acknowledge of the previous packet is sent - that is fundamental to the
>> TFTP protocol. It is a simple query-response protocol.
>
> No. Later RFC's enhanced the protocol to allow for the sender and
> receiver to negotiate a number of outstanding yet-to-be-ACKed
> packets.
>
Ah, okay. I have only used TFTP in its original /trivial/ version. I
know you can negotiate certain aspects, such as block size (and
apparently the number of outstanding blocks), but I did not bother with
such "features". That would completely miss the point, as far as I can
see. And when an 200 KB update takes a second or so, including flash
erasing and programming, there's little need to complicate things.
> So, if you specify a window size of 8 (packets), the sender can
> send up to 8 packets (the size of each can also separately be
> negotiated) before requiring an ACK from the receiver. An ACk
> indicates the block number that is being acknowledged. It
> *implies* acknowledgement of all consecutive block numbers
> up to and including the block in question.
>
> By way of example, a sender can push packets 1, then 2, then 3, then
> 4, then 5, then 6, 7, 8 onto the wire BEFORE waiting for an
> acknowledgement (in the case of a window size of 8). If it receives
> an "ACK(8)", then it knows packets 1-8 have been successfully received
> and acknowledged.
For high throughput, especially over higher latency routes, I can see
this making sense. But if I needed something like that I would be
tempted to make my own simple protocol over TCP and let the TCP stack
handle all the details here (as well as making firewall traversal much
easier).
>
>>> But, for 4K files, its a moot point.
>>
>> Agreed.
>>
>>> The problem (potentially) comes when you later decide to leverage
>>> this existing code to address some OTHER need -- that doesn't
>>> suffer from the same assumptions.
>>
>> That may be true. TFTP is designed to be simple and low overhead, but
>> not high speed, flexible, or suitable for larger networks.
>>
>> If the requirements change, you have to look at something else.
>
> Or, understand the impact on the protocol. As I said (snipped),
> just because you plug device X and Y into *a* network doesn't mean
> the protocol will behave the same as if plugged into the same
> *switch*, etc.
>
> Note that you should actively shape UDP traffic to "be a good
> network citizen". Just because you *can* issue requests/packets/ACKs
> at a particular rate doesn't mean you *should* do so!
>
> And, while it is probably NOT the case in RY's application, you
> have to remember that the receiver and transmitter may have
> other uses WHILE you are running this protocol.
>
Reply by Don Y●September 9, 20162016-09-09
On 9/9/2016 3:20 AM, David Brown wrote:
>>> For TFTP, it is simpler if you stick to 512 byte data blocks (plus the 4
>>> byte TFTP header). If you are looking for optimal transfer speed of
>>> large files, it is worth negotiating bigger block sizes (assuming both
>>> sides support it), but for 4.5 KB it is easier just to send 9 packets.
>>> You also don't want to deal with big UDP packets when your
>>> microcontroller has little ram - why mess about trying to put together
>>> separate Ethernet frames into a single UDP packet when you can have the
>>> packet entirely within one frame, and use it directly without any
>>> copying?
>>
>> A smarter move is to negotiate a larger window size. So, the "next"
>> transfer can begin before your acknowledgement of the earlier one
>> has been received. But, you have to 'remember more' to implement this
>> (on server and client).
>
> UDP does not have a "window size" - it has a packet reassembly buffer
> size. And with TFTP, your next transfer /cannot/ begin before the
> acknowledge of the previous packet is sent - that is fundamental to the
> TFTP protocol. It is a simple query-response protocol.
No. Later RFC's enhanced the protocol to allow for the sender and
receiver to negotiate a number of outstanding yet-to-be-ACKed
packets.
So, if you specify a window size of 8 (packets), the sender can
send up to 8 packets (the size of each can also separately be
negotiated) before requiring an ACK from the receiver. An ACk
indicates the block number that is being acknowledged. It
*implies* acknowledgement of all consecutive block numbers
up to and including the block in question.
By way of example, a sender can push packets 1, then 2, then 3, then
4, then 5, then 6, 7, 8 onto the wire BEFORE waiting for an
acknowledgement (in the case of a window size of 8). If it receives
an "ACK(8)", then it knows packets 1-8 have been successfully received
and acknowledged.
It then starts sending the next set of (up to 8) packets. If
(for example), the 11th packet is corrupted, the receiver will
send an "ACK(10)" -- whenever it comes to this conclusion!
This will be some time after packet 10 has been received but
can be any time up to and including the receipt of the 16th
packet (from the original ACK(8) + the window size of 8 more!).
*WHEN* the sender sees the ACK(10), it knows that all of the
packets AFTER that packet have been sent in vain. And, it
is foolish to continue sending packets 12, 13, 14, 15, 16
(the implication is that 11 *has* been sent -- and found to be
in error).
Instead, the sender resends the packet AFTER the most recently ACK'd
(in this case, 11) and continues with up to 8 more (11 - 18).
Hopefully, it receives an ACK(18) after the 18th packet. But,
if it receives an ACK(n) -- where 10 < n <= 18 -- then it
knows the receiver is essentially NAK'ing the packets that
follow 'n'.
This *effectively* increases the packet size -- while also
allowing a "partial effectively increased packet size"
to be acknowledged.
[If you negotiate a packet size of 4K -- 8 * 512 -- each ACK
indicates ALL of that has been received. But, each "NAK" implies
NONE OF IT has been received!! The window size allows you to
claim 3584 bytes have been received correctly and just the
last 512 bytes (packet!) need to be resent]
>> But, for 4K files, its a moot point.
>
> Agreed.
>
>> The problem (potentially) comes when you later decide to leverage
>> this existing code to address some OTHER need -- that doesn't
>> suffer from the same assumptions.
>
> That may be true. TFTP is designed to be simple and low overhead, but
> not high speed, flexible, or suitable for larger networks.
>
> If the requirements change, you have to look at something else.
Or, understand the impact on the protocol. As I said (snipped),
just because you plug device X and Y into *a* network doesn't mean
the protocol will behave the same as if plugged into the same
*switch*, etc.
Note that you should actively shape UDP traffic to "be a good
network citizen". Just because you *can* issue requests/packets/ACKs
at a particular rate doesn't mean you *should* do so!
And, while it is probably NOT the case in RY's application, you
have to remember that the receiver and transmitter may have
other uses WHILE you are running this protocol.
Reply by David Brown●September 9, 20162016-09-09
On 09/09/16 10:05, Don Y wrote:
> On 9/9/2016 12:03 AM, David Brown wrote:
>> On 08/09/16 19:23, George Neuner wrote:
>>> On Thu, 08 Sep 2016 11:59:37 -0400, Randy Yates
>>> <yates@digitalsignallabs.com> wrote:
>>>
>>>> Don Y <blockedofcourse@foo.invalid> writes:
>>>>>
>>>>> You can either err on the side of being compatible with the most
>>>>> recent standards and extensions to the original protocol (packet
>>>>> size negotiation, window size negotiation, timeouts, etc.) *or*
>>>>> try to aim for the "least imposing" set of requirements (512B blocks,
>>>>> 32767-1 blocks) to accommodate potentially buggy server/client
>>>>> implementations. In effect, this limits transfers to 16MB.
>>>>>
>>>>> All the caveats of UDP-based protocols also apply.
>>>>
>>>> Hi Don,
>>>>
>>>> The files are 4500 bytes each. They consist of binary data. The overall
>>>> transfer rate is low, on the order of 1 KB/s or less.
>>>>
>>>> What "caveats of UDP-based protocols" are you referring to? Are you
>>>> referring to things such as out-of-order packets and no error-checking?
>>>>
>>>> I am not really knowledgable on the lower layers of TCP/IP.
>>>
>>> Don is super, super cautious. [Sometimes rightly so 8-)]
>>> His preference for really tiny packets is based on fear of
>>> fragmentation in the network fabric.
>>>
>>> For common networks, packets up to 1KB won't be fragmented. You need
>>> to worry about smaller if you are using modem/telephone (SLiP, PPP,
>>> etc.). I haven't followed the whole thread, so I may have missed what
>>> network hardware you expect to be using.
>>>
>>>
>>> UDP packets have a 16-bit checksum. In the [ancient] past, the
>>> checksum protected only the header information, but in modern
>>> implementations it protects the entire packet.
>>>
>>> http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
>>> http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm
>>>
>>>
>>>
>>> Many desktop/server implementations limit UDP packets to match their
>>> OS VMM page size - typically 4KB. The UDP packet size technically
>>> allows for packets up to 64KB, and some implementations allow large
>>> packets, but you should limit to 4KB maximum for portability.
>>
>> For TFTP, it is simpler if you stick to 512 byte data blocks (plus the 4
>> byte TFTP header). If you are looking for optimal transfer speed of
>> large files, it is worth negotiating bigger block sizes (assuming both
>> sides support it), but for 4.5 KB it is easier just to send 9 packets.
>> You also don't want to deal with big UDP packets when your
>> microcontroller has little ram - why mess about trying to put together
>> separate Ethernet frames into a single UDP packet when you can have the
>> packet entirely within one frame, and use it directly without any
>> copying?
>
> A smarter move is to negotiate a larger window size. So, the "next"
> transfer can begin before your acknowledgement of the earlier one
> has been received. But, you have to 'remember more' to implement this
> (on server and client).
>
UDP does not have a "window size" - it has a packet reassembly buffer
size. And with TFTP, your next transfer /cannot/ begin before the
acknowledge of the previous packet is sent - that is fundamental to the
TFTP protocol. It is a simple query-response protocol.
> But, for 4K files, its a moot point.
Agreed.
>
> The problem (potentially) comes when you later decide to leverage
> this existing code to address some OTHER need -- that doesn't
> suffer from the same assumptions.
That may be true. TFTP is designed to be simple and low overhead, but
not high speed, flexible, or suitable for larger networks.
If the requirements change, you have to look at something else.
>
>> With TFTP, you don't have to worry about some common UDP concerns such
>> as out-of-order packets (you don't send the next packet until the
>> previous one is acknowledged, and you retry on timeouts), and if you are
>> one a single Ethernet network you have no worries about UDP ports (which
>> can be a pain through firewalls).
>
> You still have the problem of a packet getting delayed in the fabric.
> If your client and server are on the same network segment, then
> this isn't a real issue. But, if a packet can get caught up
> in a router, etc. then the potential for re-requesting a packet
> that you (mistakenly) thought was "lost" can lead to problems.
Yes. With TFTP you have a simple timeout - precisely because it is
mainly for simple systems on a single network segment that is usually
entirely reliable. If that's not the case, then you may need something
more complex than TFTP.
>
> I.e., you want to keep the timeout as short as possible (so you
> don't wait "a long time" before rerequesting a packet). The
> shorter the timeout, the easier it is for a "delayed packet"
> to appear to have been LOST. So, you use a dynamic/adaptive
> timeout
>
> You rerequest (re-ACK) and end up with *two* packets -- each claiming
> to be the original packet (the original that was delayed and the
> repeated packet -- that, presumably, arrives AFTER the first
> delayed packet).
>
> It also means you MUST examine the block number in the packet
> and not just ASSUME that the next packet that you SEE is the
> next packet that you *need*.
>
> [Folks implementing the protocol to the most recent RFC's
> won't have a problem. Folks dealing with legacy implementations
> *can* have a problem!]
>
>> If you can make the embedded side the server, then you also don't have
>> an issue with naming, DNS, addressing, etc. - or rather, it is the PC
>> client that has this concern rather than the embedded system.
>>
>> Just try to avoid having more than one TFTP connection to or from the
>> same embedded system at the same time - it's easier if you don't have to
>> keep track.
>
> The problem with all network protocols is similar to the problem
> that comes with "power cords" in homes/businesses. People tend to
> think "if I can plug into the outlet, everything will magically
> work"! (i.e., plug 4 hair dryers into a single branch circuit and
> WONDER why the fuse blows!).
Good analogy.
>
> An RJ45 "here" and an RJ45 "there" don't magically imply that
> two devices plugged into "here" and "there" can communicate
> without regard for protocol concerns, etc.
Reply by ●September 9, 20162016-09-09
On Fri, 09 Sep 2016 09:03:48 +0200, David Brown
<david.brown@hesbynett.no> wrote:
>On 08/09/16 19:23, George Neuner wrote:
>> On Thu, 08 Sep 2016 11:59:37 -0400, Randy Yates
>> <yates@digitalsignallabs.com> wrote:
>>
>>> Don Y <blockedofcourse@foo.invalid> writes:
>>>>
>>>> You can either err on the side of being compatible with the most
>>>> recent standards and extensions to the original protocol (packet
>>>> size negotiation, window size negotiation, timeouts, etc.) *or*
>>>> try to aim for the "least imposing" set of requirements (512B blocks,
>>>> 32767-1 blocks) to accommodate potentially buggy server/client
>>>> implementations. In effect, this limits transfers to 16MB.
>>>>
>>>> All the caveats of UDP-based protocols also apply.
>>>
>>> Hi Don,
>>>
>>> The files are 4500 bytes each. They consist of binary data. The overall
>>> transfer rate is low, on the order of 1 KB/s or less.
>>>
>>> What "caveats of UDP-based protocols" are you referring to? Are you
>>> referring to things such as out-of-order packets and no error-checking?
>>>
>>> I am not really knowledgable on the lower layers of TCP/IP.
>>
>> Don is super, super cautious. [Sometimes rightly so 8-)]
>> His preference for really tiny packets is based on fear of
>> fragmentation in the network fabric.
>>
>> For common networks, packets up to 1KB won't be fragmented. You need
>> to worry about smaller if you are using modem/telephone (SLiP, PPP,
>> etc.). I haven't followed the whole thread, so I may have missed what
>> network hardware you expect to be using.
>>
>>
>> UDP packets have a 16-bit checksum. In the [ancient] past, the
>> checksum protected only the header information, but in modern
>> implementations it protects the entire packet.
>>
>> http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
>> http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm
>>
>>
>> Many desktop/server implementations limit UDP packets to match their
>> OS VMM page size - typically 4KB. The UDP packet size technically
>> allows for packets up to 64KB, and some implementations allow large
>> packets, but you should limit to 4KB maximum for portability.
>>
>
>For TFTP, it is simpler if you stick to 512 byte data blocks (plus the 4
>byte TFTP header). If you are looking for optimal transfer speed of
>large files, it is worth negotiating bigger block sizes (assuming both
>sides support it), but for 4.5 KB it is easier just to send 9 packets.
>You also don't want to deal with big UDP packets when your
>microcontroller has little ram - why mess about trying to put together
>separate Ethernet frames into a single UDP packet when you can have the
>packet entirely within one frame, and use it directly without any copying?
>
>With TFTP, you don't have to worry about some common UDP concerns such
>as out-of-order packets (you don't send the next packet until the
>previous one is acknowledged, and you retry on timeouts), and if you are
>one a single Ethernet network you have no worries about UDP ports (which
>can be a pain through firewalls).
>
>If you can make the embedded side the server, then you also don't have
>an issue with naming, DNS, addressing, etc. - or rather, it is the PC
>client that has this concern rather than the embedded system.
>
>Just try to avoid having more than one TFTP connection to or from the
>same embedded system at the same time - it's easier if you don't have to
>keep track.
If I understood correctly, the OP has a microcontroller capable of
Ethernet, so the maximum frame size is 1500 bytes, so there is not
much point of transferring blocks larger than that.
In practical applications, the net data block size would be 1024 bytes
(assuming power of 2 block sizes), but that 512 byte block size
wouldn't be too bad.
For strictly half duplex (request/response) protocols, the Tx/Rx/Tx
latency is very critical. A direct point to point connection can have
a quite reasonable throughput.
With full duplex protocols, the main transfer direction channel can be
kept fully occupied, if the reverse channel acknowledgements can be
overlapped.
Reply by Don Y●September 9, 20162016-09-09
On 9/9/2016 12:03 AM, David Brown wrote:
> On 08/09/16 19:23, George Neuner wrote:
>> On Thu, 08 Sep 2016 11:59:37 -0400, Randy Yates
>> <yates@digitalsignallabs.com> wrote:
>>
>>> Don Y <blockedofcourse@foo.invalid> writes:
>>>>
>>>> You can either err on the side of being compatible with the most
>>>> recent standards and extensions to the original protocol (packet
>>>> size negotiation, window size negotiation, timeouts, etc.) *or*
>>>> try to aim for the "least imposing" set of requirements (512B blocks,
>>>> 32767-1 blocks) to accommodate potentially buggy server/client
>>>> implementations. In effect, this limits transfers to 16MB.
>>>>
>>>> All the caveats of UDP-based protocols also apply.
>>>
>>> Hi Don,
>>>
>>> The files are 4500 bytes each. They consist of binary data. The overall
>>> transfer rate is low, on the order of 1 KB/s or less.
>>>
>>> What "caveats of UDP-based protocols" are you referring to? Are you
>>> referring to things such as out-of-order packets and no error-checking?
>>>
>>> I am not really knowledgable on the lower layers of TCP/IP.
>>
>> Don is super, super cautious. [Sometimes rightly so 8-)]
>> His preference for really tiny packets is based on fear of
>> fragmentation in the network fabric.
>>
>> For common networks, packets up to 1KB won't be fragmented. You need
>> to worry about smaller if you are using modem/telephone (SLiP, PPP,
>> etc.). I haven't followed the whole thread, so I may have missed what
>> network hardware you expect to be using.
>>
>>
>> UDP packets have a 16-bit checksum. In the [ancient] past, the
>> checksum protected only the header information, but in modern
>> implementations it protects the entire packet.
>>
>> http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
>> http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm
>>
>>
>> Many desktop/server implementations limit UDP packets to match their
>> OS VMM page size - typically 4KB. The UDP packet size technically
>> allows for packets up to 64KB, and some implementations allow large
>> packets, but you should limit to 4KB maximum for portability.
>
> For TFTP, it is simpler if you stick to 512 byte data blocks (plus the 4
> byte TFTP header). If you are looking for optimal transfer speed of
> large files, it is worth negotiating bigger block sizes (assuming both
> sides support it), but for 4.5 KB it is easier just to send 9 packets.
> You also don't want to deal with big UDP packets when your
> microcontroller has little ram - why mess about trying to put together
> separate Ethernet frames into a single UDP packet when you can have the
> packet entirely within one frame, and use it directly without any copying?
A smarter move is to negotiate a larger window size. So, the "next"
transfer can begin before your acknowledgement of the earlier one
has been received. But, you have to 'remember more' to implement this
(on server and client).
But, for 4K files, its a moot point.
The problem (potentially) comes when you later decide to leverage
this existing code to address some OTHER need -- that doesn't
suffer from the same assumptions.
> With TFTP, you don't have to worry about some common UDP concerns such
> as out-of-order packets (you don't send the next packet until the
> previous one is acknowledged, and you retry on timeouts), and if you are
> one a single Ethernet network you have no worries about UDP ports (which
> can be a pain through firewalls).
You still have the problem of a packet getting delayed in the fabric.
If your client and server are on the same network segment, then
this isn't a real issue. But, if a packet can get caught up
in a router, etc. then the potential for re-requesting a packet
that you (mistakenly) thought was "lost" can lead to problems.
I.e., you want to keep the timeout as short as possible (so you
don't wait "a long time" before rerequesting a packet). The
shorter the timeout, the easier it is for a "delayed packet"
to appear to have been LOST. So, you use a dynamic/adaptive
timeout
You rerequest (re-ACK) and end up with *two* packets -- each claiming
to be the original packet (the original that was delayed and the
repeated packet -- that, presumably, arrives AFTER the first
delayed packet).
It also means you MUST examine the block number in the packet
and not just ASSUME that the next packet that you SEE is the
next packet that you *need*.
[Folks implementing the protocol to the most recent RFC's
won't have a problem. Folks dealing with legacy implementations
*can* have a problem!]
> If you can make the embedded side the server, then you also don't have
> an issue with naming, DNS, addressing, etc. - or rather, it is the PC
> client that has this concern rather than the embedded system.
>
> Just try to avoid having more than one TFTP connection to or from the
> same embedded system at the same time - it's easier if you don't have to
> keep track.
The problem with all network protocols is similar to the problem
that comes with "power cords" in homes/businesses. People tend to
think "if I can plug into the outlet, everything will magically
work"! (i.e., plug 4 hair dryers into a single branch circuit and
WONDER why the fuse blows!).
An RJ45 "here" and an RJ45 "there" don't magically imply that
two devices plugged into "here" and "there" can communicate
without regard for protocol concerns, etc.
Reply by David Brown●September 9, 20162016-09-09
On 08/09/16 19:23, George Neuner wrote:
> On Thu, 08 Sep 2016 11:59:37 -0400, Randy Yates
> <yates@digitalsignallabs.com> wrote:
>
>> Don Y <blockedofcourse@foo.invalid> writes:
>>>
>>> You can either err on the side of being compatible with the most
>>> recent standards and extensions to the original protocol (packet
>>> size negotiation, window size negotiation, timeouts, etc.) *or*
>>> try to aim for the "least imposing" set of requirements (512B blocks,
>>> 32767-1 blocks) to accommodate potentially buggy server/client
>>> implementations. In effect, this limits transfers to 16MB.
>>>
>>> All the caveats of UDP-based protocols also apply.
>>
>> Hi Don,
>>
>> The files are 4500 bytes each. They consist of binary data. The overall
>> transfer rate is low, on the order of 1 KB/s or less.
>>
>> What "caveats of UDP-based protocols" are you referring to? Are you
>> referring to things such as out-of-order packets and no error-checking?
>>
>> I am not really knowledgable on the lower layers of TCP/IP.
>
> Don is super, super cautious. [Sometimes rightly so 8-)]
> His preference for really tiny packets is based on fear of
> fragmentation in the network fabric.
>
> For common networks, packets up to 1KB won't be fragmented. You need
> to worry about smaller if you are using modem/telephone (SLiP, PPP,
> etc.). I haven't followed the whole thread, so I may have missed what
> network hardware you expect to be using.
>
>
> UDP packets have a 16-bit checksum. In the [ancient] past, the
> checksum protected only the header information, but in modern
> implementations it protects the entire packet.
>
> http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
> http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm
>
>
> Many desktop/server implementations limit UDP packets to match their
> OS VMM page size - typically 4KB. The UDP packet size technically
> allows for packets up to 64KB, and some implementations allow large
> packets, but you should limit to 4KB maximum for portability.
>
For TFTP, it is simpler if you stick to 512 byte data blocks (plus the 4
byte TFTP header). If you are looking for optimal transfer speed of
large files, it is worth negotiating bigger block sizes (assuming both
sides support it), but for 4.5 KB it is easier just to send 9 packets.
You also don't want to deal with big UDP packets when your
microcontroller has little ram - why mess about trying to put together
separate Ethernet frames into a single UDP packet when you can have the
packet entirely within one frame, and use it directly without any copying?
With TFTP, you don't have to worry about some common UDP concerns such
as out-of-order packets (you don't send the next packet until the
previous one is acknowledged, and you retry on timeouts), and if you are
one a single Ethernet network you have no worries about UDP ports (which
can be a pain through firewalls).
If you can make the embedded side the server, then you also don't have
an issue with naming, DNS, addressing, etc. - or rather, it is the PC
client that has this concern rather than the embedded system.
Just try to avoid having more than one TFTP connection to or from the
same embedded system at the same time - it's easier if you don't have to
keep track.
Reply by Don Y●September 8, 20162016-09-08
Hi George,
On 9/8/2016 10:23 AM, George Neuner wrote:
> Don is super, super cautious. [Sometimes rightly so 8-)]
> His preference for really tiny packets is based on fear of
> fragmentation in the network fabric.
It's also a factor in the TFTP implementation. Negotiating
packet sizes was not present in the original specification.
And, even if your client/server (depends on which end Randy is
NOT implementing) *can* negotiate a packet size > 512B,
there is no guarantee that it *will* negotiate the packet size
you desire
(RY's files are tiny so it's not really an issue. OTOH, if you
are wanting to transfer BIG files -- or, use a file oriented
protocol to transfer a stream in the guise of a "big file" -- then
relying on the ability to negotiate larger packets, *hoping*
the "other side" hasn't used an "int" for the packet counter
AND hoping it can support "block count rollover" can be
important in your implementation decision)
> For common networks, packets up to 1KB won't be fragmented. You need
> to worry about smaller if you are using modem/telephone (SLiP, PPP,
> etc.). I haven't followed the whole thread, so I may have missed what
> network hardware you expect to be using.
>
> UDP packets have a 16-bit checksum. In the [ancient] past, the
> checksum protected only the header information, but in modern
> implementations it protects the entire packet.
>
> http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htm
> http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm
But it doesn't protect the entire "file". I.e., if you don't
explicitly verify the correct number of blocks have been
received but simply rely on size of the last block being
"less than full" and INFER that the file is intact "because
all the checksums were correct" you can be lulled into thinking
you have the "correct" file contents when you may only have some
of it (assuming no adversaries).
If you allow for the file on the server to potentially be
*changed* while the transfer is underway, all sorts of interesting
problems can manifest -- depending on the TFTPd implementation
(e.g., getting packets A-J from fileA and packets K-Z from the
*revised* file A)
> Many desktop/server implementations limit UDP packets to match their
> OS VMM page size - typically 4KB. The UDP packet size technically
> allows for packets up to 64KB, and some implementations allow large
> packets, but you should limit to 4KB maximum for portability.
I see nothing wrong with sticking to the smaller packets. It's
"safe". And, the simplex/HDX nature of TFTP (REQ/ACK) isn't going
to "cost" that much for 9 packets -- not worth the effort to negotiate
a larger packet size or a larger window.
KISS.
When I PXE boot (which, in my case, relies on TFTP), I want
larger packets cuz I don't want the network to sit idle while
ACK's move up and down the network stacks -- "give me another
packet, and make it a big one, please!"
The point is to be aware of the sorts of things that *can* go
wrong so your code can be prepared for those "can't happens"
when they actually *do* happen! (WTF? I already ACK'd
packet #4 -- why am I seeing it AGAIN?? Should I *assume*
that this is a repeat copy of the previous one that I already
"verified" and ACK'd? As I happen to have a copy of that
earlier packet, should I, perhaps, verify that this new
one does, in fact, agree with it? It *should*, right??)
E.g., what do you do when some other host (misconfigured
or an adversary) starts issuing ACKs for those packets
cuz it thinks they are intended for *it*? Or, claims
the same IP? Or (gasp) the same MAC??!
"Cautious". Yeah, I guess so! :> Much easier to tell the
user that I'm seeing things that "can't happen" and explain
them than have to have him rely on being able to sniff
network traffic to figure out why data isn't going where
it should...
Reply by George Neuner●September 8, 20162016-09-08
On Thu, 08 Sep 2016 11:59:37 -0400, Randy Yates
<yates@digitalsignallabs.com> wrote:
>Don Y <blockedofcourse@foo.invalid> writes:
>>
>> You can either err on the side of being compatible with the most
>> recent standards and extensions to the original protocol (packet
>> size negotiation, window size negotiation, timeouts, etc.) *or*
>> try to aim for the "least imposing" set of requirements (512B blocks,
>> 32767-1 blocks) to accommodate potentially buggy server/client
>> implementations. In effect, this limits transfers to 16MB.
>>
>> All the caveats of UDP-based protocols also apply.
>
>Hi Don,
>
>The files are 4500 bytes each. They consist of binary data. The overall
>transfer rate is low, on the order of 1 KB/s or less.
>
>What "caveats of UDP-based protocols" are you referring to? Are you
>referring to things such as out-of-order packets and no error-checking?
>
>I am not really knowledgable on the lower layers of TCP/IP.
Don is super, super cautious. [Sometimes rightly so 8-)]
His preference for really tiny packets is based on fear of
fragmentation in the network fabric.
For common networks, packets up to 1KB won't be fragmented. You need
to worry about smaller if you are using modem/telephone (SLiP, PPP,
etc.). I haven't followed the whole thread, so I may have missed what
network hardware you expect to be using.
UDP packets have a 16-bit checksum. In the [ancient] past, the
checksum protected only the header information, but in modern
implementations it protects the entire packet.
http://www.tcpipguide.com/free/t_UDPMessageFormat-2.htmhttp://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader-2.htm
Many desktop/server implementations limit UDP packets to match their
OS VMM page size - typically 4KB. The UDP packet size technically
allows for packets up to 64KB, and some implementations allow large
packets, but you should limit to 4KB maximum for portability.
Hope this helps,
George