On 2019-09-21 David Brown wrote in comp.arch.embedded:> On 20/09/2019 18:27, Stef wrote: >> For transportation of several high speed serial ports, we want to use a >> serial over ethernet approach. At a minimum I need to transfer four ports >> with 512 byte packets that come in every 40 ms at 2 Mbps. > > If you can consider USB instead of Ethernet, an FTDI 4232H chip will > handle 4 ports at 3 Mbps without trouble.Funny you mention this, this exact combination is what we want to get rid of. The FTDI chip is much more sensitive to electromagnetic disturbances that anything else we ever used. :-( Especially EFT is a problem. Testing with an FTDI eval kit even resulted in failures when the kit was in the proximity of the EFT tester, not even connected. Even though the EFT test is a conducted immunity test. To add to the trouble, the FTDI drivers throw blue screens when the chip hangs during the test.> But there are no limits in LWIP speed for what you need. I have run a > board with a Modbus TCP/IP server handling 6 simultaneous connections > each with a transaction (request and reply) every millisecond - far more > packets than you need here.Good to know.> Your problem might be limiting parameters in the LWIP configuration, or > it might be in the basic structure of the code. You can try increasing > configuration parameters to see if it helps. You can also enable > statistics for LWIP, which can give you a clue - if you are getting > something on the failure counters, that will let you know what you need > to increase.Any idea what parameters to look at? Don't see too many in lwipopts.h already increased PBUF_POOL_SIZE and MEM_SIZE with no effect. Or parameters like LPC_NUM_BUFF_TXDESCS in lpc_17xx40xx_emac_config.h?> Another poster suggested using WireShark to see the traffic - that is > definitely a useful tool.Agreed. -- Stef (remove caps, dashes and .invalid from e-mail address to reply by mail) "It was a virgin forest, a place where the Hand of Man had never set foot."
High packet rate ethernet with LwIP
Started by ●September 20, 2019
Reply by ●September 23, 20192019-09-23
Reply by ●September 23, 20192019-09-23
On 2019-09-22 David Brown wrote in comp.arch.embedded:> On 21/09/2019 23:01, upsidedown@downunder.com wrote: > >> The OP did not specify if those 2 Mbit/s streams had some kind of flow >> control. If not, a congestion problem could cause an overflow and >> possibly lockup. > > Of course he needs to support the UART side of this with the four > channels coming in - I assumed that much was obvious, and that he could > handle that side of the problem easily enough. And I also assumed that > there is no problem with a bit of variable latency on the transmission > of packets through the system and over the Ethernet, otherwise the whole > idea is infeasible.Yes, four ports with data coming in without flow control. It's is measurement data that is produced at this rate. A bit of latency is allowed, but not too much. That is partly why the serial bit rate is that high. At this rate the 512 byte packets are transferred in under 3 ms instead of using the 'whole' 40 ms. -- Stef (remove caps, dashes and .invalid from e-mail address to reply by mail) Linux: Where do you want to GO... Oh, I'm already there! -- Ewout Stam
Reply by ●September 23, 20192019-09-23
On 23.9.19 11:37, Stef wrote:> On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded: >> On Fri, 20 Sep 2019 18:27:52 +0200, Stef >> <stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote: >> >>> For transportation of several high speed serial ports, we want to use a >>> serial over ethernet approach. At a minimum I need to transfer four ports >>> with 512 byte packets that come in every 40 ms at 2 Mbps. >>> >>> Search for a ready to use (for test at least) device has turned up nothing. >>> Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster >>> ones are single port only. And it should be a small device, embeddable, >>> logic level is all that is needed. If anyone knows of a usable device... >>> >>> Since the board that this is on incorparates an ethernet capable controller >>> (LPC4088) that has not that much work to do richt now, I thought it should >>> be possible to add ethernet to it. >>> >>> For testing I got an LPC4088 demo board with ethernet interface and got >>> LwIP running on it (stand alone, no OS). >>> >>> Got it working OK to simulate 4 ports sending slow data to 4 TCP ports >>> that can be opened with a PC terminal program. >>> >>> But when I turn up the data volume and packet rate, it stops working. >>> In some cases stuck in the hard fault handler, in other cases in a >>> timed waiting loop in the LwIP stack. >>> >>> At this moment it is running fine with 2 ports open and a 512 byte packet >>> every 100 ms on each. When I increase the speed to 50 ms, it stops after >>> a while. >>> >>> I have no previous experience with LwIP so where to start looking? >>> - First of all does it sound feasable over a 100 Mbit ethernet >>> connection using TCP? >>> - What can be tuned in LwIP to improve this? >>> I have already increased PBUF_POOL_SIZE from default 7 to 64 and >>> MEM_SIZE from 12k to 32k >> >> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for >> TCP retransmissions. If each serial line pumps data at 2 Mbit/s >> without any flow control, a TCP retransmission will upset the Ethernet >> traffic for a while, possibly overflowing buffers. Anyway, if retrials >> are expected, the gross (ethernet) speed must be several times larger >> than the net data rate. > > No, not done that yet, but indeed something I need to do. > >> Is it possible to use UDP and accept that some packets may be lost ? >> Include a serial number in each frame and the receiver can detect >> missing packets (that have failed CRC test) and mark the missing data >> accordingly. This way the net speed can be much closer to the raw data >> rate. > > That is something that came to my mind just after posting the question. > Would UDP be a better match for this application? The whole TCP > retransmission and timers may be the mechanisms causing the problem. > I think the standard TCP timeouts ar far too slow for this kind of > thing? Think I saw a 250 ms timer tick in the LwIP docs? > > Missing packets may not be a problem. It is measurement data and if > any re-transmit takes over 40 ms there is no point anyway, there is > already new data. On a much slower rate, some control connection does > have to be reliable. So that could use TCP (if is does not disturb > the parallel UDP flow in LwIP) or use UDP as well and implement the > retries in the application.UDP may be the better option here. TCP does not guarantee to transfer the packet boundaries, but UDP is just a lightweight cover for raw IP packets. If your raw data comes in with asynchronous serial lines, the non- reliability of datagram traffic is already there, and it does not matter much what UDP adds there. Your processor may be overwhelmed with serial line character interrupt load. -- -TV
Reply by ●September 23, 20192019-09-23
On 9/23/19 4:37 AM, Stef wrote:> On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded: >> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for >> TCP retransmissions. If each serial line pumps data at 2 Mbit/s >> without any flow control, a TCP retransmission will upset the Ethernet >> traffic for a while, possibly overflowing buffers. Anyway, if retrials >> are expected, the gross (ethernet) speed must be several times larger >> than the net data rate. > > No, not done that yet, but indeed something I need to do. > >> Is it possible to use UDP and accept that some packets may be lost ? >> Include a serial number in each frame and the receiver can detect >> missing packets (that have failed CRC test) and mark the missing data >> accordingly. This way the net speed can be much closer to the raw data >> rate. > > That is something that came to my mind just after posting the question. > Would UDP be a better match for this application? The whole TCP > retransmission and timers may be the mechanisms causing the problem. > I think the standard TCP timeouts ar far too slow for this kind of > thing? Think I saw a 250 ms timer tick in the LwIP docs? > > Missing packets may not be a problem. It is measurement data and if > any re-transmit takes over 40 ms there is no point anyway, there is > already new data. On a much slower rate, some control connection does > have to be reliable. So that could use TCP (if is does not disturb > the parallel UDP flow in LwIP) or use UDP as well and implement the > retries in the application. > >If occasional missing packets aren't an issue, but timely delivery is, then TCP isn't really the right choice. TCP is designed to get packets there eventually (if possible), at the expense of possible delays. It is also generally tuned for a long path (and thus the long delays for re-transmission).
Reply by ●September 23, 20192019-09-23
On 23/09/2019 10:37, Stef wrote:> On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded: >> On Fri, 20 Sep 2019 18:27:52 +0200, Stef >> <stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote: >> >>> For transportation of several high speed serial ports, we want to use a >>> serial over ethernet approach. At a minimum I need to transfer four ports >>> with 512 byte packets that come in every 40 ms at 2 Mbps. >>> >>> Search for a ready to use (for test at least) device has turned up nothing. >>> Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster >>> ones are single port only. And it should be a small device, embeddable, >>> logic level is all that is needed. If anyone knows of a usable device... >>> >>> Since the board that this is on incorparates an ethernet capable controller >>> (LPC4088) that has not that much work to do richt now, I thought it should >>> be possible to add ethernet to it. >>> >>> For testing I got an LPC4088 demo board with ethernet interface and got >>> LwIP running on it (stand alone, no OS). >>> >>> Got it working OK to simulate 4 ports sending slow data to 4 TCP ports >>> that can be opened with a PC terminal program. >>> >>> But when I turn up the data volume and packet rate, it stops working. >>> In some cases stuck in the hard fault handler, in other cases in a >>> timed waiting loop in the LwIP stack. >>> >>> At this moment it is running fine with 2 ports open and a 512 byte packet >>> every 100 ms on each. When I increase the speed to 50 ms, it stops after >>> a while. >>> >>> I have no previous experience with LwIP so where to start looking? >>> - First of all does it sound feasable over a 100 Mbit ethernet >>> connection using TCP? >>> - What can be tuned in LwIP to improve this? >>> I have already increased PBUF_POOL_SIZE from default 7 to 64 and >>> MEM_SIZE from 12k to 32k >> >> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for >> TCP retransmissions. If each serial line pumps data at 2 Mbit/s >> without any flow control, a TCP retransmission will upset the Ethernet >> traffic for a while, possibly overflowing buffers. Anyway, if retrials >> are expected, the gross (ethernet) speed must be several times larger >> than the net data rate. > > No, not done that yet, but indeed something I need to do. > >> Is it possible to use UDP and accept that some packets may be lost ? >> Include a serial number in each frame and the receiver can detect >> missing packets (that have failed CRC test) and mark the missing data >> accordingly. This way the net speed can be much closer to the raw data >> rate. > > That is something that came to my mind just after posting the question. > Would UDP be a better match for this application? The whole TCP > retransmission and timers may be the mechanisms causing the problem. > I think the standard TCP timeouts ar far too slow for this kind of > thing? Think I saw a 250 ms timer tick in the LwIP docs?TCP/IP timeouts are slow, yes - you need to consider if that will be an issue. But TCP/IP gives you better guarantees about delivery - either the packets will all arrive, with correct contents, sorted in the right order, or you will get a clear failure indication for the whole connection. With UDP, you need to handle this manually - you might need more checksumming, re-ordering, re-tries, etc., depending on the application. But UDP has fewer overheads, and with manual coding comes greater control and flexibility. TCP/IP is also /much/ easier to handle over longer networks - through NAT routers, VPN's, ssh tunnels, and the like. Note that if you are getting retries or re-ordering of TCP/IP packets while you are on local network, you have something badly wrong with the software or the network - retries should be very rare.> > Missing packets may not be a problem. It is measurement data and if > any re-transmit takes over 40 ms there is no point anyway, there is > already new data. On a much slower rate, some control connection does > have to be reliable. So that could use TCP (if is does not disturb > the parallel UDP flow in LwIP) or use UDP as well and implement the > retries in the application. > >
Reply by ●September 23, 20192019-09-23
On 9/23/2019 15:04, David Brown wrote:> ..... > > Note that if you are getting retries or re-ordering of TCP/IP packets > while you are on local network, you have something badly wrong with the > software or the network - retries should be very rare. >Generally true but you can be easily surprised - like I was a while ago. A new windows 10 based laptop, about top of the line Acer had less than a year ago, was communicating with my DPS system a lot slower than my older windows XP based laptop (also Acer, about 10 years old). This via ftp at 100 Mbps, all 3 devices connected to the same switch (I let some huge file, 1G or more, transfer and watch how it goes). I started to investigate and it turned out the windows 10 machine was just losing too many packets and my (well, that of the DPS machine) retries were not aggressive enough. Making them so took care of that, but the surprise was there allright. Did not investigate seriously the windows side as for me it was easier to do it on the DPS side. Dimiter ====================================================== Dimiter Popoff, TGI http://www.tgi-sci.com ====================================================== http://www.flickr.com/photos/didi_tgi/
Reply by ●September 23, 20192019-09-23
On 23.9.19 14:13, Richard Damon wrote:> On 9/23/19 4:37 AM, Stef wrote: >> On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded: >>> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for >>> TCP retransmissions. If each serial line pumps data at 2 Mbit/s >>> without any flow control, a TCP retransmission will upset the Ethernet >>> traffic for a while, possibly overflowing buffers. Anyway, if retrials >>> are expected, the gross (ethernet) speed must be several times larger >>> than the net data rate. >> >> No, not done that yet, but indeed something I need to do. >> >>> Is it possible to use UDP and accept that some packets may be lost ? >>> Include a serial number in each frame and the receiver can detect >>> missing packets (that have failed CRC test) and mark the missing data >>> accordingly. This way the net speed can be much closer to the raw data >>> rate. >> >> That is something that came to my mind just after posting the question. >> Would UDP be a better match for this application? The whole TCP >> retransmission and timers may be the mechanisms causing the problem. >> I think the standard TCP timeouts ar far too slow for this kind of >> thing? Think I saw a 250 ms timer tick in the LwIP docs? >> >> Missing packets may not be a problem. It is measurement data and if >> any re-transmit takes over 40 ms there is no point anyway, there is >> already new data. On a much slower rate, some control connection does >> have to be reliable. So that could use TCP (if is does not disturb >> the parallel UDP flow in LwIP) or use UDP as well and implement the >> retries in the application. >> >> > > If occasional missing packets aren't an issue, but timely delivery is, > then TCP isn't really the right choice. TCP is designed to get packets > there eventually (if possible), at the expense of possible delays. It is > also generally tuned for a long path (and thus the long delays for > re-transmission).TCP will get the octets (bytes) there in the correct order, but it is explicitly specified that it does not transfer the packet boundaries. Many TCP stacks transfer the data in the same size chunks the user gives them for transport, but it is allowed to re-package the octet stream in any way the routers under way feel fit. UDP does not need any more checksumming, the Ethernet, IP and UDP layers have already taken care of that, discarding bad datagrams. There are good reasons to add packet sequence numbering to the datagrams sent, to track possibly discarded datagrams. -- -TV
Reply by ●September 23, 20192019-09-23
On 23/09/2019 16:03, Dimiter_Popoff wrote:> On 9/23/2019 15:04, David Brown wrote: >> ..... >> >> Note that if you are getting retries or re-ordering of TCP/IP packets >> while you are on local network, you have something badly wrong with the >> software or the network - retries should be very rare. >> > > Generally true but you can be easily surprised - like I was a while ago. > A new windows 10 based laptop, about top of the line Acer had less than > a year ago, was communicating with my DPS system a lot slower than > my older windows XP based laptop (also Acer, about 10 years old). > This via ftp at 100 Mbps, all 3 devices connected to the same switch > (I let some huge file, 1G or more, transfer and watch how it goes). > > I started to investigate and it turned out the windows 10 machine > was just losing too many packets and my (well, that of the DPS machine) > retries were not aggressive enough. Making them so took care of that, > but the surprise was there allright. Did not investigate seriously the > windows side as for me it was easier to do it on the DPS side. >There you go - something was badly wrong with the network. It had a Windows 10 machine on it :-)
Reply by ●September 23, 20192019-09-23
On Mon, 23 Sep 2019 10:37:17 +0200, Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:>On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded: >> On Fri, 20 Sep 2019 18:27:52 +0200, Stef >><stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote: >> >>>For transportation of several high speed serial ports, we want to use a >>>serial over ethernet approach. At a minimum I need to transfer four ports >>>with 512 byte packets that come in every 40 ms at 2 Mbps. >>> >>>Search for a ready to use (for test at least) device has turned up nothing. >>>Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster >>>ones are single port only. And it should be a small device, embeddable, >>>logic level is all that is needed. If anyone knows of a usable device... >>> >>>Since the board that this is on incorparates an ethernet capable controller >>>(LPC4088) that has not that much work to do richt now, I thought it should >>>be possible to add ethernet to it. >>> >>>For testing I got an LPC4088 demo board with ethernet interface and got >>>LwIP running on it (stand alone, no OS). >>> >>>Got it working OK to simulate 4 ports sending slow data to 4 TCP ports >>>that can be opened with a PC terminal program. >>> >>>But when I turn up the data volume and packet rate, it stops working. >>>In some cases stuck in the hard fault handler, in other cases in a >>>timed waiting loop in the LwIP stack. >>> >>>At this moment it is running fine with 2 ports open and a 512 byte packet >>>every 100 ms on each. When I increase the speed to 50 ms, it stops after >>>a while. >>> >>>I have no previous experience with LwIP so where to start looking? >>>- First of all does it sound feasable over a 100 Mbit ethernet >>> connection using TCP? >>>- What can be tuned in LwIP to improve this? >>> I have already increased PBUF_POOL_SIZE from default 7 to 64 and >>> MEM_SIZE from 12k to 32k >> >> Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for >> TCP retransmissions. If each serial line pumps data at 2 Mbit/s >> without any flow control, a TCP retransmission will upset the Ethernet >> traffic for a while, possibly overflowing buffers. Anyway, if retrials >> are expected, the gross (ethernet) speed must be several times larger >> than the net data rate. > >No, not done that yet, but indeed something I need to do.Looking more closely on your original question, the actual duty cycle is less than 10 %, so you could do the same thing with four 230 kbit/s streams, thus the speed requirement is no big deal. There are lots of eth/serial converters supporting 4-8 serial lines at 230 kbit/s.>> Is it possible to use UDP and accept that some packets may be lost ? >> Include a serial number in each frame and the receiver can detect >> missing packets (that have failed CRC test) and mark the missing data >> accordingly. This way the net speed can be much closer to the raw data >> rate. > >That is something that came to my mind just after posting the question. >Would UDP be a better match for this application?Using MAC or UDP framing is just using Ethernet as some very high speed serial connections. The advantage of UDP compared to MAC addressing is that ARP and more user friendly IP addressed compared to Ethernet MAC addresses can be used. In addition, there is a better UDP support in Windows, compared to raw Ethernet MAC framing.>The whole TCP >retransmission and timers may be the mechanisms causing the problem. >I think the standard TCP timeouts ar far too slow for this kind of >thing? Think I saw a 250 ms timer tick in the LwIP docs?You really have to do the transmission and possible retransmission in your 40 ms time window.>Missing packets may not be a problem. It is measurement data and if >any re-transmit takes over 40 ms there is no point anyway, there is >already new data.Just make sure you know if some frames are missing (e.g. serial numbering).>On a much slower rate, some control connection does >have to be reliable. So that could use TCP (if is does not disturb >the parallel UDP flow in LwIP) or use UDP as well and implement the >retries in the application.Why not, you can use both TCP and UDP as two separate links, even with the same IP-address and port number.
Reply by ●September 23, 20192019-09-23
On 2019-09-23 upsidedown@downunder.com wrote in comp.arch.embedded:> On Mon, 23 Sep 2019 10:37:17 +0200, Stef ><stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote: > >>On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded: >>> On Fri, 20 Sep 2019 18:27:52 +0200, Stef >>><stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote: >>> >>>>For transportation of several high speed serial ports, we want to use a >>>>serial over ethernet approach. At a minimum I need to transfer four ports >>>>with 512 byte packets that come in every 40 ms at 2 Mbps. >>>> >>>>Search for a ready to use (for test at least) device has turned up nothing. >>>>Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster >>>>ones are single port only. And it should be a small device, embeddable, >>>>logic level is all that is needed. If anyone knows of a usable device...> Looking more closely on your original question, the actual duty cycle > is less than 10 %, so you could do the same thing with four 230 kbit/s > streams, thus the speed requirement is no big deal. There are lots of > eth/serial converters supporting 4-8 serial lines at 230 kbit/s.Yes, I found those. Unfortunately most are quite bulky and include level conversion. At 230kbs, the packets would need almost 25 ms to transfer. Although this will fit in the 40 ms interval, the latency is higher than desired.>>> Is it possible to use UDP and accept that some packets may be lost ? >>> Include a serial number in each frame and the receiver can detect >>> missing packets (that have failed CRC test) and mark the missing data >>> accordingly. This way the net speed can be much closer to the raw data >>> rate. >> >>That is something that came to my mind just after posting the question. >>Would UDP be a better match for this application? > > Using MAC or UDP framing is just using Ethernet as some very high > speed serial connections. The advantage of UDP compared to MAC > addressing is that ARP and more user friendly IP addressed compared to > Ethernet MAC addresses can be used. In addition, there is a better UDP > support in Windows, compared to raw Ethernet MAC framing. > > >>The whole TCP >>retransmission and timers may be the mechanisms causing the problem. >>I think the standard TCP timeouts ar far too slow for this kind of >>thing? Think I saw a 250 ms timer tick in the LwIP docs? > > You really have to do the transmission and possible retransmission in > your 40 ms time window.Yes, and I think (but have not yet been able to verify) this is the problem with the (setup of the) LwIP stack right now.>>Missing packets may not be a problem. It is measurement data and if >>any re-transmit takes over 40 ms there is no point anyway, there is >>already new data. > > Just make sure you know if some frames are missing (e.g. serial > numbering).Yes, we need something to detect missing packets.>>On a much slower rate, some control connection does >>have to be reliable. So that could use TCP (if is does not disturb >>the parallel UDP flow in LwIP) or use UDP as well and implement the >>retries in the application. > > Why not, you can use both TCP and UDP as two separate links, even with > the same IP-address and port number.Yes, that is true, but does LwIP support this? I don't know how the final output to the MAC is treated, so in a (dumb) implementation, a TCP retransmission could hold up everything. Not that I expect this from LwIP (as it is widely used), but it is something to check. -- Stef (remove caps, dashes and .invalid from e-mail address to reply by mail) Your own mileage may vary.
