For transportation of several high speed serial ports, we want to use a serial over ethernet approach. At a minimum I need to transfer four ports with 512 byte packets that come in every 40 ms at 2 Mbps. Search for a ready to use (for test at least) device has turned up nothing. Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster ones are single port only. And it should be a small device, embeddable, logic level is all that is needed. If anyone knows of a usable device... Since the board that this is on incorparates an ethernet capable controller (LPC4088) that has not that much work to do richt now, I thought it should be possible to add ethernet to it. For testing I got an LPC4088 demo board with ethernet interface and got LwIP running on it (stand alone, no OS). Got it working OK to simulate 4 ports sending slow data to 4 TCP ports that can be opened with a PC terminal program. But when I turn up the data volume and packet rate, it stops working. In some cases stuck in the hard fault handler, in other cases in a timed waiting loop in the LwIP stack. At this moment it is running fine with 2 ports open and a 512 byte packet every 100 ms on each. When I increase the speed to 50 ms, it stops after a while. I have no previous experience with LwIP so where to start looking? - First of all does it sound feasable over a 100 Mbit ethernet connection using TCP? - What can be tuned in LwIP to improve this? I have already increased PBUF_POOL_SIZE from default 7 to 64 and MEM_SIZE from 12k to 32k -- Stef (remove caps, dashes and .invalid from e-mail address to reply by mail) A man who keeps stealing mopeds is an obvious cycle-path.
High packet rate ethernet with LwIP
Started by ●September 20, 2019
Reply by ●September 20, 20192019-09-20
On Fri, 20 Sep 2019 18:27:52 +0200, Stef <stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote:>For transportation of several high speed serial ports, we want to use a >serial over ethernet approach. At a minimum I need to transfer four ports >with 512 byte packets that come in every 40 ms at 2 Mbps. > >Search for a ready to use (for test at least) device has turned up nothing. >Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster >ones are single port only. And it should be a small device, embeddable, >logic level is all that is needed. If anyone knows of a usable device... > >Since the board that this is on incorparates an ethernet capable controller >(LPC4088) that has not that much work to do richt now, I thought it should >be possible to add ethernet to it. > >For testing I got an LPC4088 demo board with ethernet interface and got >LwIP running on it (stand alone, no OS). > >Got it working OK to simulate 4 ports sending slow data to 4 TCP ports >that can be opened with a PC terminal program. > >But when I turn up the data volume and packet rate, it stops working. >In some cases stuck in the hard fault handler, in other cases in a >timed waiting loop in the LwIP stack. > >At this moment it is running fine with 2 ports open and a 512 byte packet >every 100 ms on each. When I increase the speed to 50 ms, it stops after >a while. > >I have no previous experience with LwIP so where to start looking? >- First of all does it sound feasable over a 100 Mbit ethernet > connection using TCP? >- What can be tuned in LwIP to improve this? > I have already increased PBUF_POOL_SIZE from default 7 to 64 and > MEM_SIZE from 12k to 32kHave you looked at the Ethernet traffic with e.g. Wireshark ? Look for TCP retransmissions. If each serial line pumps data at 2 Mbit/s without any flow control, a TCP retransmission will upset the Ethernet traffic for a while, possibly overflowing buffers. Anyway, if retrials are expected, the gross (ethernet) speed must be several times larger than the net data rate. Is it possible to use UDP and accept that some packets may be lost ? Include a serial number in each frame and the receiver can detect missing packets (that have failed CRC test) and mark the missing data accordingly. This way the net speed can be much closer to the raw data rate.
Reply by ●September 20, 20192019-09-20
On Fri, 20 Sep 2019 18:27:52 +0200, Stef wrote:> For transportation of several high speed serial ports, we want to use a > serial over ethernet approach. At a minimum I need to transfer four > ports with 512 byte packets that come in every 40 ms at 2 Mbps. > > Search for a ready to use (for test at least) device has turned up > nothing. > Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster > ones are single port only. And it should be a small device, embeddable, > logic level is all that is needed. If anyone knows of a usable device... > > Since the board that this is on incorparates an ethernet capable > controller (LPC4088) that has not that much work to do richt now, I > thought it should be possible to add ethernet to it. > > For testing I got an LPC4088 demo board with ethernet interface and got > LwIP running on it (stand alone, no OS). > > Got it working OK to simulate 4 ports sending slow data to 4 TCP ports > that can be opened with a PC terminal program. > > But when I turn up the data volume and packet rate, it stops working. > In some cases stuck in the hard fault handler, in other cases in a timed > waiting loop in the LwIP stack. > > At this moment it is running fine with 2 ports open and a 512 byte > packet every 100 ms on each. When I increase the speed to 50 ms, it > stops after a while. > > I have no previous experience with LwIP so where to start looking? > - First of all does it sound feasable over a 100 Mbit ethernet > connection using TCP? > - What can be tuned in LwIP to improve this? > I have already increased PBUF_POOL_SIZE from default 7 to 64 and > MEM_SIZE from 12k to 32kPost to the lwip-users list. They may be able to help. It's very active https://lists.nongnu.org/mailman/listinfo/lwip-users -- Chisolm Texas-American
Reply by ●September 21, 20192019-09-21
On 20/09/2019 18:27, Stef wrote:> For transportation of several high speed serial ports, we want to use a > serial over ethernet approach. At a minimum I need to transfer four ports > with 512 byte packets that come in every 40 ms at 2 Mbps.If you can consider USB instead of Ethernet, an FTDI 4232H chip will handle 4 ports at 3 Mbps without trouble. But there are no limits in LWIP speed for what you need. I have run a board with a Modbus TCP/IP server handling 6 simultaneous connections each with a transaction (request and reply) every millisecond - far more packets than you need here. Your problem might be limiting parameters in the LWIP configuration, or it might be in the basic structure of the code. You can try increasing configuration parameters to see if it helps. You can also enable statistics for LWIP, which can give you a clue - if you are getting something on the failure counters, that will let you know what you need to increase. Another poster suggested using WireShark to see the traffic - that is definitely a useful tool.> > Search for a ready to use (for test at least) device has turned up nothing. > Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster > ones are single port only. And it should be a small device, embeddable, > logic level is all that is needed. If anyone knows of a usable device... > > Since the board that this is on incorparates an ethernet capable controller > (LPC4088) that has not that much work to do richt now, I thought it should > be possible to add ethernet to it. > > For testing I got an LPC4088 demo board with ethernet interface and got > LwIP running on it (stand alone, no OS). > > Got it working OK to simulate 4 ports sending slow data to 4 TCP ports > that can be opened with a PC terminal program. > > But when I turn up the data volume and packet rate, it stops working. > In some cases stuck in the hard fault handler, in other cases in a > timed waiting loop in the LwIP stack. > > At this moment it is running fine with 2 ports open and a 512 byte packet > every 100 ms on each. When I increase the speed to 50 ms, it stops after > a while. > > I have no previous experience with LwIP so where to start looking? > - First of all does it sound feasable over a 100 Mbit ethernet > connection using TCP? > - What can be tuned in LwIP to improve this? > I have already increased PBUF_POOL_SIZE from default 7 to 64 and > MEM_SIZE from 12k to 32k > >
Reply by ●September 21, 20192019-09-21
On Sat, 21 Sep 2019 19:00:16 +0200, David Brown <david.brown@hesbynett.no> wrote:>On 20/09/2019 18:27, Stef wrote: >> For transportation of several high speed serial ports, we want to use a >> serial over ethernet approach. At a minimum I need to transfer four ports >> with 512 byte packets that come in every 40 ms at 2 Mbps. > >If you can consider USB instead of Ethernet, an FTDI 4232H chip will >handle 4 ports at 3 Mbps without trouble. > >But there are no limits in LWIP speed for what you need. I have run a >board with a Modbus TCP/IP server handling 6 simultaneous connections >each with a transaction (request and reply) every millisecond - far more >packets than you need here.That is essentially a half duplex environment with inherit flow control. So if there is some congestion issues, it will clear sooner or later. The OP did not specify if those 2 Mbit/s streams had some kind of flow control. If not, a congestion problem could cause an overflow and possibly lockup. Anyway, I did not notice that the packets came with such low duty cycle, so even a 10baseT connections should be able to handle four streams even with TCP/IP even with some retransmissions, provided that a missed frame is reacted upon in less than 40 ms.> >Your problem might be limiting parameters in the LWIP configuration, or >it might be in the basic structure of the code. You can try increasing >configuration parameters to see if it helps. You can also enable >statistics for LWIP, which can give you a clue - if you are getting >something on the failure counters, that will let you know what you need >to increase. > >Another poster suggested using WireShark to see the traffic - that is >definitely a useful tool. > >> >> Search for a ready to use (for test at least) device has turned up nothing. >> Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster >> ones are single port only. And it should be a small device, embeddable, >> logic level is all that is needed. If anyone knows of a usable device... >> >> Since the board that this is on incorparates an ethernet capable controller >> (LPC4088) that has not that much work to do richt now, I thought it should >> be possible to add ethernet to it. >> >> For testing I got an LPC4088 demo board with ethernet interface and got >> LwIP running on it (stand alone, no OS). >> >> Got it working OK to simulate 4 ports sending slow data to 4 TCP ports >> that can be opened with a PC terminal program. >> >> But when I turn up the data volume and packet rate, it stops working. >> In some cases stuck in the hard fault handler, in other cases in a >> timed waiting loop in the LwIP stack. >> >> At this moment it is running fine with 2 ports open and a 512 byte packet >> every 100 ms on each. When I increase the speed to 50 ms, it stops after >> a while. >> >> I have no previous experience with LwIP so where to start looking? >> - First of all does it sound feasable over a 100 Mbit ethernet >> connection using TCP? >> - What can be tuned in LwIP to improve this? >> I have already increased PBUF_POOL_SIZE from default 7 to 64 and >> MEM_SIZE from 12k to 32k >> >>
Reply by ●September 22, 20192019-09-22
On 21/09/2019 23:01, upsidedown@downunder.com wrote:> On Sat, 21 Sep 2019 19:00:16 +0200, David Brown > <david.brown@hesbynett.no> wrote: > >> On 20/09/2019 18:27, Stef wrote: >>> For transportation of several high speed serial ports, we want to use a >>> serial over ethernet approach. At a minimum I need to transfer four ports >>> with 512 byte packets that come in every 40 ms at 2 Mbps. >> >> If you can consider USB instead of Ethernet, an FTDI 4232H chip will >> handle 4 ports at 3 Mbps without trouble. >> >> But there are no limits in LWIP speed for what you need. I have run a >> board with a Modbus TCP/IP server handling 6 simultaneous connections >> each with a transaction (request and reply) every millisecond - far more >> packets than you need here. > > That is essentially a half duplex environment with inherit flow > control. So if there is some congestion issues, it will clear sooner > or later. >It is half duplex on each connection, but full duplex overall (you can receive a request on one connection while sending out a reply on another, and there is always other traffic such as ARP or broadcasts flying around). That is always the case with TCP/IP and Ethernet.> The OP did not specify if those 2 Mbit/s streams had some kind of flow > control. If not, a congestion problem could cause an overflow and > possibly lockup.Of course he needs to support the UART side of this with the four channels coming in - I assumed that much was obvious, and that he could handle that side of the problem easily enough. And I also assumed that there is no problem with a bit of variable latency on the transmission of packets through the system and over the Ethernet, otherwise the whole idea is infeasible.> > Anyway, I did not notice that the packets came with such low duty > cycle, so even a 10baseT connections should be able to handle four > streams even with TCP/IP even with some retransmissions, provided > that a missed frame is reacted upon in less than 40 ms. >Yes. It should be entirely possible, without pushing anything to its limits.
Reply by ●September 22, 20192019-09-22
On Sun, 22 Sep 2019 09:06:24 +0200, David Brown <david.brown@hesbynett.no> wrote:>On 21/09/2019 23:01, upsidedown@downunder.com wrote: >> On Sat, 21 Sep 2019 19:00:16 +0200, David Brown >> <david.brown@hesbynett.no> wrote: >> >>> On 20/09/2019 18:27, Stef wrote: >>>> For transportation of several high speed serial ports, we want to use a >>>> serial over ethernet approach. At a minimum I need to transfer four ports >>>> with 512 byte packets that come in every 40 ms at 2 Mbps. >>> >>> If you can consider USB instead of Ethernet, an FTDI 4232H chip will >>> handle 4 ports at 3 Mbps without trouble. >>> >>> But there are no limits in LWIP speed for what you need. I have run a >>> board with a Modbus TCP/IP server handling 6 simultaneous connections >>> each with a transaction (request and reply) every millisecond - far more >>> packets than you need here. >> >> That is essentially a half duplex environment with inherit flow >> control. So if there is some congestion issues, it will clear sooner >> or later. >> > >It is half duplex on each connection, but full duplex overall (you can >receive a request on one connection while sending out a reply on >another, and there is always other traffic such as ARP or broadcasts >flying around). That is always the case with TCP/IP and Ethernet.While it is legal for the Modbus/TCP client to send out multiple request to a single server, without first waiting for response for previous requests, there are some practical maximum number of outstanding request If not otherwise but at least when the client send window is full (request not TCP/IP acknowledged by the server TCP/IP stack). On the server side, if there are performance issues, it should artificially delay the responses, thus ultimately forcing the client to suspend the next request. When receiving requests from multiple clients, the TCP/IP stack should not acknowledge a request, until there are Rx buffers available or when the Modbus/TCP server is ready to handle next Modbus request/reply sequence, All this reduces throughput during a congestion, but at least maintains link sanity.> >> The OP did not specify if those 2 Mbit/s streams had some kind of flow >> control. If not, a congestion problem could cause an overflow and >> possibly lockup. > >Of course he needs to support the UART side of this with the four >channels coming in - I assumed that much was obvious, and that he could >handle that side of the problem easily enough. And I also assumed that >there is no problem with a bit of variable latency on the transmission >of packets through the system and over the Ethernet, otherwise the whole >idea is infeasible.If there is no flow control on the serial side, then this is a _hard_ realtime case, so the system *must* respond to a request no matter what, before the next packet arrives, in this case 40 ms. In some cases it might be possible to do some buffering on the serial side, but the buffer size needs to be large to survive longer hiccups. Anyway, the high level system design must know, how to handle overload issues (alarming, freezing, interpolation etc.) in case of failure.>> Anyway, I did not notice that the packets came with such low duty >> cycle, so even a 10baseT connections should be able to handle four >> streams even with TCP/IP even with some retransmissions, provided >> that a missed frame is reacted upon in less than 40 ms. >> > >Yes. It should be entirely possible, without pushing anything to its >limits. >
Reply by ●September 22, 20192019-09-22
On 22/09/2019 10:29, upsidedown@downunder.com wrote:> On Sun, 22 Sep 2019 09:06:24 +0200, David Brown > <david.brown@hesbynett.no> wrote: > >> On 21/09/2019 23:01, upsidedown@downunder.com wrote: >>> On Sat, 21 Sep 2019 19:00:16 +0200, David Brown >>> <david.brown@hesbynett.no> wrote: >>> >>>> On 20/09/2019 18:27, Stef wrote: >>>>> For transportation of several high speed serial ports, we want to use a >>>>> serial over ethernet approach. At a minimum I need to transfer four ports >>>>> with 512 byte packets that come in every 40 ms at 2 Mbps. >>>> >>>> If you can consider USB instead of Ethernet, an FTDI 4232H chip will >>>> handle 4 ports at 3 Mbps without trouble. >>>> >>>> But there are no limits in LWIP speed for what you need. I have run a >>>> board with a Modbus TCP/IP server handling 6 simultaneous connections >>>> each with a transaction (request and reply) every millisecond - far more >>>> packets than you need here. >>> >>> That is essentially a half duplex environment with inherit flow >>> control. So if there is some congestion issues, it will clear sooner >>> or later. >>> >> >> It is half duplex on each connection, but full duplex overall (you can >> receive a request on one connection while sending out a reply on >> another, and there is always other traffic such as ARP or broadcasts >> flying around). That is always the case with TCP/IP and Ethernet. > > While it is legal for the Modbus/TCP client to send out multiple > request to a single server, without first waiting for response for > previous requests, there are some practical maximum number of > outstanding request If not otherwise but at least when the client send > window is full (request not TCP/IP acknowledged by the server TCP/IP > stack).The clients here were multiple independent processes on the same PC. This is perfectly reasonable for any type of TCP/IP service. On the server side, being a single core processor, it is entirely possible to have multiple requests waiting, though only one request is processed at a time. And of course only one request packet could be coming in on the Ethernet MAC at a time (multiple packets could be in-flight, buffered at switches along the way from the client to the server).> > On the server side, if there are performance issues, it should > artificially delay the responses, thus ultimately forcing the client > to suspend the next request.I was not pushing requests as fast as I could - they were being sent at approximately 1 ms intervals from each client.> When receiving requests from multiple > clients, the TCP/IP stack should not acknowledge a request, until > there are Rx buffers available or when the Modbus/TCP server is ready > to handle next Modbus request/reply sequence,Of course incoming packets are delayed if there are no Rx buffers available. But 6 buffers are all that are needed for 6 simultaneous connections (plus a buffer or two for "noise" Ethernet packets). The Modbus server code sees there is a request from a client, processes it, generates a reply, and sends that out to the LWIP stack. These replies are buffered, and passed out through the MAC when it is free.> > All this reduces throughput during a congestion, but at least > maintains link sanity.The link has a bandwidth ceiling - 100 Mbps total, the processor has speed limits for handling the packets, and there is a limit to the LWIP buffers. Unlimited links are hard to find. I am afraid I don't see where you are going with this. My point was merely to show that an LWIP based microcontroller card can handle a great deal more throughput of packets than the OP needed for his use - LWIP itself is not the limitation he feared.> >> >>> The OP did not specify if those 2 Mbit/s streams had some kind of flow >>> control. If not, a congestion problem could cause an overflow and >>> possibly lockup. >> >> Of course he needs to support the UART side of this with the four >> channels coming in - I assumed that much was obvious, and that he could >> handle that side of the problem easily enough. And I also assumed that >> there is no problem with a bit of variable latency on the transmission >> of packets through the system and over the Ethernet, otherwise the whole >> idea is infeasible. > > If there is no flow control on the serial side, then this is a _hard_ > realtime case, so the system *must* respond to a request no matter > what, before the next packet arrives, in this case 40 ms.We have no information about what flow control there might be, or what real-time and latency requirements there might be (and hard time limits are independent from flow control - no flow control does not imply hard timing requirements).> In some > cases it might be possible to do some buffering on the serial side, > but the buffer size needs to be large to survive longer hiccups.Obviously.> Anyway, the high level system design must know, how to handle overload > issues (alarming, freezing, interpolation etc.) in case of failure.Also obvious. I think it is fair to assume that the OP is aware of such things until he indicates otherwise. He merely wants to know if his throughput problems are due to a limitation of LWIP, or if LWIP can be configured to improve the throughput here.> > >>> Anyway, I did not notice that the packets came with such low duty >>> cycle, so even a 10baseT connections should be able to handle four >>> streams even with TCP/IP even with some retransmissions, provided >>> that a missed frame is reacted upon in less than 40 ms. >>> >> >> Yes. It should be entirely possible, without pushing anything to its >> limits. >> >
Reply by ●September 23, 20192019-09-23
On 2019-09-20 upsidedown@downunder.com wrote in comp.arch.embedded:> On Fri, 20 Sep 2019 18:27:52 +0200, Stef ><stef33d@yahooI-N-V-A-L-I-D.com.invalid> wrote: > >>For transportation of several high speed serial ports, we want to use a >>serial over ethernet approach. At a minimum I need to transfer four ports >>with 512 byte packets that come in every 40 ms at 2 Mbps. >> >>Search for a ready to use (for test at least) device has turned up nothing. >>Standard devices usually go up to 230 kbps, some to 1 Mbps. Most faster >>ones are single port only. And it should be a small device, embeddable, >>logic level is all that is needed. If anyone knows of a usable device... >> >>Since the board that this is on incorparates an ethernet capable controller >>(LPC4088) that has not that much work to do richt now, I thought it should >>be possible to add ethernet to it. >> >>For testing I got an LPC4088 demo board with ethernet interface and got >>LwIP running on it (stand alone, no OS). >> >>Got it working OK to simulate 4 ports sending slow data to 4 TCP ports >>that can be opened with a PC terminal program. >> >>But when I turn up the data volume and packet rate, it stops working. >>In some cases stuck in the hard fault handler, in other cases in a >>timed waiting loop in the LwIP stack. >> >>At this moment it is running fine with 2 ports open and a 512 byte packet >>every 100 ms on each. When I increase the speed to 50 ms, it stops after >>a while. >> >>I have no previous experience with LwIP so where to start looking? >>- First of all does it sound feasable over a 100 Mbit ethernet >> connection using TCP? >>- What can be tuned in LwIP to improve this? >> I have already increased PBUF_POOL_SIZE from default 7 to 64 and >> MEM_SIZE from 12k to 32k > > Have you looked at the Ethernet traffic with e.g. Wireshark ? Look for > TCP retransmissions. If each serial line pumps data at 2 Mbit/s > without any flow control, a TCP retransmission will upset the Ethernet > traffic for a while, possibly overflowing buffers. Anyway, if retrials > are expected, the gross (ethernet) speed must be several times larger > than the net data rate.No, not done that yet, but indeed something I need to do.> Is it possible to use UDP and accept that some packets may be lost ? > Include a serial number in each frame and the receiver can detect > missing packets (that have failed CRC test) and mark the missing data > accordingly. This way the net speed can be much closer to the raw data > rate.That is something that came to my mind just after posting the question. Would UDP be a better match for this application? The whole TCP retransmission and timers may be the mechanisms causing the problem. I think the standard TCP timeouts ar far too slow for this kind of thing? Think I saw a 250 ms timer tick in the LwIP docs? Missing packets may not be a problem. It is measurement data and if any re-transmit takes over 40 ms there is no point anyway, there is already new data. On a much slower rate, some control connection does have to be reliable. So that could use TCP (if is does not disturb the parallel UDP flow in LwIP) or use UDP as well and implement the retries in the application. -- Stef (remove caps, dashes and .invalid from e-mail address to reply by mail) Your temporary financial embarrassment will be relieved in a surprising manner.
Reply by ●September 23, 20192019-09-23
On 2019-09-20 Joe Chisolm wrote in comp.arch.embedded:> On Fri, 20 Sep 2019 18:27:52 +0200, Stef wrote:<snip>>> I have no previous experience with LwIP so where to start looking?<snip>> Post to the lwip-users list. They may be able to help. > It's very active > > https://lists.nongnu.org/mailman/listinfo/lwip-usersIndeed looks very active, thanks for the link. Never used a mailing list before :$. -- Stef (remove caps, dashes and .invalid from e-mail address to reply by mail)
