Shared Communications Bus - RS-422 or RS-485

I have a test fixture that uses RS-232 to communicate with a PC.  It actually uses the voltage levels of RS-232, even though this is from a USB cable on the PC, so it's only RS-232 for maybe four inches.  lol 

I'm redesigning the test fixtures to hold more units and fully automate a few features that presently requires an operator.  There will now be 8 UUTs on each test fixture and I expect to have 10 to 20 test fixtures in a card rack.  That's 80 to 160 UUTs total.  There will be an FPGA controlling each pair of UUTs, so 80 FPGAs in total that the PC needs to talk to.  

Rather than working on a way to mux 80 RS-232 interfaces, I'm thinking it would be better to either daisy chain, or connect in parallel all these devices.  The protocol is master-slave where the master sends a command and the slaves are idle until they reply.  The four FPGAs on a test fixture board could be connected in parallel easily enough.  But I don't think I want to run TTL level signals between so many boards.  

I could do an RS-422 interface with a master to slave pair and a slave to master pair.  The slaves do not speak until spoken to, so there will be no collisions.  

RS-485 would allow all this to be over a single pair of wires.  But the one big issue I see people complain about is getting PC software to not clobber the slaves, or I should say, to get the master to wait long enough that it's not clobbering it's own start bit by overwriting the stop bit of the slave.  I suppose someone, somewhere has dealt with this on the PC and has a solution that doesn't impact bus speed.  I run the single test fixture version of this at about 100 kbps.  I'm going to want as much speed as I can get for 80 FPGAs controlling 160 UUTs.  Maybe I should give that some analysis, because this might not be true.  

The tests are of two types, most of them are setting up a state and reading a signal.  This can go pretty fast and doesn't take too many commands.  Then there are the audio tests where the FPGA sends digital data to the UUT, which does it's thing and returns digital data which is crunched by the FPGA.  This takes some small number of seconds and presently the protocol is to poll the status until it is done.  That's a lot of messages, but it's not necessarily a slow point.  The same test can be started on every UUT in parallel, so the waiting is in parallel.  So maybe the serial port won't need to be any faster.  

Still, I want to use RS-422 or RS-485 to deal with ground noise since this will be spread over multiple boards that don't have terribly solid grounds, just the power cable really.  

I'm thinking out loud here as much as anything.  I intended to simply ask if anyone had experience with RS-485 that would be helpful.  Running two wires rather than eight would be a help.  I'll probably use a 10 pin connector just to be on the safe side, allowing the transceivers to be used either way.  

-- 

Rick C.

- Get 1,000 miles of free Supercharging
- Tesla referral code - https://ts.la/richard11209

On 02/11/2022 06:28, Rick C wrote:
> I have a test fixture that uses RS-232 to communicate with a PC.  It
> actually uses the voltage levels of RS-232, even though this is from
> a USB cable on the PC, so it's only RS-232 for maybe four inches.
> lol
> 
> I'm redesigning the test fixtures to hold more units and fully
> automate a few features that presently requires an operator.  There
> will now be 8 UUTs on each test fixture and I expect to have 10 to 20
> test fixtures in a card rack.  That's 80 to 160 UUTs total.  There
> will be an FPGA controlling each pair of UUTs, so 80 FPGAs in total
> that the PC needs to talk to.
> 
> Rather than working on a way to mux 80 RS-232 interfaces, I'm
> thinking it would be better to either daisy chain, or connect in
> parallel all these devices.  The protocol is master-slave where the
> master sends a command and the slaves are idle until they reply.  The
> four FPGAs on a test fixture board could be connected in parallel
> easily enough.  But I don't think I want to run TTL level signals
> between so many boards.
> 
> I could do an RS-422 interface with a master to slave pair and a
> slave to master pair.  The slaves do not speak until spoken to, so
> there will be no collisions.
> 

RS-422 is normally a point-to-point interface.  It is one line in each 
direction, but using balanced pairs instead of a TTL signal.  You would 
not normally connect multiple receivers or transmitters to an RS-422 
bus, as the standard practice is that each transmitter is always driving 
the pair it is attached to - there is no multi-drop.

> RS-485 would allow all this to be over a single pair of wires.  But
> the one big issue I see people complain about is getting PC software
> to not clobber the slaves, or I should say, to get the master to wait
> long enough that it's not clobbering it's own start bit by
> overwriting the stop bit of the slave.  I suppose someone, somewhere
> has dealt with this on the PC and has a solution that doesn't impact
> bus speed.  I run the single test fixture version of this at about
> 100 kbps.  I'm going to want as much speed as I can get for 80 FPGAs
> controlling 160 UUTs.  Maybe I should give that some analysis,
> because this might not be true.

RS-485 is easy from a PC using appropriate USB devices.  We make heavy 
use of FTDI's chips and cables - they handle the drive enables for the 
RS-485 automatically so that from the PC side you just read and write to 
the serial port.

The reception of the last byte from a slave is not finished until the 
stop bit has been properly received by the master - that means at least 
half-way through the sending of the stop bit.  Then there is a delay 
before the data gets sent back to the host PC, a delay through the 
kernel and drivers before it reaches the user program, time for the 
program to handle that message, time for it to prepare the next message, 
delays through the kernel and drivers before it gets to the USB bus, 
latency in the USB device that receives the USB message and then starts 
transmitting.  There can be no collision unless all that delay is less 
than half a bit time.  And no matter how fast your computer is, you are 
always going to need at least one full USB polling cycle for all this, 
which for USB 2.0 is 0.125 us.  That means that if you have a baud rate 
of 16 kbaud or higher, there is no possibility of a collision.


> 
> The tests are of two types, most of them are setting up a state and
> reading a signal.  This can go pretty fast and doesn't take too many
> commands.  Then there are the audio tests where the FPGA sends
> digital data to the UUT, which does it's thing and returns digital
> data which is crunched by the FPGA.  This takes some small number of
> seconds and presently the protocol is to poll the status until it is
> done.  That's a lot of messages, but it's not necessarily a slow
> point.  The same test can be started on every UUT in parallel, so the
> waiting is in parallel.  So maybe the serial port won't need to be
> any faster.
> 
> Still, I want to use RS-422 or RS-485 to deal with ground noise since
> this will be spread over multiple boards that don't have terribly
> solid grounds, just the power cable really.
> 
> I'm thinking out loud here as much as anything.  I intended to simply
> ask if anyone had experience with RS-485 that would be helpful.
> Running two wires rather than eight would be a help.  I'll probably
> use a 10 pin connector just to be on the safe side, allowing the
> transceivers to be used either way.
> 

You need to do some calculations to see if you can get enough telegrams 
and enough data through a single serial port.  You'll be hard pushed to 
find a USB serial port device of any kind that goes above 3 Mbaud, and 
you need to be careful about your selection of RS-485 drivers for those 
kinds of rates.  You will also find that much of your bandwidth is taken 
up with pauses between telegrams and reply latency, unless you make your 
telegrams quite large.

When we have made testbenches that required serial communication to 
multiple parallel devices, we typically put a USB hub in the testbench 
and use multiple FDTI USB to serial cables.  You only make one (or 
possibly a few) of the testbenches - it's much cheaper to use 
off-the-shelf parts than to spend time designing something more 
advanced.  You can buy a /lot/ of hubs and USB cables for the price of 
the time to design, build and program a custom card for the job.  It 
also makes the system more scalable, as the communication to different 
devices runs in parallel.

We have also done systems where there is a Raspberry Pi driving the hub 
and multiple FTDI converters.  The PC is connected to the Pi by Ethernet 
(useful for galvanic isolation), and the Pi runs forwarders between the 
serial ports and TCP/IP ports.

To be fair, I don't recall any testbenches we've made that needed more 
than perhaps 8 serial ports.  If I needed to handle 80 lines, I would 
probably split things up - a Pi handling 8-10 lines from a local 
program, communicating with a PC master program by Ethernet.

Il 02/11/2022 10:28, David Brown ha scritto:
> On 02/11/2022 06:28, Rick C wrote:

> RS-485 is easy from a PC using appropriate USB devices.  We make heavy 
> use of FTDI's chips and cables - they handle the drive enables for the 
> RS-485 automatically so that from the PC side you just read and write to 
> the serial port.
> 
> The reception of the last byte from a slave is not finished until the 
> stop bit has been properly received by the master - that means at least 
> half-way through the sending of the stop bit.  Then there is a delay 
> before the data gets sent back to the host PC, a delay through the 
> kernel and drivers before it reaches the user program, time for the 
> program to handle that message, time for it to prepare the next message, 
> delays through the kernel and drivers before it gets to the USB bus, 
> latency in the USB device that receives the USB message and then starts 
> transmitting.  There can be no collision unless all that delay is less 
> than half a bit time.  And no matter how fast your computer is, you are 
> always going to need at least one full USB polling cycle for all this, 
> which for USB 2.0 is 0.125 us.  That means that if you have a baud rate 
> of 16 kbaud or higher, there is no possibility of a collision.

However this depends on the speed of slave too, because it could be slow 
to move *its* direction from TX to RX. If the master starts transmitting 
after the stop bit from the slave, but *before* it changes *its* 
direction from TX to RX, the first bytes could be corrupted.

Unfortunately, not all UARTs in MCUs are able to drive automatically the 
DE (Drive Enable) signal, so it sometimes happens that DE is a normal GPIO.
If you are lucky, you have the TXC (transmit complete) interrupt that 
fires *after* stop bit is transmitted, a safe time to move DE signal.

In this case interrupt delay is short, but you could have other active 
interrupts that occasionally could delay the TXC interrupt for some time.

On 02/11/2022 11:54, pozz wrote:
> Il 02/11/2022 10:28, David Brown ha scritto:
>> On 02/11/2022 06:28, Rick C wrote:
> 
>> RS-485 is easy from a PC using appropriate USB devices.  We make heavy 
>> use of FTDI's chips and cables - they handle the drive enables for the 
>> RS-485 automatically so that from the PC side you just read and write 
>> to the serial port.
>>
>> The reception of the last byte from a slave is not finished until the 
>> stop bit has been properly received by the master - that means at 
>> least half-way through the sending of the stop bit.  Then there is a 
>> delay before the data gets sent back to the host PC, a delay through 
>> the kernel and drivers before it reaches the user program, time for 
>> the program to handle that message, time for it to prepare the next 
>> message, delays through the kernel and drivers before it gets to the 
>> USB bus, latency in the USB device that receives the USB message and 
>> then starts transmitting.  There can be no collision unless all that 
>> delay is less than half a bit time.  And no matter how fast your 
>> computer is, you are always going to need at least one full USB 
>> polling cycle for all this, which for USB 2.0 is 0.125 us.  That means 
>> that if you have a baud rate of 16 kbaud or higher, there is no 
>> possibility of a collision.
> 
> However this depends on the speed of slave too, because it could be slow 
> to move *its* direction from TX to RX. If the master starts transmitting 
> after the stop bit from the slave, but *before* it changes *its* 
> direction from TX to RX, the first bytes could be corrupted.

True, and it that is an important point.

> 
> Unfortunately, not all UARTs in MCUs are able to drive automatically the 
> DE (Drive Enable) signal, so it sometimes happens that DE is a normal GPIO.
> If you are lucky, you have the TXC (transmit complete) interrupt that 
> fires *after* stop bit is transmitted, a safe time to move DE signal.

I think the OP has FPGA's for the slave side of the equation, so there 
should not be a delay in switching their drivers off after the last byte 
is sent.  Even if it is a microcontroller and has no hardware control 
for the DE line, pretty much any half-decent microcontroller from this 
century has a TXC interrupt and can react and turn off the driver within 
a few microseconds.  I am assuming he is not trying to do this project 
using PIC's or 8051's !

> 
> In this case interrupt delay is short, but you could have other active 
> interrupts that occasionally could delay the TXC interrupt for some time.
> 

If this kind of thing is a risk, then it's not hard to put a short delay 
on the PC side between receiving a reply and sending out the next 
telegram.  But it's good that you brought it up, so that the OP can 
decide if it /is/ a risk.

On Wednesday, November 2, 2022 at 5:28:21 AM UTC-4, David Brown wrote:
> On 02/11/2022 06:28, Rick C wrote: 
> > I have a test fixture that uses RS-232 to communicate with a PC. It 
> > actually uses the voltage levels of RS-232, even though this is from 
> > a USB cable on the PC, so it's only RS-232 for maybe four inches. 
> > lol 
> > 
> > I'm redesigning the test fixtures to hold more units and fully 
> > automate a few features that presently requires an operator. There 
> > will now be 8 UUTs on each test fixture and I expect to have 10 to 20 
> > test fixtures in a card rack. That's 80 to 160 UUTs total. There 
> > will be an FPGA controlling each pair of UUTs, so 80 FPGAs in total 
> > that the PC needs to talk to. 
> > 
> > Rather than working on a way to mux 80 RS-232 interfaces, I'm 
> > thinking it would be better to either daisy chain, or connect in 
> > parallel all these devices. The protocol is master-slave where the 
> > master sends a command and the slaves are idle until they reply. The 
> > four FPGAs on a test fixture board could be connected in parallel 
> > easily enough. But I don't think I want to run TTL level signals 
> > between so many boards. 
> > 
> > I could do an RS-422 interface with a master to slave pair and a 
> > slave to master pair. The slaves do not speak until spoken to, so 
> > there will be no collisions. 
> >
> RS-422 is normally a point-to-point interface. It is one line in each 
> direction, but using balanced pairs instead of a TTL signal. You would 
> not normally connect multiple receivers or transmitters to an RS-422 
> bus, as the standard practice is that each transmitter is always driving 
> the pair it is attached to - there is no multi-drop.

That is simply not true.  Data sheets for RS-422 devices often show multidrop applications and how to best terminate them. 


> > RS-485 would allow all this to be over a single pair of wires. But 
> > the one big issue I see people complain about is getting PC software 
> > to not clobber the slaves, or I should say, to get the master to wait 
> > long enough that it's not clobbering it's own start bit by 
> > overwriting the stop bit of the slave. I suppose someone, somewhere 
> > has dealt with this on the PC and has a solution that doesn't impact 
> > bus speed. I run the single test fixture version of this at about 
> > 100 kbps. I'm going to want as much speed as I can get for 80 FPGAs 
> > controlling 160 UUTs. Maybe I should give that some analysis, 
> > because this might not be true.
> RS-485 is easy from a PC using appropriate USB devices. We make heavy 
> use of FTDI's chips and cables - they handle the drive enables for the 
> RS-485 automatically so that from the PC side you just read and write to 
> the serial port. 

I've yet to be convinced of this.  Admittedly, my last interaction with RS-485 was many years ago, but there were some four or five different devices being integrated with a PC, and no two of them handled it the bus the same way.  The PC in particular, would cut on and off the driver in the middle of bits.  No one put a bias on the bus, so it was indeterminate when no one was driving.  It was a horrible failure.  

Every time I've seen this discussed, the driver control has been an issue.  


> The reception of the last byte from a slave is not finished until the 
> stop bit has been properly received by the master - that means at least 
> half-way through the sending of the stop bit. 

That's not sufficient.  Everyone's halfway is a bit different and start bit detection may not be enabled on some device when the next driver outputs a start bit, or the last driver may not be turned off when the next driver starts.  


> Then there is a delay 
> before the data gets sent back to the host PC, a delay through the 
> kernel and drivers before it reaches the user program, time for the 
> program to handle that message, time for it to prepare the next message, 
> delays through the kernel and drivers before it gets to the USB bus, 
> latency in the USB device that receives the USB message and then starts 
> transmitting. There can be no collision unless all that delay is less 
> than half a bit time. And no matter how fast your computer is, you are 
> always going to need at least one full USB polling cycle for all this, 
> which for USB 2.0 is 0.125 us. That means that if you have a baud rate 
> of 16 kbaud or higher, there is no possibility of a collision.

If your numbers are accurate, that might be ok, but I'm looking for data rates closer to 1 Mbps.  Admittedly, I have not done an analysis of what will actually be required, but 128 UUT, or possibly 256, can do a lot of damage to a shared bus.  At 1 Mbps, 128 UUT results in an effective bit rate maximum of 7.8 kbps.  With 256 UUTs, that's 3.9 kbps.  No, I don't think this will work properly at much slower speeds than 1 Mbps.   At 16 kbps, the effective rate to each UUT is just 62.5 bps, not kbps. 


> > The tests are of two types, most of them are setting up a state and 
> > reading a signal. This can go pretty fast and doesn't take too many 
> > commands. Then there are the audio tests where the FPGA sends 
> > digital data to the UUT, which does it's thing and returns digital 
> > data which is crunched by the FPGA. This takes some small number of 
> > seconds and presently the protocol is to poll the status until it is 
> > done. That's a lot of messages, but it's not necessarily a slow 
> > point. The same test can be started on every UUT in parallel, so the 
> > waiting is in parallel. So maybe the serial port won't need to be 
> > any faster. 
> > 
> > Still, I want to use RS-422 or RS-485 to deal with ground noise since 
> > this will be spread over multiple boards that don't have terribly 
> > solid grounds, just the power cable really. 
> > 
> > I'm thinking out loud here as much as anything. I intended to simply 
> > ask if anyone had experience with RS-485 that would be helpful. 
> > Running two wires rather than eight would be a help. I'll probably 
> > use a 10 pin connector just to be on the safe side, allowing the 
> > transceivers to be used either way. 
> >
> You need to do some calculations to see if you can get enough telegrams 
> and enough data through a single serial port. You'll be hard pushed to 
> find a USB serial port device of any kind that goes above 3 Mbaud, and 
> you need to be careful about your selection of RS-485 drivers for those 
> kinds of rates. You will also find that much of your bandwidth is taken 
> up with pauses between telegrams and reply latency, unless you make your 
> telegrams quite large. 

I assume by "telegrams", you mean the messages.  They will be small by necessity.  The protocol is interactive with a command message and a reply message.  Read a register, write a register.  


> When we have made testbenches that required serial communication to 
> multiple parallel devices, we typically put a USB hub in the testbench 
> and use multiple FDTI USB to serial cables. You only make one (or 
> possibly a few) of the testbenches - it's much cheaper to use 
> off-the-shelf parts than to spend time designing something more 
> advanced. You can buy a /lot/ of hubs and USB cables for the price of 
> the time to design, build and program a custom card for the job. It 
> also makes the system more scalable, as the communication to different 
> devices runs in parallel. 

USB hubs are a last resort.  I've found many issues with such devices, especially larger than 4 ports. 


> We have also done systems where there is a Raspberry Pi driving the hub 
> and multiple FTDI converters. The PC is connected to the Pi by Ethernet 
> (useful for galvanic isolation), and the Pi runs forwarders between the 
> serial ports and TCP/IP ports. 

There is a possibility of using an rPi on an Ethernet cable to the PC with direct comms to each test fixture board, but that's more work that I'm interested in.  


> To be fair, I don't recall any testbenches we've made that needed more 
> than perhaps 8 serial ports. If I needed to handle 80 lines, I would 
> probably split things up - a Pi handling 8-10 lines from a local 
> program, communicating with a PC master program by Ethernet.

That's the advantage of the shared bus.  No programming required, other than extending the protocol to move from "selecting" a device on the FPGA, to selecting the FPGA as well. 

-- 

Rick C.

+ Get 1,000 miles of free Supercharging
+ Tesla referral code - https://ts.la/richard11209

On 02/11/2022 20:20, Rick C wrote:
> On Wednesday, November 2, 2022 at 5:28:21 AM UTC-4, David Brown
> wrote:
>> On 02/11/2022 06:28, Rick C wrote:
>>> I have a test fixture that uses RS-232 to communicate with a PC.
>>> It actually uses the voltage levels of RS-232, even though this
>>> is from a USB cable on the PC, so it's only RS-232 for maybe four
>>> inches. lol
>>> 
>>> I'm redesigning the test fixtures to hold more units and fully 
>>> automate a few features that presently requires an operator.
>>> There will now be 8 UUTs on each test fixture and I expect to
>>> have 10 to 20 test fixtures in a card rack. That's 80 to 160 UUTs
>>> total. There will be an FPGA controlling each pair of UUTs, so 80
>>> FPGAs in total that the PC needs to talk to.
>>> 
>>> Rather than working on a way to mux 80 RS-232 interfaces, I'm 
>>> thinking it would be better to either daisy chain, or connect in 
>>> parallel all these devices. The protocol is master-slave where
>>> the master sends a command and the slaves are idle until they
>>> reply. The four FPGAs on a test fixture board could be connected
>>> in parallel easily enough. But I don't think I want to run TTL
>>> level signals between so many boards.
>>> 
>>> I could do an RS-422 interface with a master to slave pair and a 
>>> slave to master pair. The slaves do not speak until spoken to,
>>> so there will be no collisions.
>>> 
>> RS-422 is normally a point-to-point interface. It is one line in
>> each direction, but using balanced pairs instead of a TTL signal.
>> You would not normally connect multiple receivers or transmitters
>> to an RS-422 bus, as the standard practice is that each transmitter
>> is always driving the pair it is attached to - there is no
>> multi-drop.
> 
> That is simply not true.  Data sheets for RS-422 devices often show
> multidrop applications and how to best terminate them.
> 

RS-422 is not multidrop.  Occasionally you will see multiple receivers 
on a bus, but not multiple transmitters.

Of course the same driver chips can be used in different combinations of 
wiring and drive enables.  An RS-422 driver chip can be viewed as two 
RS-485 driver chips - alternatively, a RS-485 driver can be viewed as an 
RS-422 driver with the two differential pairs connected together. 
Really, all you are talking about is a differential driver and a 
differential receiver.

So yes, you can do multidrop using an RS-422 driver chip.  But it is not 
RS-422, which is a point-to-point serial bus standard.

> 
>>> RS-485 would allow all this to be over a single pair of wires.
>>> But the one big issue I see people complain about is getting PC
>>> software to not clobber the slaves, or I should say, to get the
>>> master to wait long enough that it's not clobbering it's own
>>> start bit by overwriting the stop bit of the slave. I suppose
>>> someone, somewhere has dealt with this on the PC and has a
>>> solution that doesn't impact bus speed. I run the single test
>>> fixture version of this at about 100 kbps. I'm going to want as
>>> much speed as I can get for 80 FPGAs controlling 160 UUTs. Maybe
>>> I should give that some analysis, because this might not be
>>> true.
>> RS-485 is easy from a PC using appropriate USB devices. We make
>> heavy use of FTDI's chips and cables - they handle the drive
>> enables for the RS-485 automatically so that from the PC side you
>> just read and write to the serial port.
> 
> I've yet to be convinced of this.  Admittedly, my last interaction
> with RS-485 was many years ago, but there were some four or five
> different devices being integrated with a PC, and no two of them
> handled it the bus the same way.  The PC in particular, would cut on
> and off the driver in the middle of bits.  No one put a bias on the
> bus, so it was indeterminate when no one was driving.  It was a
> horrible failure.
> 
> Every time I've seen this discussed, the driver control has been an
> issue.

I can tell you it works perfectly with FTDI's RS-485 cables - every 
time, every OS, regardless of the software.  Some RS-485 drivers rely on 
RTS for the drive enable - this was the standard for RS-232 to RS-485 
converters from the old days of 9-pin and 25-pin serial ports on PC's. 
With such drivers, it is certainly possible to get things wrong.  With 
the drive enable handled directly by the UART hardware on the USB chip, 
it is /far/ harder to get it wrong.

I would expect there to be many alternatives to FTDI that work similarly 
well, but that's the ones we generally use.

<https://ftdichip.com/product-category/products/cables/?series_products=55>

> 
>> The reception of the last byte from a slave is not finished until
>> the stop bit has been properly received by the master - that means
>> at least half-way through the sending of the stop bit.
> 
> That's not sufficient.  Everyone's halfway is a bit different and
> start bit detection may not be enabled on some device when the next
> driver outputs a start bit, or the last driver may not be turned off
> when the next driver starts.
> 

"At least half-way" means "at least 50% of the bit time".  As long as 
the start bit from the next message is not sent until at least 50% of a 
bit time after the stop bit is detected, it will not conflict and all 
listening devices will be ready to see the start bit.  (Devices that 
needed two stop bits haven't existed in the last 50 years.)

You asked specifically about bus turnaround at the host side - I assume 
that is because on the slave devices, you have control of the drive 
enables and bus turnaround happens with negligible latency.

> 
>> Then there is a delay before the data gets sent back to the host
>> PC, a delay through the kernel and drivers before it reaches the
>> user program, time for the program to handle that message, time for
>> it to prepare the next message, delays through the kernel and
>> drivers before it gets to the USB bus, latency in the USB device
>> that receives the USB message and then starts transmitting. There
>> can be no collision unless all that delay is less than half a bit
>> time. And no matter how fast your computer is, you are always going
>> to need at least one full USB polling cycle for all this, which for
>> USB 2.0 is 0.125 us. That means that if you have a baud rate of 16
>> kbaud or higher, there is no possibility of a collision.
> 
> If your numbers are accurate, that might be ok, but I'm looking for
> data rates closer to 1 Mbps.

USB serial ports generally use the 48 MHz base USB reference frequency 
as their source clock to scale down by a baud rate divisor, and common 
practice is 16 sub-bit clocks per line bit (so that you can have 
multiple samples for noise immunity).  Thus baud rates of integer 
divisions of 3 MBaud are common.  Certainly the FTDI chips handle 1, 2 
and 3 MBaud.  (I haven't had need of such speeds with RS-485, but have 
happily used the common 3v3 TTL cables at 3 MBaud.)

>  Admittedly, I have not done an analysis
> of what will actually be required, but 128 UUT, or possibly 256, can
> do a lot of damage to a shared bus.  At 1 Mbps, 128 UUT results in an
> effective bit rate maximum of 7.8 kbps.  With 256 UUTs, that's 3.9
> kbps.  No, I don't think this will work properly at much slower
> speeds than 1 Mbps.   At 16 kbps, the effective rate to each UUT is
> just 62.5 bps, not kbps.
> 

As long as you are /above/ 16 kbaud, you should be fine (at the PC 
side).  At 1 Mbaud, you do not need to worry about the PC starting a new 
telegram before the last received stop bit is completed.

> 
>>> The tests are of two types, most of them are setting up a state
>>> and reading a signal. This can go pretty fast and doesn't take
>>> too many commands. Then there are the audio tests where the FPGA
>>> sends digital data to the UUT, which does it's thing and returns
>>> digital data which is crunched by the FPGA. This takes some small
>>> number of seconds and presently the protocol is to poll the
>>> status until it is done. That's a lot of messages, but it's not
>>> necessarily a slow point. The same test can be started on every
>>> UUT in parallel, so the waiting is in parallel. So maybe the
>>> serial port won't need to be any faster.
>>> 
>>> Still, I want to use RS-422 or RS-485 to deal with ground noise
>>> since this will be spread over multiple boards that don't have
>>> terribly solid grounds, just the power cable really.
>>> 
>>> I'm thinking out loud here as much as anything. I intended to
>>> simply ask if anyone had experience with RS-485 that would be
>>> helpful. Running two wires rather than eight would be a help.
>>> I'll probably use a 10 pin connector just to be on the safe side,
>>> allowing the transceivers to be used either way.
>>> 
>> You need to do some calculations to see if you can get enough
>> telegrams and enough data through a single serial port. You'll be
>> hard pushed to find a USB serial port device of any kind that goes
>> above 3 Mbaud, and you need to be careful about your selection of
>> RS-485 drivers for those kinds of rates. You will also find that
>> much of your bandwidth is taken up with pauses between telegrams
>> and reply latency, unless you make your telegrams quite large.
> 
> I assume by "telegrams", you mean the messages.  They will be small
> by necessity.  The protocol is interactive with a command message and
> a reply message.  Read a register, write a register.
> 

Telegram, message, packet - whatever term you prefer.  At faster baud 
rates, the inevitable pauses between messages take proportionately more 
of the total bandwidth.  Longer messages will be more efficient.  But 
you'll have to do the sums yourself to see what rates you need, and 
whether or not this will be an issue.

> 
>> When we have made testbenches that required serial communication
>> to multiple parallel devices, we typically put a USB hub in the
>> testbench and use multiple FDTI USB to serial cables. You only make
>> one (or possibly a few) of the testbenches - it's much cheaper to
>> use off-the-shelf parts than to spend time designing something
>> more advanced. You can buy a /lot/ of hubs and USB cables for the
>> price of the time to design, build and program a custom card for
>> the job. It also makes the system more scalable, as the
>> communication to different devices runs in parallel.
> 
> USB hubs are a last resort.  I've found many issues with such
> devices, especially larger than 4 ports.
> 

We find they work fine - I have very rarely seen any issues with 
off-the-shelf hubs, regardless of the number of ports.  (They are almost 
all made with 1-to-4 hub chips, which is why hubs are often found in 
sizes of 4 ports, 7 ports, or 10 ports.)

A key complication with multiple serial ports on hubs is if you are 
using Windows, it can be a big pain to keep consistent numbering for the 
serial ports.  You may have to use driver-specific libraries (like 
FTDI's DLL's) to check serial numbers and use that information.  It's 
far easier on Linux where you can make a udev configuration file that 
gives aliases to your ports ordered by physical tree address.

> 
>> We have also done systems where there is a Raspberry Pi driving the
>> hub and multiple FTDI converters. The PC is connected to the Pi by
>> Ethernet (useful for galvanic isolation), and the Pi runs
>> forwarders between the serial ports and TCP/IP ports.
> 
> There is a possibility of using an rPi on an Ethernet cable to the PC
> with direct comms to each test fixture board, but that's more work
> that I'm interested in.
> 

Or you could use one Pi for a set of boards - whatever is physically 
convenient.

> 
>> To be fair, I don't recall any testbenches we've made that needed
>> more than perhaps 8 serial ports. If I needed to handle 80 lines, I
>> would probably split things up - a Pi handling 8-10 lines from a
>> local program, communicating with a PC master program by Ethernet.
> 
> That's the advantage of the shared bus.  No programming required,
> other than extending the protocol to move from "selecting" a device
> on the FPGA, to selecting the FPGA as well.
> 

If you are familiar with socat, the Pi doesn't necessarily need any 
programming either.  (In our case we wanted some extra monitoring and 
logging, which was more than we could get from socat - so it was a 
couple of hundred lines of Python in the end.)

Rick C <gnuarm.deletethisbit@gmail.com> writes:
> That's 80 to 160 UUTs total.  There will be an FPGA controlling each
> pair of UUTs, so 80 FPGAs in total that the PC needs to talk to.

Rather than make a huge shared bus, I wonder if you could move the test
controller from a PC to a small microprocessor board that controls two
FPGA's or whatever.  Then use a bunch of separate boards of that type,
communicating with a PC using some method that doesn't have to be
especially fast.  The microprocessor board could be something like a
Raspberry Pi Pico, which costs $4 and can run Mecrisp, if that is what
your software is written with.  It is quite a powerful little board.

On Wednesday, November 2, 2022 at 4:49:16 PM UTC-4, David Brown wrote:
> On 02/11/2022 20:20, Rick C wrote: 
> > On Wednesday, November 2, 2022 at 5:28:21 AM UTC-4, David Brown 
> > wrote: 
> >> On 02/11/2022 06:28, Rick C wrote: 
> >>> I have a test fixture that uses RS-232 to communicate with a PC. 
> >>> It actually uses the voltage levels of RS-232, even though this 
> >>> is from a USB cable on the PC, so it's only RS-232 for maybe four 
> >>> inches. lol 
> >>> 
> >>> I'm redesigning the test fixtures to hold more units and fully 
> >>> automate a few features that presently requires an operator. 
> >>> There will now be 8 UUTs on each test fixture and I expect to 
> >>> have 10 to 20 test fixtures in a card rack. That's 80 to 160 UUTs 
> >>> total. There will be an FPGA controlling each pair of UUTs, so 80 
> >>> FPGAs in total that the PC needs to talk to. 
> >>> 
> >>> Rather than working on a way to mux 80 RS-232 interfaces, I'm 
> >>> thinking it would be better to either daisy chain, or connect in 
> >>> parallel all these devices. The protocol is master-slave where 
> >>> the master sends a command and the slaves are idle until they 
> >>> reply. The four FPGAs on a test fixture board could be connected 
> >>> in parallel easily enough. But I don't think I want to run TTL 
> >>> level signals between so many boards. 
> >>> 
> >>> I could do an RS-422 interface with a master to slave pair and a 
> >>> slave to master pair. The slaves do not speak until spoken to, 
> >>> so there will be no collisions. 
> >>> 
> >> RS-422 is normally a point-to-point interface. It is one line in 
> >> each direction, but using balanced pairs instead of a TTL signal. 
> >> You would not normally connect multiple receivers or transmitters 
> >> to an RS-422 bus, as the standard practice is that each transmitter 
> >> is always driving the pair it is attached to - there is no 
> >> multi-drop. 
> > 
> > That is simply not true. Data sheets for RS-422 devices often show 
> > multidrop applications and how to best terminate them. 
> >
> RS-422 is not multidrop. Occasionally you will see multiple receivers 
> on a bus, but not multiple transmitters. 

Not sure of your point.  Multi-drop is multiple receivers on a single transmitter.  Multi-point is multiple drivers and receivers.  Look at a few references.  Even wikipedia says, "RS-422 provides for data transmission, using balanced, or differential, signaling, with unidirectional/non-reversible, terminated or non-terminated transmission lines, point to point, or multi-drop. In contrast to EIA-485, RS-422/V.11 does not allow multiple drivers but only multiple receivers."  


> Of course the same driver chips can be used in different combinations of 
> wiring and drive enables. An RS-422 driver chip can be viewed as two 
> RS-485 driver chips - alternatively, a RS-485 driver can be viewed as an 
> RS-422 driver with the two differential pairs connected together. 
> Really, all you are talking about is a differential driver and a 
> differential receiver. 

Sure, but the point is, nothing in RS-422 precludes multiple receivers, and in fact, every reference I've found (not paying for the actual spec) shows multi-drop receivers. 


> So yes, you can do multidrop using an RS-422 driver chip. But it is not 
> RS-422, which is a point-to-point serial bus standard.

I don't believe that is correct.  If you have a copy of the spec to share, I'd love to look at it.  I might have one myself, but it would be a paper copy somewhere unknown.  The diagrams showing multi-drop RS-422 is so ubiquitous, I expect they are from the standard itself. 


> >>> RS-485 would allow all this to be over a single pair of wires. 
> >>> But the one big issue I see people complain about is getting PC 
> >>> software to not clobber the slaves, or I should say, to get the 
> >>> master to wait long enough that it's not clobbering it's own 
> >>> start bit by overwriting the stop bit of the slave. I suppose 
> >>> someone, somewhere has dealt with this on the PC and has a 
> >>> solution that doesn't impact bus speed. I run the single test 
> >>> fixture version of this at about 100 kbps. I'm going to want as 
> >>> much speed as I can get for 80 FPGAs controlling 160 UUTs. Maybe 
> >>> I should give that some analysis, because this might not be 
> >>> true. 
> >> RS-485 is easy from a PC using appropriate USB devices. We make 
> >> heavy use of FTDI's chips and cables - they handle the drive 
> >> enables for the RS-485 automatically so that from the PC side you 
> >> just read and write to the serial port. 
> > 
> > I've yet to be convinced of this. Admittedly, my last interaction 
> > with RS-485 was many years ago, but there were some four or five 
> > different devices being integrated with a PC, and no two of them 
> > handled it the bus the same way. The PC in particular, would cut on 
> > and off the driver in the middle of bits. No one put a bias on the 
> > bus, so it was indeterminate when no one was driving. It was a 
> > horrible failure. 
> > 
> > Every time I've seen this discussed, the driver control has been an 
> > issue.
> I can tell you it works perfectly with FTDI's RS-485 cables - every 
> time, every OS, regardless of the software. Some RS-485 drivers rely on 
> RTS for the drive enable - this was the standard for RS-232 to RS-485 
> converters from the old days of 9-pin and 25-pin serial ports on PC's. 
> With such drivers, it is certainly possible to get things wrong. With 
> the drive enable handled directly by the UART hardware on the USB chip, 
> it is /far/ harder to get it wrong. 

So, what range of speeds have you used?  It is actually the UART hardware that *does* get it very wrong by working off the transmitter empty signal, which changes in the middle of the bit.  The control has to be specially designed to transition at the *end* of the stop bit of the transmitted character.  Knowing when it is ok to enable the driver has the same problem.  The data is "received" in the middle of the stop bit.  So the enable has to be a half bit time later, at the end of the stop bit. 

None of this matters to me really.  I'm going to use more wires, and do the multi-drop from the PC to the slaves on one pair and use RS-422 to multi-point from the slaves to the PC.  Since the slaves are controlled by the master, they will never collide.  The master can't collide with itself, so I can ignore any issues with this.  I will use the bias resistors to assure a valid idle state.  I may need to select different devices than the ones I use in the product.  I think there are differences in the input load and I want to be sure I can chain up to 32 units. 


> I would expect there to be many alternatives to FTDI that work similarly 
> well, but that's the ones we generally use. 
> 
> <https://ftdichip.com/product-category/products/cables/?series_products=55>
> > 
> >> The reception of the last byte from a slave is not finished until 
> >> the stop bit has been properly received by the master - that means 
> >> at least half-way through the sending of the stop bit. 
> > 
> > That's not sufficient. Everyone's halfway is a bit different and 
> > start bit detection may not be enabled on some device when the next 
> > driver outputs a start bit, or the last driver may not be turned off 
> > when the next driver starts. 
> >
> "At least half-way" means "at least 50% of the bit time". As long as 
> the start bit from the next message is not sent until at least 50% of a 
> bit time after the stop bit is detected, it will not conflict and all 
> listening devices will be ready to see the start bit. (Devices that 
> needed two stop bits haven't existed in the last 50 years.) 

You don't seem to understand that there is nothing timing from the start of the bit.  The timing is from the first detected low of the start bit.  From there, all timing is done by an internal clock.  Check the math, you don't get 50% of the stop bit, guaranteed.  That's why they call it "asynchronous" serial.  


> You asked specifically about bus turnaround at the host side - I assume 
> that is because on the slave devices, you have control of the drive 
> enables and bus turnaround happens with negligible latency.

I know the master has the most trouble with this.  The slaves tend to not have a problem because they are operated by MCUs and can wait a bit time before replying, or even a character time.  I suppose they don't have any magic on turning off the driver though, but early is the easy way and generally doesn't cause a problem.  The master has trouble on both ends of it's message, needing to be careful to not turn on the driver too soon and not turning it off too late to clobber the reply. 


> >> Then there is a delay before the data gets sent back to the host 
> >> PC, a delay through the kernel and drivers before it reaches the 
> >> user program, time for the program to handle that message, time for 
> >> it to prepare the next message, delays through the kernel and 
> >> drivers before it gets to the USB bus, latency in the USB device 
> >> that receives the USB message and then starts transmitting. There 
> >> can be no collision unless all that delay is less than half a bit 
> >> time. And no matter how fast your computer is, you are always going 
> >> to need at least one full USB polling cycle for all this, which for 
> >> USB 2.0 is 0.125 us. That means that if you have a baud rate of 16 
> >> kbaud or higher, there is no possibility of a collision. 
> > 
> > If your numbers are accurate, that might be ok, but I'm looking for 
> > data rates closer to 1 Mbps.
> USB serial ports generally use the 48 MHz base USB reference frequency 
> as their source clock to scale down by a baud rate divisor, and common 
> practice is 16 sub-bit clocks per line bit (so that you can have 
> multiple samples for noise immunity). Thus baud rates of integer 
> divisions of 3 MBaud are common. Certainly the FTDI chips handle 1, 2 
> and 3 MBaud. (I haven't had need of such speeds with RS-485, but have 
> happily used the common 3v3 TTL cables at 3 MBaud.)

At some point you have to worry with the line waveforms.  So too fast can cause problems when using *lots* of receivers. 


> > Admittedly, I have not done an analysis 
> > of what will actually be required, but 128 UUT, or possibly 256, can 
> > do a lot of damage to a shared bus. At 1 Mbps, 128 UUT results in an 
> > effective bit rate maximum of 7.8 kbps. With 256 UUTs, that's 3.9 
> > kbps. No, I don't think this will work properly at much slower 
> > speeds than 1 Mbps. At 16 kbps, the effective rate to each UUT is 
> > just 62.5 bps, not kbps. 
> >
> As long as you are /above/ 16 kbaud, you should be fine (at the PC 
> side). At 1 Mbaud, you do not need to worry about the PC starting a new 
> telegram before the last received stop bit is completed.

Not entirely.  The master has to turn *off* the driver before the slave replies.  At higher speeds that's a problem.  But it all depends on how it is being done.  This is why I'm going with two busses, one for master transmit and one for master input. 


> >>> The tests are of two types, most of them are setting up a state 
> >>> and reading a signal. This can go pretty fast and doesn't take 
> >>> too many commands. Then there are the audio tests where the FPGA 
> >>> sends digital data to the UUT, which does it's thing and returns 
> >>> digital data which is crunched by the FPGA. This takes some small 
> >>> number of seconds and presently the protocol is to poll the 
> >>> status until it is done. That's a lot of messages, but it's not 
> >>> necessarily a slow point. The same test can be started on every 
> >>> UUT in parallel, so the waiting is in parallel. So maybe the 
> >>> serial port won't need to be any faster. 
> >>> 
> >>> Still, I want to use RS-422 or RS-485 to deal with ground noise 
> >>> since this will be spread over multiple boards that don't have 
> >>> terribly solid grounds, just the power cable really. 
> >>> 
> >>> I'm thinking out loud here as much as anything. I intended to 
> >>> simply ask if anyone had experience with RS-485 that would be 
> >>> helpful. Running two wires rather than eight would be a help. 
> >>> I'll probably use a 10 pin connector just to be on the safe side, 
> >>> allowing the transceivers to be used either way. 
> >>> 
> >> You need to do some calculations to see if you can get enough 
> >> telegrams and enough data through a single serial port. You'll be 
> >> hard pushed to find a USB serial port device of any kind that goes 
> >> above 3 Mbaud, and you need to be careful about your selection of 
> >> RS-485 drivers for those kinds of rates. You will also find that 
> >> much of your bandwidth is taken up with pauses between telegrams 
> >> and reply latency, unless you make your telegrams quite large. 
> > 
> > I assume by "telegrams", you mean the messages. They will be small 
> > by necessity. The protocol is interactive with a command message and 
> > a reply message. Read a register, write a register. 
> >
> Telegram, message, packet - whatever term you prefer. At faster baud 
> rates, the inevitable pauses between messages take proportionately more 
> of the total bandwidth. Longer messages will be more efficient. But 
> you'll have to do the sums yourself to see what rates you need, and 
> whether or not this will be an issue.

Not sure what delays you are talking about.  Every message is either selecting a slave, or reading a register or writing a register.  You are probably thinking like a code banger where you have to worry with software delays.  The protocol at the interface to the UUT is serial, at 30 MHz and the turn around is so quick, I had to use a mux to select the first bit and shift the rest of the data from a shift register.  All in all it is under a &mu;s for a transfer, and the data can be sent to the PC in ASCII Hex format which even at 1 Mbps is much slower.  I can't say what the delays on the PC are.  It's never been of interest, but I can't imagine it's much.  


> >> When we have made testbenches that required serial communication 
> >> to multiple parallel devices, we typically put a USB hub in the 
> >> testbench and use multiple FDTI USB to serial cables. You only make 
> >> one (or possibly a few) of the testbenches - it's much cheaper to 
> >> use off-the-shelf parts than to spend time designing something 
> >> more advanced. You can buy a /lot/ of hubs and USB cables for the 
> >> price of the time to design, build and program a custom card for 
> >> the job. It also makes the system more scalable, as the 
> >> communication to different devices runs in parallel. 
> > 
> > USB hubs are a last resort. I've found many issues with such 
> > devices, especially larger than 4 ports. 
> >
> We find they work fine - I have very rarely seen any issues with 
> off-the-shelf hubs, regardless of the number of ports. (They are almost 
> all made with 1-to-4 hub chips, which is why hubs are often found in 
> sizes of 4 ports, 7 ports, or 10 ports.) 

Exactly, and I find combining them like that has issues. 


> A key complication with multiple serial ports on hubs is if you are 
> using Windows, it can be a big pain to keep consistent numbering for the 
> serial ports. You may have to use driver-specific libraries (like 
> FTDI's DLL's) to check serial numbers and use that information. It's 
> far easier on Linux where you can make a udev configuration file that 
> gives aliases to your ports ordered by physical tree address.

Yet another reason to avoid such complications.  The reality is there's no gain.  The multi-drop is the right way to go here. 


> >> We have also done systems where there is a Raspberry Pi driving the 
> >> hub and multiple FTDI converters. The PC is connected to the Pi by 
> >> Ethernet (useful for galvanic isolation), and the Pi runs 
> >> forwarders between the serial ports and TCP/IP ports. 
> > 
> > There is a possibility of using an rPi on an Ethernet cable to the PC 
> > with direct comms to each test fixture board, but that's more work 
> > that I'm interested in. 
> >
> Or you could use one Pi for a set of boards - whatever is physically 
> convenient.

But it's yet another piece to keep working.  Much easier to just use the multi-drop.  I will keep that idea as a backup plan.  But getting RS-422 on an rPi is a hassle.  That would need to be a hat, or a shield or whatever they call daughter cards on rPis.  Last time I checked, it was hard to find rPis.  They are part of the unobtainium universe now, it seems. 


> >> To be fair, I don't recall any testbenches we've made that needed 
> >> more than perhaps 8 serial ports. If I needed to handle 80 lines, I 
> >> would probably split things up - a Pi handling 8-10 lines from a 
> >> local program, communicating with a PC master program by Ethernet. 
> > 
> > That's the advantage of the shared bus. No programming required, 
> > other than extending the protocol to move from "selecting" a device 
> > on the FPGA, to selecting the FPGA as well. 
> >
> If you are familiar with socat, the Pi doesn't necessarily need any 
> programming either. (In our case we wanted some extra monitoring and 
> logging, which was more than we could get from socat - so it was a 
> couple of hundred lines of Python in the end.)

A couple hundred lines I'd rather not write. 

Thanks for the comments. 

-- 

Rick C.

-- Get 1,000 miles of free Supercharging
-- Tesla referral code - https://ts.la/richard11209

On Wednesday, November 2, 2022 at 6:01:02 PM UTC-4, Paul Rubin wrote:
> Rick C <gnuarm.del...@gmail.com> writes: 
> > That's 80 to 160 UUTs total. There will be an FPGA controlling each 
> > pair of UUTs, so 80 FPGAs in total that the PC needs to talk to.
> Rather than make a huge shared bus, I wonder if you could move the test 
> controller from a PC to a small microprocessor board that controls two 
> FPGA's or whatever. Then use a bunch of separate boards of that type, 
> communicating with a PC using some method that doesn't have to be 
> especially fast. The microprocessor board could be something like a 
> Raspberry Pi Pico, which costs $4 and can run Mecrisp, if that is what 
> your software is written with. It is quite a powerful little board.

I'm lost.  How is this any better?  The data collection is running on the PC, so there still has to be communications.  If I want to make changes to the test program, it's one place, not 32 places, or 128 places.  All they would be doing is acting as a serial port concentrator.  They went out 40 years ago! 

-- 

Rick C.

-+ Get 1,000 miles of free Supercharging
-+ Tesla referral code - https://ts.la/richard11209

Rick C <gnuarm.deletethisbit@gmail.com> writes:
> I'm lost.  How is this any better?  The data collection is running on
> the PC, so there still has to be communications... All they would be
> doing is acting as a serial port concentrator.  They went out 40 years
> ago!

Well, I always like doing stuff in software instead of hardware.  Serial
port concentrators are still around and I posted a link to one in your
other thread.  It seems like a reasonable approach too.  Oddly, a quick
web search doesn't find any big cheap serial to ethernet ones, but maybe
you could use USB hubs and FTDI-like cables.

The idea of using an MCU is to move almost everything speed critical
away from the PC.  The MCU only has control the FPGA's and transfer
transfer digested data upwards to the PC.  By having some fanout you
could potentially control 1000s of UUT instead of 80 from a single PC,
without having to build anything special.  It would just mean plugging
some off the shelf boxes together, and writing some code.

You're more comfortable with hardware than I am, so maybe that approach
has less attraction for you than it does for me.