Forums

Udev-based serial number dependent symlinks for PCIe UIO devices

Started by Wojciech Zabolotny May 18, 2021
I'm working at a system, where a server manages multiple PCIe data acquisition cards. Each card has its individual serial number available in the extended PCIe configuration space. The cards are controlled as UIO devices.
Unfortunately, it appears that the card are sometimes enumerated in the random order. Proper management of the system requires that there are symlinks created with names permanently associated with the individual cards.
Of course, I wanted to use udev for that purpose. Unfortunately, udev does not provide the Device Serial Number capability between the device attributes.
The DSN may be read with "lspci -vvv":

[...]
	Capabilities: [100 v1] Device Serial Number 12-34-56-78-90-ab-cd-ef
[...]

My solution consists of two parts.
1. The udev rule (file: /etc/udev/rules.d/30-daq.rules)
===========
SUBSYSTEM=="uio" PROGRAM="/opt/bin/uio_namer" SYMLINK="%c"
===========
2. The program that is run for each found uio device and:
   a. Checks if it is our board (end returns non-zero status if it is not)
   b. Outputs the name of the desired symlink to the standard output
        and returns 0 status when it is our device.
        I have implemented it fully in bash. Of course it can be implemented
        in C using the libpci, or in Python.
        The program receives the information from udev in the environment 
        variables. The most important is DEVPATH.

(file: /opt/bin/uio_namer)
=======================
#!/bin/sh
DEVID=abba:3342
SLOTNAME=`basename \`readlink /sys/${DEVPATH}/device\``
DLSPCI=`lspci -vvv -s ${SLOTNAME}`
#Check if it is our device
if [ -z  "`echo ${DLSPCI} | grep Device | grep ${DEVID}`" ] ; then
  # It is not our device
  exit 1
fi
DSNCAP=`lspci -vvv -s ${SLOTNAME} | grep "Device Serial Number" `
echo daq/${DSNCAP##*" "}
======================== 

For recognized boards, the device files are created in the /dev/daq directory as the board's serial numbers. For example: /dev/daq/12-34-56-78-90-ab-cd-ef

I hope that the above solution may be useful for others as well.
With best regards,
Wojtek
On 5/18/2021 2:17 PM, Wojciech Zabolotny wrote:
> I'm working at a system, where a server manages multiple PCIe data > acquisition cards. Each card has its individual serial number available in > the extended PCIe configuration space. The cards are controlled as UIO > devices. Unfortunately, it appears that the card are sometimes enumerated in > the random order. Proper management of the system requires that there are > symlinks created with names permanently associated with the individual > cards.
[I don't run Linux, so...] Can't you adjust the kernel configuration so that it forces certain probes to yield certain "devices"? E.g., instead of letting NetBSD probe disk drives in whatever order it encounters them -- and assigning device names in that order -- I build my kernels so certain device names are set aside for specific devices (which may or may not be present at any given boot). So, the first "free" probed device might be /dev/xx4 instead of /dev/xx0 -- because I want to ensure /dev/xx0 is available for a *specific* device instead of letting it go to the "first one found".
Don Y <blockedofcourse@foo.invalid> wrote:
> Can't you adjust the kernel configuration so that it forces > certain probes to yield certain "devices"? > > E.g., instead of letting NetBSD probe disk drives in whatever order > it encounters them -- and assigning device names in that order -- I > build my kernels so certain device names are set aside for specific > devices (which may or may not be present at any given boot).
That's what udev does. udev has a set of rules that indicate how devices should be turned into device nodes, be it by physical location (PCI slot), vendor/product ID, serial number, etc. It also sets ownership and permissions of those device nodes (so only certain users can access a device node). It handles creating and removing device nodes as devices are hot plugged. It's all just some textual config files, no kernel compilation needed. The problem in this case is that PCI devices don't as standard support a serial number in the way that USB or SATA does, and so this manufacturer has done their own thing in the way the serial number is embedded. That is why some helper code is needed to expose the serial number to udev to use in its device naming. (although I would have expected a rule that sets up device nodes named by PCI slot to be equally predictable) Theo
On 5/19/2021 1:30 AM, Theo wrote:
> Don Y <blockedofcourse@foo.invalid> wrote: >> Can't you adjust the kernel configuration so that it forces >> certain probes to yield certain "devices"? >> >> E.g., instead of letting NetBSD probe disk drives in whatever order >> it encounters them -- and assigning device names in that order -- I >> build my kernels so certain device names are set aside for specific >> devices (which may or may not be present at any given boot). > > That's what udev does. udev has a set of rules that indicate how devices > should be turned into device nodes, be it by physical location (PCI slot), > vendor/product ID, serial number, etc. It also sets ownership and > permissions of those device nodes (so only certain users can access a > device node).
OK. In NetBSD, that's part of the kernel configuration. Things like permissions and device types (char vs block) are handled in reasonably static files: --------- device-major ccd char 300 block 300 ccd device-major vnd char 301 block 301 vnd device-major md char 302 block 302 md device-major ld char 303 block 303 ld device-major raid char 304 block 304 raid device-major cgd char 305 block 305 cgd # scsibus and its children device-major scsibus char 310 scsibus device-major sd char 311 block 311 sd device-major st char 312 block 312 st device-major cd char 313 block 313 cd device-major ch char 314 ch device-major ss char 315 ss device-major uk char 316 uk device-major ses char 317 ses device-major se char 318 se device-major fd char 320 block 320 fdc ... device-major com char 260 com device-major lpt char 261 lpt ... ------- while the binding of device identifiers is part of the "config" file: ------ # SCSI devices sd* at scsibus? target ? lun ? # SCSI disk drives st* at scsibus? target ? lun ? # SCSI tape drives cd* at scsibus? target ? lun ? # SCSI CD-ROM drives ch* at scsibus? target ? lun ? # SCSI autochangers ses* at scsibus? target ? lun ? # SCSI Enclosure Services devices ss* at scsibus? target ? lun ? # SCSI scanners uk* at scsibus? target ? lun ? # SCSI unknown ---- E.g., in the above, I could ensure sd3 was always a particular device on a particular scsibus at a particular SCSI ID by creating an entry with the various wildcards (? *) replaced by specific values. The probe will preallocate those so they aren't assigned to some other device that happened to be Nth in order of detection.
> It handles creating and removing device nodes as devices are > hot plugged. It's all just some textual config files, no kernel compilation > needed. > > The problem in this case is that PCI devices don't as standard support a > serial number in the way that USB or SATA does, and so this manufacturer has > done their own thing in the way the serial number is embedded. That is why > some helper code is needed to expose the serial number to udev to use in its > device naming.
Ah. I'd have thought an alternative would just be to ensure devices of that specific *type* (manufacturer) were probed in an order that could be enforced by the equivalent of the scheme I mentioned above.
> (although I would have expected a rule that sets up device nodes named by > PCI slot to be equally predictable)
I guess one unanswered question is whether Wojciech is bolting his solution onto an existing system *or* designing a system from scratch (in which case, you would assume he could exercise some control over how the hardware is "built"/pieced together)
&#347;roda, 19 maja 2021 o&nbsp;10:52:06 UTC+2 Don Y napisa&#322;(a):
> On 5/19/2021 1:30 AM, Theo wrote: > > Don Y <blocked...@foo.invalid> wrote: > >> Can't you adjust the kernel configuration so that it forces > >> certain probes to yield certain "devices"? > >> > >> E.g., instead of letting NetBSD probe disk drives in whatever order > >> it encounters them -- and assigning device names in that order -- I > >> build my kernels so certain device names are set aside for specific > >> devices (which may or may not be present at any given boot). > > > > That's what udev does. udev has a set of rules that indicate how devices > > should be turned into device nodes, be it by physical location (PCI slot), > > vendor/product ID, serial number, etc. It also sets ownership and > > permissions of those device nodes (so only certain users can access a > > device node). > OK. In NetBSD, that's part of the kernel configuration. > Things like permissions and device types (char vs block) > are handled in reasonably static files: > > --------- > device-major ccd char 300 block 300 ccd > device-major vnd char 301 block 301 vnd > device-major md char 302 block 302 md > device-major ld char 303 block 303 ld > device-major raid char 304 block 304 raid > device-major cgd char 305 block 305 cgd > > # scsibus and its children > device-major scsibus char 310 scsibus > device-major sd char 311 block 311 sd > device-major st char 312 block 312 st > device-major cd char 313 block 313 cd > device-major ch char 314 ch > device-major ss char 315 ss > device-major uk char 316 uk > device-major ses char 317 ses > device-major se char 318 se > > device-major fd char 320 block 320 fdc > ... > device-major com char 260 com > device-major lpt char 261 lpt > ... > ------- > > while the binding of device identifiers is part of the "config" > file: > > ------ > # SCSI devices > sd* at scsibus? target ? lun ? # SCSI disk drives > st* at scsibus? target ? lun ? # SCSI tape drives > cd* at scsibus? target ? lun ? # SCSI CD-ROM drives > ch* at scsibus? target ? lun ? # SCSI autochangers > ses* at scsibus? target ? lun ? # SCSI Enclosure Services devices > ss* at scsibus? target ? lun ? # SCSI scanners > uk* at scsibus? target ? lun ? # SCSI unknown > ---- > > E.g., in the above, I could ensure sd3 was always a particular > device on a particular scsibus at a particular SCSI ID by > creating an entry with the various wildcards (? *) replaced > by specific values. The probe will preallocate those so > they aren't assigned to some other device that happened > to be Nth in order of detection. > > It handles creating and removing device nodes as devices are > > hot plugged. It's all just some textual config files, no kernel compilation > > needed. > > > > The problem in this case is that PCI devices don't as standard support a > > serial number in the way that USB or SATA does, and so this manufacturer has > > done their own thing in the way the serial number is embedded. That is why > > some helper code is needed to expose the serial number to udev to use in its > > device naming. > Ah. I'd have thought an alternative would just be to ensure devices > of that specific *type* (manufacturer) were probed in an order that > could be enforced by the equivalent of the scheme I mentioned above. > > (although I would have expected a rule that sets up device nodes named by > > PCI slot to be equally predictable) > I guess one unanswered question is whether Wojciech is bolting his solution > onto an existing system *or* designing a system from scratch (in which case, > you would assume he could exercise some control over how the hardware is > "built"/pieced together)
The end user wants to be able to run it on standard Debian GNU/Linux OS. BR, Wojtek