On Fri, May 28, 2021 at 10:58:05PM +0300, Dmitrii Shcherbakov wrote: > Hello Libvirt Developers, > > I am looking for some feedback on a planned enhancement to Libvirt: the aim > is > to store a portion of PCI(e) Vital Product Data (VPD) for each device along > with other PCI/PCIe device information already collected. Specifically, the > SN > (Serial Number) read-only field of a VPD data structure of a device is of > interest which is described in PCI/PCIe specs (PCI local bus 2.1+ and PCIe > 4.0+). > > The context for this is the cross-project work in OpenStack (Nova, Neutron), > OVS and OVN to support for off-path SmartNIC DPUs ([1], [2], [3], [4]). The > Nova specification [1] provides an overview of the relevant hardware and the > use-case for board serial numbers, however, VPD is the standard capability > in > the PCI/PCIe specifications not tied to the use-case in particular so the > suggestion from the Nova core team was to aim at introducing means of > collecting this information via Libvirt. It can then be retrieved by the > respective virt driver in Nova via Libvirt without having to introduce this > code into Nova itself. I've talked with Sean Mooney at little about the use case this morning. IIUC, the high level scenario is as follows - The main host machine has a PCI controller topology to which the NICs are attached. This is how the host OS and by extension libvirt, nova, etc, see the PCI devices. - There is a second PCI controller topology to which the NICs are attached. This is only visible to the arm cores for the offload engine - Nova/Neutron can identify the NICs based on the PCI topology seen by the host OS, but need to tell the NIC mgmt software which NIC to use in a way that can be undersood by the offload cores. IOW, the PCI address is not usable as a unique identifier because there are two completely independant PCI topologies with no mapping between them. The VPD data provides a replacement way to identify a NIC based on a unique serial number that is indendant of PCI topology. Nova needs this serial number in order to configure the device offload featues. This seems like a reasonable feature request to me, since there is a piece of info that apps using libvirt need, and libvirt does not expose this. Requiring the mgmt app like Nova to dig into the host PCI config space indicates a clear gap in libvirt functionality. > I would like to suggest the following to be done in Libvirt: > > 1) adding the code for extracting a serial number from VPD for PCI/PCIe > devices > in general and storing it for exposure via the Libvirt API; > More specifically, I propose adding a nested capability called "vpd" under > VIR_NODE_DEV_CAP_PCI_DEV: > <capability type='pci'> > <capability type='vpd'> > <serial>UNIQUESERIAL</serial> > <!-- ... other VPD attributes if present --> > </capability> > <!-- ... --> > </capability> This looks like a reasonable proposal > 2) (optional) implementing functionality to obtain a board serial number via > devlink-info for PCIe devices if they do not expose a VPD capability > but the device driver can retrieve it via firmware. The board serial number > can be stored in the same element as suggested above. Is scenario (2) going to be at all common ? What would be a reason why the info is not exposed via the standardized VPD - is it just a legacy hardware issue ? > Not all devices expose the devlink API and even fewer do expose board serial > via devlink-info: > > * devlink was added in 4.10 [11]; > * devlink-info was introduced in 5.1 [12]; > * querying for board.serial_number was added in kernel 5.9 [13] and iproute2 > 5.9.0 [14]; > * Besides the generic devlink infrastructure support above, device drivers > also need to support exposing this field. > > Therefore, implementing two approaches (sysfs VPD, devlink) is preferable > for better compatibility. > > I would appreciate any feedback on whether this potential addition makes > sense. > If so, I can look into implementing this. It makes sense to me. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
