Hello Libvirt Developers,
I am looking for some feedback on a planned enhancement to Libvirt: the aim is
to store a portion of PCI(e) Vital Product Data (VPD) for each device along
with other PCI/PCIe device information already collected. Specifically, the SN
(Serial Number) read-only field of a VPD data structure of a device is of
interest which is described in PCI/PCIe specs (PCI local bus 2.1+ and PCIe
4.0+).
The context for this is the cross-project work in OpenStack (Nova, Neutron),
OVS and OVN to support for off-path SmartNIC DPUs ([1], [2], [3], [4]). The
Nova specification [1] provides an overview of the relevant hardware and the
use-case for board serial numbers, however, VPD is the standard capability in
the PCI/PCIe specifications not tied to the use-case in particular so the
suggestion from the Nova core team was to aim at introducing means of
collecting this information via Libvirt. It can then be retrieved by the
respective virt driver in Nova via Libvirt without having to introduce this
code into Nova itself.
Quoting the PCI(e) specs:
* "Vital Product Data (VPD) is the information that uniquely defines items such
as the hardware, software, and microcode elements of a system.";
* "Vital Product Data is made up of Small and Large Resource Data Types.";
* "Large resource type VPD-R Tag: This tag contains the read only VPD keywords
for an add-in card."
* SN read-only field: "The characters are alphanumeric and represent the unique
add-in card Serial Number."
The VPD capability is optional per the specification so it may or may not
appear for PCI(e) endpoints. The devices of interest (SmartNIC DPUs), however,
generally have it exposed.
The PCI/PCIe specs define a binary format for VPD and a sysfs entry exposing
a binary blob in that format has been available since kernel v2.6.26 [5]. The
relevant sections of specs are:
* "6.4. Vital Product Data" in the PCI Local Bus specification;
* "6.28 Vital Product Data (VPD)" in the PCIe 4.0 Base Specification.
Note that the serial number stored in VPD is not identical to the information
stored in the Device Serial Number (DSN) capability also present in the specs
as it may identify a component on a board which presents a multi-function
device but the board itself may have multiple components ([9] also makes a
distinction between a board serial and a device serial).
As a reference, there is some code to parse and print the VPD in lspci [6] and
there is a prototype along those lines in Python [7], a polished version of
which I plan to use in Nova until the relevant functionality appears in
Libvirt.
Likewise, the devlink kernel infrastructure, which is already used in Libvirt
to query additional device capabilities [8] (e.g. the presence of an eswitch
and its switchdev mode) has a devlink-info API [9] that exposes a way to query
a board serial number if a device driver exposes it (in turn, by querying
controller firmware or via PCIe VPD). This allows doing that in a
bus-independent manner (e.g. it would work for PCIe, platform devices or other
I/O interconnects) but in the context of devices that implement devlink API
only (which are not necessarily network devices [10] but most of them currently
are).
I would like to suggest the following to be done in Libvirt:
1) adding the code for extracting a serial number from VPD for PCI/PCIe devices
in general and storing it for exposure via the Libvirt API;
More specifically, I propose adding a nested capability called "vpd" under
VIR_NODE_DEV_CAP_PCI_DEV:
<capability type='pci'>
<capability type='vpd'>
<serial>UNIQUESERIAL</serial>
<!-- ... other VPD attributes if present -->
</capability>
<!-- ... -->
</capability>
2) (optional) implementing functionality to obtain a board serial number via
devlink-info for PCIe devices if they do not expose a VPD capability
but the device driver can retrieve it via firmware. The board serial number
can be stored in the same element as suggested above.
I am looking for some feedback on a planned enhancement to Libvirt: the aim is
to store a portion of PCI(e) Vital Product Data (VPD) for each device along
with other PCI/PCIe device information already collected. Specifically, the SN
(Serial Number) read-only field of a VPD data structure of a device is of
interest which is described in PCI/PCIe specs (PCI local bus 2.1+ and PCIe
4.0+).
The context for this is the cross-project work in OpenStack (Nova, Neutron),
OVS and OVN to support for off-path SmartNIC DPUs ([1], [2], [3], [4]). The
Nova specification [1] provides an overview of the relevant hardware and the
use-case for board serial numbers, however, VPD is the standard capability in
the PCI/PCIe specifications not tied to the use-case in particular so the
suggestion from the Nova core team was to aim at introducing means of
collecting this information via Libvirt. It can then be retrieved by the
respective virt driver in Nova via Libvirt without having to introduce this
code into Nova itself.
Quoting the PCI(e) specs:
* "Vital Product Data (VPD) is the information that uniquely defines items such
as the hardware, software, and microcode elements of a system.";
* "Vital Product Data is made up of Small and Large Resource Data Types.";
* "Large resource type VPD-R Tag: This tag contains the read only VPD keywords
for an add-in card."
* SN read-only field: "The characters are alphanumeric and represent the unique
add-in card Serial Number."
The VPD capability is optional per the specification so it may or may not
appear for PCI(e) endpoints. The devices of interest (SmartNIC DPUs), however,
generally have it exposed.
The PCI/PCIe specs define a binary format for VPD and a sysfs entry exposing
a binary blob in that format has been available since kernel v2.6.26 [5]. The
relevant sections of specs are:
* "6.4. Vital Product Data" in the PCI Local Bus specification;
* "6.28 Vital Product Data (VPD)" in the PCIe 4.0 Base Specification.
Note that the serial number stored in VPD is not identical to the information
stored in the Device Serial Number (DSN) capability also present in the specs
as it may identify a component on a board which presents a multi-function
device but the board itself may have multiple components ([9] also makes a
distinction between a board serial and a device serial).
As a reference, there is some code to parse and print the VPD in lspci [6] and
there is a prototype along those lines in Python [7], a polished version of
which I plan to use in Nova until the relevant functionality appears in
Libvirt.
Likewise, the devlink kernel infrastructure, which is already used in Libvirt
to query additional device capabilities [8] (e.g. the presence of an eswitch
and its switchdev mode) has a devlink-info API [9] that exposes a way to query
a board serial number if a device driver exposes it (in turn, by querying
controller firmware or via PCIe VPD). This allows doing that in a
bus-independent manner (e.g. it would work for PCIe, platform devices or other
I/O interconnects) but in the context of devices that implement devlink API
only (which are not necessarily network devices [10] but most of them currently
are).
I would like to suggest the following to be done in Libvirt:
1) adding the code for extracting a serial number from VPD for PCI/PCIe devices
in general and storing it for exposure via the Libvirt API;
More specifically, I propose adding a nested capability called "vpd" under
VIR_NODE_DEV_CAP_PCI_DEV:
<capability type='pci'>
<capability type='vpd'>
<serial>UNIQUESERIAL</serial>
<!-- ... other VPD attributes if present -->
</capability>
<!-- ... -->
</capability>
2) (optional) implementing functionality to obtain a board serial number via
devlink-info for PCIe devices if they do not expose a VPD capability
but the device driver can retrieve it via firmware. The board serial number
can be stored in the same element as suggested above.
Not all devices expose the devlink API and even fewer do expose board serial
via devlink-info:
* devlink was added in 4.10 [11];
* devlink-info was introduced in 5.1 [12];
* querying for board.serial_number was added in kernel 5.9 [13] and iproute2
5.9.0 [14];
* Besides the generic devlink infrastructure support above, device drivers
also need to support exposing this field.
Therefore, implementing two approaches (sysfs VPD, devlink) is preferable for better compatibility.
I would appreciate any feedback on whether this potential addition makes sense.
If so, I can look into implementing this.
[1] https://review.opendev.org/c/openstack/nova-specs/+/787458
[2] https://review.opendev.org/c/openstack/neutron-specs/+/788821
[3] https://patchwork.ozlabs.org/project/openvswitch/patch/20210323145032.453120-1-frode.nordahl@xxxxxxxxxxxxx/
[4] https://patchwork.ozlabs.org/project/ovn/patch/20210509140305.1910796-1-frode.nordahl@xxxxxxxxxxxxx/
[5] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=94e6108803469a37ee1e3c92dafdd1d59298602f
[6] https://github.com/pciutils/pciutils/blob/v3.7.0/ls-vpd.c#L95-L216
[7] https://gist.github.com/dshcherb/40e982989599a757e5b1e25999501019
[8] https://github.com/libvirt/libvirt/blob/v7.3.0/src/util/virnetdev.c#L3167-L3245
[9] https://www.kernel.org/doc/html/latest/networking/devlink/devlink-info.html
[10] https://www.kernel.org/doc/html/latest/networking/devlink/index.html
[11] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bfcd3a46617209454cfc0947ab093e37fd1e84ef
[12] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f9cf22882c606f3ffe06f620bb6d03b9eff18d3d
[13] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b5872cd0e823e4cb50b3a75cd9522167eeb676a2
[14] https://git.kernel.org/pub/scm/network/iproute2/iproute2.git/commit/?id=7332b188a6f8e5c5d67c9a03d1591a813a4c908c
Best Regards,
Dmitrii Shcherbakov
LP: ~dmitriis
