On Wed, 29 Mar 2023 16:47:49 -0600
Alex Williamson <alex.williamson@xxxxxxxxxx> wrote:
> On Mon, 27 Mar 2023 02:40:47 -0700
> Yi Liu <yi.l.liu@xxxxxxxxx> wrote:
>
> > This gives notes for userspace applications on device cdev usage.
> >
> > Reviewed-by: Kevin Tian <kevin.tian@xxxxxxxxx>
> > Signed-off-by: Yi Liu <yi.l.liu@xxxxxxxxx>
> > ---
> > Documentation/driver-api/vfio.rst | 127 ++++++++++++++++++++++++++++++
> > 1 file changed, 127 insertions(+)
> >
> > diff --git a/Documentation/driver-api/vfio.rst b/Documentation/driver-api/vfio.rst
> > index 363e12c90b87..77408788b98d 100644
> > --- a/Documentation/driver-api/vfio.rst
> > +++ b/Documentation/driver-api/vfio.rst
> > @@ -239,6 +239,125 @@ group and can access them as follows::
> > /* Gratuitous device reset and go... */
> > ioctl(device, VFIO_DEVICE_RESET);
> >
> > +IOMMUFD and vfio_iommu_type1
> > +----------------------------
> > +
> > +IOMMUFD is the new user API to manage I/O page tables from userspace.
> > +It intends to be the portal of delivering advanced userspace DMA
> > +features (nested translation [5], PASID [6], etc.) while being backward
> > +compatible with the vfio_iommu_type1 driver. Eventually vfio_iommu_type1
> > +will be deprecated.
>
> "... while also providing a backwards compatibility interface for
> existing VFIO_TYPE1v2_IOMMU use cases. Eventually the vfio_iommu_type1
> driver, as well as the legacy vfio container and group model is
> intended to be deprecated."
>
> > +
> > +With the backward compatibility, no change is required for legacy
> > VFIO +drivers or applications to connect a VFIO device to IOMMUFD.
> > +
> > + When CONFIG_IOMMUFD_VFIO_CONTAINER=n, VFIO container still provides
> > + /dev/vfio/vfio which connects to vfio_iommu_type1. To disable VFIO
> > + container and vfio_iommu_type1, the administrator could symbol link
> > + /dev/vfio/vfio to /dev/iommu to enable VFIO container emulation
> > + in IOMMUFD.
> > +
> > + When CONFIG_IOMMUFD_VFIO_CONTAINER=y, IOMMUFD directly provides
> > + /dev/vfio/vfio while the VFIO container and vfio_iommu_type1 are
> > + explicitly disabled.
> > +
>
> "The IOMMUFD backwards compatibility interface can be enabled two ways.
> In the first method, the kernel can be configured with
> CONFIG_IOMMUFD_VFIO_CONTAINER, in which case the IOMMUFD subsystem
> transparently provides the entire infrastructure for the the VFIO
> container and IOMMU backend interfaces. The compatibility mode can
> also be accessed if the VFIO container interface, ie. /dev/vfio/vfio is
> simply symlink'd to /dev/iommu. Note that at the time of writing, the
> compatibility mode is not entirely feature complete relative to
> VFIO_TYPE1v2_IOMMU (ex. DMA mapping MMIO) and does not attempt to
> provide compatibility to the VFIO_SPAPR_TCE_IOMMU interface. Therefore
> it is not generally advisable at this time to switch from native VFIO
> implementations to the IOMMUFD compatibility interfaces.
>
> Long term, VFIO users should migrate to device access through the cdev
> interface described below, and native access through the IOMMUFD
> provided interfaces."
>
> Thanks,
> Alex
>
> > +VFIO Device cdev
> > +----------------
> > +
> > +Traditionally user acquires a device fd via VFIO_GROUP_GET_DEVICE_FD
> > +in a VFIO group.
> > +
> > +With CONFIG_VFIO_DEVICE_CDEV=y the user can now acquire a device fd
> > +by directly opening a character device /dev/vfio/devices/vfioX where
> > +"X" is the number allocated uniquely by VFIO for registered devices.
> > +For noiommu devices, the character device would be named with
> > "noiommu-" +prefix. e.g. /dev/vfio/devices/noiommu-vfioX.
> > +
> > +The cdev only works with IOMMUFD. Both VFIO drivers and applications
> > +must adapt to the new cdev security model which requires using
> > +VFIO_DEVICE_BIND_IOMMUFD to claim DMA ownership before starting to
> > +actually use the device. Once BIND succeeds then a VFIO device can
> > +be fully accessed by the user.
> > +
> > +VFIO device cdev doesn't rely on VFIO group/container/iommu drivers.
> > +Hence those modules can be fully compiled out in an environment
> > +where no legacy VFIO application exists.
> > +
> > +So far SPAPR does not support IOMMUFD yet. So it cannot support
> > device +cdev neither.
> > +
> > +Device cdev Example
> > +-------------------
> > +
> > +Assume user wants to access PCI device 0000:6a:01.0::
> > +
> > + $ ls /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/
> > + vfio0
> > +
> > +This device is therefore represented as vfio0. The user can verify
> > +its existence::
> > +
> > + $ ls -l /dev/vfio/devices/vfio0
> > + crw------- 1 root root 511, 0 Feb 16 01:22
> > /dev/vfio/devices/vfio0
> > + $ cat /sys/bus/pci/devices/0000:6a:01.0/vfio-dev/vfio0/dev
> > + 511:0
> > + $ ls -l /dev/char/511\:0
> > + lrwxrwxrwx 1 root root 21 Feb 16 01:22 /dev/char/511:0 ->
> > ../vfio/devices/vfio0 +
> > +Then provide the user with access to the device if unprivileged
> > +operation is desired::
> > +
> > + $ chown user:user /dev/vfio/devices/vfio0
> > +
> > +Finally the user could get cdev fd by::
> > +
> > + cdev_fd = open("/dev/vfio/devices/vfio0", O_RDWR);
> > +
> > +An opened cdev_fd doesn't give the user any permission of accessing
> > +the device except binding the cdev_fd to an iommufd. After that
> > point +then the device is fully accessible including attaching it to
> > an +IOMMUFD IOAS/HWPT to enable userspace DMA::
> > +
> > + struct vfio_device_bind_iommufd bind = {
> > + .argsz = sizeof(bind),
> > + .flags = 0,
> > + };
> > + struct iommu_ioas_alloc alloc_data = {
> > + .size = sizeof(alloc_data),
> > + .flags = 0,
> > + };
> > + struct vfio_device_attach_iommufd_pt attach_data = {
> > + .argsz = sizeof(attach_data),
> > + .flags = 0,
> > + };
> > + struct iommu_ioas_map map = {
> > + .size = sizeof(map),
> > + .flags = IOMMU_IOAS_MAP_READABLE |
> > + IOMMU_IOAS_MAP_WRITEABLE |
> > + IOMMU_IOAS_MAP_FIXED_IOVA,
> > + .__reserved = 0,
> > + };
> > +
> > + iommufd = open("/dev/iommu", O_RDWR);
> > +
> > + bind.iommufd = iommufd; // negative value means vfio-noiommu
> > mode
> > + ioctl(cdev_fd, VFIO_DEVICE_BIND_IOMMUFD, &bind);
Even here the dev_cookie is nowhere to be found! Thanks,
Alex
> > +
> > + ioctl(iommufd, IOMMU_IOAS_ALLOC, &alloc_data);
> > + attach_data.pt_id = alloc_data.out_ioas_id;
> > + ioctl(cdev_fd, VFIO_DEVICE_ATTACH_IOMMUFD_PT, &attach_data);
> > +
> > + /* Allocate some space and setup a DMA mapping */
> > + map.user_va = (int64_t)mmap(0, 1024 * 1024, PROT_READ |
> > PROT_WRITE,
> > + MAP_PRIVATE | MAP_ANONYMOUS, 0,
> > 0);
> > + map.iova = 0; /* 1MB starting at 0x0 from device view */
> > + map.length = 1024 * 1024;
> > + map.ioas_id = alloc_data.out_ioas_id;;
> > +
> > + ioctl(iommufd, IOMMU_IOAS_MAP, &map);
> > +
> > + /* Other device operations as stated in "VFIO Usage Example"
> > */ +
> > VFIO User API
> > -------------------------------------------------------------------------------
> >
> > @@ -566,3 +685,11 @@ This implementation has some specifics:
> > \-0d.1
> >
> > 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev
> > 90) +
> > +.. [5] Nested translation is an IOMMU feature which supports two
> > stage
> > + address translations. This improves the address translation
> > efficiency
> > + in IOMMU virtualization.
> > +
> > +.. [6] PASID stands for Process Address Space ID, introduced by PCI
> > + Express. It is a prerequisite for Shared Virtual Addressing (SVA)
> > + and Scalable I/O Virtualization (Scalable IOV).
>
