On Tue, 13 Jun 2023 14:48:02 +0000 "Liu, Yi L" <yi.l.liu@xxxxxxxxx> wrote: > > From: Alex Williamson <alex.williamson@xxxxxxxxxx> > > Sent: Tuesday, June 13, 2023 10:24 PM > > > > On Tue, 13 Jun 2023 12:01:51 +0000 > > "Liu, Yi L" <yi.l.liu@xxxxxxxxx> wrote: > > > > > > From: Alex Williamson <alex.williamson@xxxxxxxxxx> > > > > Sent: Tuesday, June 13, 2023 7:06 AM > > > > > > > > On Fri, 2 Jun 2023 05:16:53 -0700 > > > > Yi Liu <yi.l.liu@xxxxxxxxx> wrote: > > > > > > > > > This gives notes for userspace applications on device cdev usage. > > > > > > > > > > Reviewed-by: Kevin Tian <kevin.tian@xxxxxxxxx> > > > > > Signed-off-by: Yi Liu <yi.l.liu@xxxxxxxxx> > > > > > --- > > > > > Documentation/driver-api/vfio.rst | 132 ++++++++++++++++++++++++++++++ > > > > > 1 file changed, 132 insertions(+) > > > > > > > > > > diff --git a/Documentation/driver-api/vfio.rst b/Documentation/driver-api/vfio.rst > > > > > index 363e12c90b87..f00c9b86bda0 100644 > > > > > --- a/Documentation/driver-api/vfio.rst > > > > > +++ b/Documentation/driver-api/vfio.rst > > > > > @@ -239,6 +239,130 @@ group and can access them as follows:: > > > > > /* Gratuitous device reset and go... */ > > > > > ioctl(device, VFIO_DEVICE_RESET); > > > > > > > > > > +IOMMUFD and vfio_iommu_type1 > > > > > +---------------------------- > > > > > + > > > > > +IOMMUFD is the new user API to manage I/O page tables from userspace. > > > > > +It intends to be the portal of delivering advanced userspace DMA > > > > > +features (nested translation [5]_, PASID [6]_, etc.) while also providing > > > > > +a backwards compatibility interface for existing VFIO_TYPE1v2_IOMMU use > > > > > +cases. Eventually the vfio_iommu_type1 driver, as well as the legacy > > > > > +vfio container and group model is intended to be deprecated. > > > > > + > > > > > +The IOMMUFD backwards compatibility interface can be enabled two ways. > > > > > +In the first method, the kernel can be configured with > > > > > +CONFIG_IOMMUFD_VFIO_CONTAINER, in which case the IOMMUFD subsystem > > > > > +transparently provides the entire infrastructure for the VFIO > > > > > +container and IOMMU backend interfaces. The compatibility mode can > > > > > +also be accessed if the VFIO container interface, ie. /dev/vfio/vfio is > > > > > +simply symlink'd to /dev/iommu. Note that at the time of writing, the > > > > > +compatibility mode is not entirely feature complete relative to > > > > > +VFIO_TYPE1v2_IOMMU (ex. DMA mapping MMIO) and does not attempt to > > > > > +provide compatibility to the VFIO_SPAPR_TCE_IOMMU interface. Therefore > > > > > +it is not generally advisable at this time to switch from native VFIO > > > > > +implementations to the IOMMUFD compatibility interfaces. > > > > > + > > > > > +Long term, VFIO users should migrate to device access through the cdev > > > > > +interface described below, and native access through the IOMMUFD > > > > > +provided interfaces. > > > > > + > > > > > +VFIO Device cdev > > > > > +---------------- > > > > > + > > > > > +Traditionally user acquires a device fd via VFIO_GROUP_GET_DEVICE_FD > > > > > +in a VFIO group. > > > > > + > > > > > +With CONFIG_VFIO_DEVICE_CDEV=y the user can now acquire a device fd > > > > > +by directly opening a character device /dev/vfio/devices/vfioX where > > > > > +"X" is the number allocated uniquely by VFIO for registered devices. > > > > > +cdev interface does not support noiommu, so user should use the legacy > > > > > +group interface if noiommu is needed. > > > > > + > > > > > +The cdev only works with IOMMUFD. Both VFIO drivers and applications > > > > > +must adapt to the new cdev security model which requires using > > > > > +VFIO_DEVICE_BIND_IOMMUFD to claim DMA ownership before starting to > > > > > +actually use the device. Once BIND succeeds then a VFIO device can > > > > > +be fully accessed by the user. > > > > > + > > > > > +VFIO device cdev doesn't rely on VFIO group/container/iommu drivers. > > > > > +Hence those modules can be fully compiled out in an environment > > > > > +where no legacy VFIO application exists. > > > > > + > > > > > +So far SPAPR does not support IOMMUFD yet. So it cannot support device > > > > > +cdev neither. > > > > > > > > s/neither/either/ > > > > > > Got it. > > > > > > > > > > > Unless I missed it, we've not described that vfio device cdev access is > > > > still bound by IOMMU group semantics, ie. there can be one DMA owner > > > > for the group. That's a pretty common failure point for multi-function > > > > consumer device use cases, so the why, where, and how it fails should > > > > be well covered. > > > > > > Yes. this needs to be documented. How about below words: > > > > > > vfio device cdev access is still bound by IOMMU group semantics, ie. there > > > can be only one DMA owner for the group. Devices belonging to the same > > > group can not be bound to multiple iommufd_ctx. > > > > ... or shared between native kernel and vfio drivers. > > I suppose you mean the devices in one group are bound to different > drivers. right? Essentially, but we need to be careful that we're developing multiple vfio drivers for a given bus now, which is why I try to distinguish between the two sets of drivers. Thanks, Alex > > > The users that try to bind > > > such device to different iommufd shall be failed in VFIO_DEVICE_BIND_IOMMUFD > > > which is the start point to get full access for the device. > > > > "A violation of this ownership requirement will fail at the > > VFIO_DEVICE_BIND_IOMMUFD ioctl, which gates full device access." > > Got it. > > Regards, > Yi Liu >
