> From: Alex Williamson [mailto:alex.williamson@xxxxxxxxxx] > Sent: Friday, February 15, 2019 4:15 AM > To: Lu Baolu <baolu.lu@xxxxxxxxxxxxxxx> > Subject: Re: [PATCH v6 0/9] vfio/mdev: IOMMU aware mediated device > > On Wed, 13 Feb 2019 12:02:52 +0800 > Lu Baolu <baolu.lu@xxxxxxxxxxxxxxx> wrote: > > > Hi, > > > > The Mediate Device is a framework for fine-grained physical device > > sharing across the isolated domains. Currently the mdev framework is > > designed to be independent of the platform IOMMU support. As the > > result, the DMA isolation relies on the mdev parent device in a vendor > > specific way. > > > > There are several cases where a mediated device could be protected and > > isolated by the platform IOMMU. For example, Intel vt-d rev3.0 [1] > > introduces a new translation mode called 'scalable mode', which > > enables PASID-granular translations. The vt-d scalable mode is the key > > ingredient for Scalable I/O Virtualization [2] [3] which allows > > sharing a device in minimal possible granularity (ADI - Assignable > > Device Interface). > > > > A mediated device backed by an ADI could be protected and isolated by > > the IOMMU since 1) the parent device supports tagging an unique PASID > > to all DMA traffic out of the mediated device; and 2) the DMA > > translation unit (IOMMU) supports the PASID granular translation. > > We can apply IOMMU protection and isolation to this kind of devices > > just as what we are doing with an assignable PCI device. > > > > In order to distinguish the IOMMU-capable mediated devices from those > > which still need to rely on parent devices, this patch set adds one > > new member in struct mdev_device. > > > > * iommu_device > > - This, if set, indicates that the mediated device could > > be fully isolated and protected by IOMMU via attaching > > an iommu domain to this device. If empty, it indicates > > using vendor defined isolation. > > > > Below helpers are added to set and get above iommu device in mdev core > > implementation. > > > > * mdev_set/get_iommu_device(dev, iommu_device) > > - Set or get the iommu device which represents this mdev > > in IOMMU's device scope. Drivers don't need to set the > > iommu device if it uses vendor defined isolation. > > > > The mdev parent device driver could opt-in that the mdev could be > > fully isolated and protected by the IOMMU when the mdev is being > > created by invoking mdev_set_iommu_device() in its @create(). > > > > In the vfio_iommu_type1_attach_group(), a domain allocated through > > iommu_domain_alloc() will be attached to the mdev iommu device if an > > iommu device has been set. Otherwise, the dummy external domain will > > be used and all the DMA isolation and protection are routed to parent > > driver as the result. > > > > On IOMMU side, a basic requirement is allowing to attach multiple > > domains to a PCI device if the device advertises the capability and > > the IOMMU hardware supports finer granularity translations than the > > normal PCI Source ID based translation. > > > > As the result, a PCI device could work in two modes: normal mode and > > auxiliary mode. In the normal mode, a pci device could be isolated in > > the Source ID granularity; the pci device itself could be assigned to > > a user application by attaching a single domain to it. In the > > auxiliary mode, a pci device could be isolated in finer granularity, > > hence subsets of the device could be assigned to different user level > > application by attaching a different domain to each subset. > > > > Below APIs are introduced in iommu generic layer for aux-domain > > purpose: > > > > * iommu_dev_has_feature(dev, IOMMU_DEV_FEAT_AUX) > > - Check whether both IOMMU and device support IOMMU aux > > domain feature. Below aux-domain specific interfaces > > are available only after this returns true. > > > > * iommu_dev_enable/disable_feature(dev, IOMMU_DEV_FEAT_AUX) > > - Enable/disable device specific aux-domain feature. > > > > * iommu_dev_feature_enabled(dev, IOMMU_DEV_FEAT_AUX) > > - Check whether the aux domain specific feature enabled or > > not. > > > > * iommu_aux_attach_device(domain, dev) > > - Attaches @domain to @dev in the auxiliary mode. Multiple > > domains could be attached to a single device in the > > auxiliary mode with each domain representing an isolated > > address space for an assignable subset of the device. > > > > * iommu_aux_detach_device(domain, dev) > > - Detach @domain which has been attached to @dev in the > > auxiliary mode. > > > > * iommu_aux_get_pasid(domain, dev) > > - Return ID used for finer-granularity DMA translation. > > For the Intel Scalable IOV usage model, this will be > > a PASID. The device which supports Scalable IOV needs > > to write this ID to the device register so that DMA > > requests could be tagged with a right PASID prefix. > > > > In order for the ease of discussion, sometimes we call "a domain in > > auxiliary mode' or simply 'an auxiliary domain' when a domain is > > attached to a device for finer granularity translations. But we need > > to keep in mind that this doesn't mean there is a differnt domain > > type. A same domain could be bound to a device for Source ID based > > translation, and bound to another device for finer granularity > > translation at the same time. > > > > This patch series extends both IOMMU and vfio components to support > > mdev device passing through when it could be isolated and protected by > > the IOMMU units. The first part of this series (PATCH 1/09~6/09) adds > > the interfaces and implementation of the multiple domains per device. > > The second part (PATCH 7/09~9/09) adds the iommu device attribute to > > each mdev, determines isolation type according to the existence of an > > iommu device when attaching group in vfio type1 iommu module, and > > attaches the domain to iommu aware mediated devices. > > > > References: > > [1] > > https://software.intel.com/en-us/download/intel-virtualization-technol > > ogy-for-directed-io-architecture-specification > > [2] > > https://software.intel.com/en-us/download/intel-scalable-io-virtualiza > > tion-technical-specification [3] > > https://schd.ws/hosted_files/lc32018/00/LC3-SIOV-final.pdf > > > > Best regards, > > Lu Baolu > > > > Change log: > > v5->v6: > > This looks pretty reasonable with Jean-Philippe's nit fixups. Where do we go from > here? I think we need an ack from Kirti since they have an interest here. Presumably > this looks ok to the ARM folks. Do we have any consumers of this code yet? > Theoretically I think a vfio-pci-like meta driver could be written as an mdev vendor > driver with this support (restricted to type1 iommu use cases). Thanks, Hi Alex, I'll do some work on the vfio-pci-like meta driver to be a sample consumer of the vfio mdev extensions in this patchset. Will send it later. Thanks, Yi Liu
