> From: Jason Gunthorpe <jgg@xxxxxxxxxx>
> Sent: Tuesday, November 8, 2022 8:53 AM
>
> iommufd doesn't establish the iommu_domains until after the device FD is
> opened, even if the container has been set. This design is part of moving
> away from the group centric iommu APIs.
>
> This is fine, except that the normal sequence of establishing the kvm
> wbinvd won't work:
>
> group = open("/dev/vfio/XX")
> ioctl(group, VFIO_GROUP_SET_CONTAINER)
> ioctl(kvm, KVM_DEV_VFIO_GROUP_ADD)
> ioctl(group, VFIO_GROUP_GET_DEVICE_FD)
>
> As the domains don't start existing until GET_DEVICE_FD. Further,
> GET_DEVICE_FD requires that KVM_DEV_VFIO_GROUP_ADD already be
> done as that
> is what sets the group->kvm and thus device->kvm for the driver to use
> during open.
>
> Now that we have device centric cap ops and the new
> IOMMU_CAP_ENFORCE_CACHE_COHERENCY we know what the
> iommu_domain will be
> capable of without having to create it. Use this to compute
> vfio_file_enforced_coherent() and resolve the ordering problems.
>
> VFIO always tries to upgrade domains to enforce cache coherency, it never
> attaches a device that supports enforce cache coherency to a less capable
> domain, so the cap test is a sufficient proxy for the ultimate
> outcome. iommufd also ensures that devices that set the cap will be
> connected to enforcing domains.
>
> Tested-by: Nicolin Chen <nicolinc@xxxxxxxxxx>
> Signed-off-by: Jason Gunthorpe <jgg@xxxxxxxxxx>
Reviewed-by: Kevin Tian <kevin.tian@xxxxxxxxx>
