Hi, Jason, > From: Jason Gunthorpe <jgg@xxxxxxxxxx> > Sent: Thursday, August 25, 2022 8:44 AM > > On Wed, Aug 24, 2022 at 04:02:01PM -0600, Alex Williamson wrote: > > > Do we expect mdev drivers supporting migration to track their dirty > > iovas locally and implement this feature? > > Assuming we ever get a SW only device that can do migration.. > > I would expect there to be a library to manage the dirty bitmap > datastructure and the instance of that datastructure to be linked to > the IOAS the device is attached to. Ie one IOAS one datastructure. > > The additional code in the mdev driver would be a handful of lines. > > If we want it to report out through the vfio or through a iommufd API > is an interesting question I don't have a solid answer to right > now. Either will work - iommufd has a nice hole to put a "sw page > table" object parallel to the "hw page table" object that the draft > series put the API on. The appeal is it makes the sharing of the > datastructure clearer to userpsace. > > The trouble with mlx5 is there isn't a nice iommufd spot to put a > device tracker. The closest is on the device id, but using the device > id in APIs intended for "page table" id's is wonky. And there is no > datatructure sharing here, obviously. Not such a nice fit. > A bit more thinking on this. Although mlx5 internal tracking is not generic, the uAPI proposed in Yishai's series is pretty generic. I wonder whether it can be extended as a formal interface in iommufd with iommu dirty tracking also being a dirty tracker implementation. Currently the concept between Yishai's and Joao's series is actually similar, both having a capability interface and a command to read/clear the bitmap of a tracked range except that Yishai's series allows userspace to do range-based tracking while Joao's series currently have the tracking enabled per the entire page table. But in concept iommu dirty tracking can support range-based interfaces too. For global dirty bit control (e.g. Intel and AMD) the iommu driver can just turn it on globally upon any enabled range. For per-PTE dirty bit control (e.g. ARM) it's actually a nice fit for range-based tracking. This pays the complexity of introducing a new object (dirty tracker) in iommufd object hierarchy, with the gain of providing a single dirty tracking interface to userspace. Kind of like an page table can have a list of dirty trackers and each tracker is used by a list of devices. If iommu dirty tracker is available it is preferred to and used by all devices (except mdev) attached to the page table. Otherwise mlx5 etc. picks a vendor specific dirty tracker if available. mdev's just share a generic software dirty tracker, not affected by the presence of iommu dirty tracker. Multiple trackers might be used for a page table at the same time. Is above all worth it? Thanks Kevin
