> From: Xiang Zheng > Sent: Wednesday, July 22, 2020 10:56 AM > > Hi Alex, > > Thank you for your suggestion. > > On 2020/7/22 6:43, Alex Williamson wrote: > > On Tue, 21 Jul 2020 10:43:21 +0800 > > Xiang Zheng <zhengxiang9@xxxxxxxxxx> wrote: > > > >> Hi Kirti, > >> > >> Sorry to disturb you since this patch set has been merged, and I cannot > >> receive the qemu-side emails about this patch set. > >> > >> We are going to support migration for VFIO devices which support dirty > >> pages tracking. > >> > >> And we also plan to leverage SMMU HTTU feature to do the dirty pages > >> tracking for the devices which don't support dirty pages tracking. > >> > >> For the above two cases, which side determines to choose IOMMU driver > or > >> vendor driver to do dirty bitmap tracking, Qemu or VFIO? > >> > >> In brief, if both IOMMU and VFIO devices support dirty pages tracking, > >> we can check the capability and prefer to track dirty pages on device > >> vendor driver which is more efficient. > >> > >> The qusetion is which side to do the check and selection? In my opinion, > >> Qemu/userspace seems more suitable. > > > > Dirty page tracking is consolidated at the vfio container level. > > Userspace has no basis for determining or interface for selecting a > > dirty bitmap provider, so I would disagree that QEMU should play any > > role here. The container dirty bitmap tries to provide the finest > > granularity available based on the support of all the devices/groups > > managed by the container. If there are groups attached to the > > container that have not participated in page pinning, then we consider > > all DMA mappings within the container as persistently dirty. Once all > > of the participants subscribe to page pinning, the dirty scope is > > reduced to the pinned pages. IOMMU support for dirty page logging would > > introduce finer granularity yet, which we would probably prefer over > > page pinning, but interfaces for this have not been devised. > > Kevin and his colleagues may add these APIs in the future. > We also plan to support these interfaces on SMMU driver and afterwards we > can have a further discussion. Yes, we are working on that part. Generally speaking I agree with Alex that the kernel should decide dirty bitmap provider and IOMMU dirty logging is likely preferred over page pinning. Page pinning either provides only coarse-grained dirty info (e.g. 100's MB pinned pages observed in vGPU case) or when reaching finer granularity it may affect fast-path performance (e.g. driver may have to intercept fast-path operations to track dirtied pages closely). IOMMU provides an architectural and lighter approach for tracking dirty pages. Thanks Kevin > > > > > Ideally userspace should be unaware of any of this, the benefit would > > be seen transparently by having a more sparsely filled dirty bitmap, > > which more accurately reflects how memory is actually being dirtied. > > Yes, indeed. > > -- > Thanks, > Xiang
