Hi folks,
I'm looking at the way we start dirty tracking for the vfio container
(we start it at the save_setup stage of a device) and have a few
questions about it. Please fix me up if I had missed something obvious.
- We may end up with the situation where we try to get dirty bitmap of
the vfio container whilst the dirty tracking hasn't yet been enabled.
ram_save_setup
ram_init_bitmaps
migration_bitmap_sync_precopy (i.e., vfio_listener_log_sync)
vfio_save_setup
vfio_migration_set_state += _SAVING
vfio_set_dirty_page_tracking(true)
Practically there is no problem since in vfio_listener_log_sync(), we
will check that if all devices are having the _SAVING state *before*
actually syncing dirty bitmap. As we add _SAVING to the device state
exactly before starting dirty tracking, we'll actually do nothing in
the first run of vfio_listener_log_sync().
But I'm wondering what prevents us from starting dirty tracking via
the "standard" log_start() callback of the vfio container.
- Before start dirty tracking, we will check and ensure that the device
is at _SAVING state and return error otherwise. But the question is
that what is the rationale? Why does the VFIO_IOMMU_DIRTY_PAGES ioctl
have something to do with the device state?
VFIO_IOMMU_DIRTY_PAGES is clearly designed as a API for type1 VFIO
IOMMU. Is it possible that the *device* itself becomes the actual
dirty bitmap provider of this ioctl (e.g., the device is smart enough
to track its own dirty pages during migration)? This is the only case
I can think out where we may have to set the device to pre-copy state
before starting dirty tracking, so that device can report dirty pages
properly in response to the GET_BITMAP ioctl. But afaict it isn't
covered by the existing documentation [1][2]...
Btw, it isn't clear that what possible dirty bitmap providers will be
(IOMMU, devices, etc) behind the back of the VFIO_IOMMU_DIRTY_PAGES
ioctl. Although userspace can be unaware of any of this, it'd be
appreciated if someone can clarify it.
- If there is indeed some dependency between the device state and the
VFIO_IOMMU_DIRTY_PAGES_FLAG_START operation, shouldn't we check that
*all* devices are now having the _SAVING state before START, rather
than just checking the *given* device in its (per-device) save_setup
stage?
[1] https://git.kernel.org/torvalds/c/a8a24f3f6e38
[2] https://git.kernel.org/torvalds/c/b704fd14a06f
Thanks,
Zenghui
