On Mon, 7 Feb 2022 19:22:16 +0200 Yishai Hadas <yishaih@xxxxxxxxxx> wrote: > From: Jason Gunthorpe <jgg@xxxxxxxxxx> > > The optional PRE_COPY states open the saving data transfer FD before > reaching STOP_COPY and allows the device to dirty track internal state > changes with the general idea to reduce the volume of data transferred > in the STOP_COPY stage. > > While in PRE_COPY the device remains RUNNING, but the saving FD is open. > > Only if the device also supports RUNNING_P2P can it support PRE_COPY_P2P, > which halts P2P transfers while continuing the saving FD. > > PRE_COPY, with P2P support, requires the driver to implement 7 new arcs > and exists as an optional FSM branch between RUNNING and STOP_COPY: > RUNNING -> PRE_COPY -> PRE_COPY_P2P -> STOP_COPY > > A new ioctl VFIO_DEVICE_MIG_PRECOPY is provided to allow userspace to > query the progress of the precopy operation in the driver with the idea it > will judge to move to STOP_COPY at least once the initial data set is > transferred, and possibly after the dirty size has shrunk appropriately. > > We think there may also be merit in future extensions to the > VFIO_DEVICE_MIG_PRECOPY ioctl to also command the device to throttle the > rate it generates internal dirty state. > > Compared to the v1 clarification, STOP_COPY -> PRE_COPY is made optional > and to be defined in future. While making the whole PRE_COPY feature > optional eliminates the concern from mlx5, this is still a complicated arc > to implement and seems prudent to leave it closed until a proper use case > is developed. We also split the pending_bytes report into the initial and > sustaining values, and define the protocol to get an event via poll() for > new dirty data during PRE_COPY. I feel obligated to ask, is PRE_COPY support essentially RFC at this point since we have no proposed in-kernel users? It seems like we're winding down comments on the remainder of the series and I feel ok with where it's headed and the options we have available for future extensions. Pre-copy seems like an important gap to fill and I think this patch shows that a future extension could allow it, but with the scrutiny not to add unused code to the kernel, I'm not sure there's a valid justification to add it now. Thanks, Alex PS - Why is this a stand-alone ioctl rather than a DEVICE_FEATURE?
