On Mon, 28 Feb 2022 19:47:09 -0400 Jason Gunthorpe <jgg@xxxxxxxxxx> wrote: > On Mon, Feb 28, 2022 at 02:20:34PM -0700, Alex Williamson wrote: > > > > Unless you think we should block it. > > > > What's the meaning of initial_bytes and dirty_bytes while in > > STOP_COPY? > > Same as during pre-copy - both numbers are the bytes remaining to be > read() from the FD in each bucket. They should continue to decline as > read() progresses regardless of what state the data_fd is in. > > The only special thing about STOP_COPY is that dirty_bytes should not > increase as the device should not be generating new dirty data. > > How about: > > * Drivers should attempt to return estimates so that initial_bytes + > * dirty_bytes matches the amount of data an immediate transition to STOP_COPY > * will require to be streamed. While in STOP_COPY the initial_bytes > * and dirty_bytes should continue to be decrease as the data_fd > * progresses streaming out the data. > > Remove the 'in the precopy phase' from the first sentance > > Adjust the last paragraph as: > > + * returning readable. ENOMSG may not be returned in STOP_COPY. Support > + * for this ioctl is required when VFIO_MIGRATION_PRE_COPY is set. This entire ioctl on the data_fd seems a bit strange given the previous fuss about how difficult it is for a driver to estimate their migration data size. Now drivers are forced to provide those estimates even if they only intend to use PRE_COPY as an early compatibility test? Obviously it's trivial for the acc driver that doesn't support dirty tracking and only has a fixed size migration structure, but it seems to contradict your earlier statements. For instance, can mlx5 implement a PRE_COPY solely for compatibility testing or is it blocked by an inability to provide data estimates for this ioctl? Now if we propose that this ioctl is useful during the STOP_COPY phase, how does a non-PRE_COPY driver opt-in to that beneficial use case? Do we later add a different, optional ioctl for non-PRE_COPY and then require userspace to support two different methods of getting remaining data estimates for a device in STOP_COPY? If our primary goal is to simplify the FSM, I'm actually a little surprised we support the PRE_COPY* -> STOP_COPY transition directly versus passing through STOP. It seems this exists due to our policy that we can only generate one data_fd as a result of any sequence of state transitions, but I think there might also be an option to achieve similar if the PRE_COPY* states are skipped if they aren't the ultimate end state of the arc. I'm sure that raises questions about how we correlate a PRE_COPY* session to a STOP_COPY session though, but this PRE_COPY* specific but ongoing usage in STOP_COPY ioctl seems ad-hoc. Thanks, Alex
