On Tue, 1 Mar 2022 16:39:38 -0400 Jason Gunthorpe <jgg@xxxxxxxxxx> wrote: > On Tue, Mar 01, 2022 at 12:30:47PM -0700, Alex Williamson wrote: > > Wouldn't it make more sense if initial-bytes started at QM_MATCH_SIZE > > and dirty-bytes was always sizeof(vf_data) - QM_MATCH_SIZE? ie. QEMU > > would know that it has sizeof(vf_data) - QM_MATCH_SIZE remaining even > > while it's getting ENOMSG after reading QM_MATCH_SIZE bytes of data. > > The purpose of this ioctl is to help userspace guess when moving on to > STOP_COPY is a good idea ie when the device has done almost all the > work it is going to be able to do in PRE_COPY. ENOMSG is a similar > indicator. > > I expect all devices to have some additional STOP_COPY trailer_data in > addition to their PRE_COPY initial_data and dirty_data > > There is a choice to make if we report the trailer_data during > PRE_COPY or not. As this is all estimates, it doesn't matter unless > the trailer_data is very big. > > Having all devices trend toward a 0 dirty_bytes to say they are are > done all the pre-copy they can do makes sense from an API > perspective. If one device trends toward 10MB due to a big > trailer_data and one trends toward 0 bytes, how will qemu consistently > decide when best to trigger STOP_COPY? It makes the API less useful. > > So, I would not include trailer_data in the dirty_bytes. That assumes that it's possible to keep up with the device dirty rate. It seems like a better approach for userspace would be to look at how dirty_bytes is trending. A zero value and a steady state value are equivalent, there's nothing more to be gained by further iterations. If the value is trending down, it might be worthwhile to iterate in PRE_COPY a while longer. If the value is trending up, it might be time to cut to STOP_COPY or abort the migration. If we exclude STOP_COPY trailing data from the VFIO_DEVICE_MIG_PRECOPY ioctl, it seems even more of a disconnect that when we enter the STOP_COPY state, suddenly we start getting new data out of a PRECOPY ioctl. BTW, "VFIO_DEVICE" should be reserved for ioctls and data structures relative to the device FD, appending it with _MIG is too subtle for me. This is also a GET operation for INFO, so I'd think for consistency with the existing vfio uAPI we'd name this something like VFIO_MIG_GET_PRECOPY_INFO where the structure might be named vfio_precopy_info. > Estimating when to move on to STOP_COPY and trying to enforce a SLA on > STOP_COPY are different tasks and will probably end up with different > interfaces. > > I still think the right way to approach the SLA is to inform the > driver what the permitted time and data size target is for STOP_COPY > and the driver can proceed or not based on its own internal > calculation. So if we don't think this is the right approach for STOP_COPY, then why are we pushing that it has any purpose outside of PRECOPY or might be implemented by a non-PRECOPY driver for use in STOP_COPY? > > useful yet and you don't want to add dead kernel code, then let's > > define that this ioctl is only available in the PRE_COPY* states and > > returns -errno in the STOP_COPY state. > > I'm OK with that, in acc it is done by checking migf->total_bytes > > QM_MATCH_SIZE during the read fop > > > devices in STOP_COPY and let's also define if there's actually anything > > userspace can infer about remaining STOP_COPY data size while in > > PRE_COPY* via this ioctl. For example, is dirty-bytes zero or the > > remaining data structure size? > > If we keep it then I would say it doesn't matter, userspace has to sum > the two values to get the total remaining length estimate, it is just > a bit quirky. For the reasons above, I just can't figure out why wouldn't decide that use of this outside of PRECOPY is too quirky to bother with. Thanks, Alex
