On Wed, 27 Sep 2023 13:59:06 +0300 Yishai Hadas <yishaih@xxxxxxxxxx> wrote: > On 20/09/2023 21:31, Jason Gunthorpe wrote: > > On Mon, Sep 11, 2023 at 12:38:47PM +0300, Yishai Hadas wrote: > >> This series adds 'chunk mode' support for mlx5 driver upon the migration > >> flow. > >> > >> Before this series, we were limited to 4GB state size, as of the 4 bytes > >> max value based on the device specification for the query/save/load > >> commands. > >> > >> Once the device supports 'chunk mode' the driver can support state size > >> which is larger than 4GB. > >> > >> In that case, the device has the capability to split a single image to > >> multiple chunks as long as the software provides a buffer in the minimum > >> size reported by the device. > >> > >> The driver should query for the minimum buffer size required using > >> QUERY_VHCA_MIGRATION_STATE command with the 'chunk' bit set in its > >> input, in that case, the output will include both the minimum buffer > >> size and also the remaining total size to be reported/used where it will > >> be applicable. > >> > >> Upon chunk mode, there may be multiple images that will be read from the > >> device upon STOP_COPY. The driver will read ahead from the firmware the > >> full state in small/optimized chunks while letting QEMU/user space read > >> in parallel the available data. > >> > >> The chunk buffer size is picked up based on the minimum size that > >> firmware requires, the total full size and some max value in the driver > >> code which was set to 8MB to achieve some optimized downtime in the > >> general case. > >> > >> With that series in place, we could migrate successfully a device state > >> with a larger size than 4GB, while even improving the downtime in some > >> scenarios. > >> > >> Note: > >> As the first patch should go to net/mlx5 we may need to send it as a > >> pull request format to VFIO to avoid conflicts before acceptance. > >> > >> Yishai > >> > >> Yishai Hadas (9): > >> net/mlx5: Introduce ifc bits for migration in a chunk mode > >> vfio/mlx5: Wake up the reader post of disabling the SAVING migration > >> file > >> vfio/mlx5: Refactor the SAVE callback to activate a work only upon an > >> error > >> vfio/mlx5: Enable querying state size which is > 4GB > >> vfio/mlx5: Rename some stuff to match chunk mode > >> vfio/mlx5: Pre-allocate chunks for the STOP_COPY phase > >> vfio/mlx5: Add support for SAVING in chunk mode > >> vfio/mlx5: Add support for READING in chunk mode > >> vfio/mlx5: Activate the chunk mode functionality > > I didn't check in great depth but this looks OK to me > > > > Reviewed-by: Jason Gunthorpe <jgg@xxxxxxxxxx> > > Thanks Jason > > > > > I think this is a good design to start motivating more qmeu > > improvements, eg using io_uring as we could go further in the driver > > to optimize with that kind of support. > > > > Jason > > Alex, > > Can we move forward with the series and send a PR for the first patch > that needs to go also to net/mlx5 ? Yeah, I don't spot any issues with it either. Thanks, Alex
