On 20/09/2023 21:31, Jason Gunthorpe wrote:
On Mon, Sep 11, 2023 at 12:38:47PM +0300, Yishai Hadas wrote:
This series adds 'chunk mode' support for mlx5 driver upon the migration
flow.
Before this series, we were limited to 4GB state size, as of the 4 bytes
max value based on the device specification for the query/save/load
commands.
Once the device supports 'chunk mode' the driver can support state size
which is larger than 4GB.
In that case, the device has the capability to split a single image to
multiple chunks as long as the software provides a buffer in the minimum
size reported by the device.
The driver should query for the minimum buffer size required using
QUERY_VHCA_MIGRATION_STATE command with the 'chunk' bit set in its
input, in that case, the output will include both the minimum buffer
size and also the remaining total size to be reported/used where it will
be applicable.
Upon chunk mode, there may be multiple images that will be read from the
device upon STOP_COPY. The driver will read ahead from the firmware the
full state in small/optimized chunks while letting QEMU/user space read
in parallel the available data.
The chunk buffer size is picked up based on the minimum size that
firmware requires, the total full size and some max value in the driver
code which was set to 8MB to achieve some optimized downtime in the
general case.
With that series in place, we could migrate successfully a device state
with a larger size than 4GB, while even improving the downtime in some
scenarios.
Note:
As the first patch should go to net/mlx5 we may need to send it as a
pull request format to VFIO to avoid conflicts before acceptance.
Yishai
Yishai Hadas (9):
net/mlx5: Introduce ifc bits for migration in a chunk mode
vfio/mlx5: Wake up the reader post of disabling the SAVING migration
file
vfio/mlx5: Refactor the SAVE callback to activate a work only upon an
error
vfio/mlx5: Enable querying state size which is > 4GB
vfio/mlx5: Rename some stuff to match chunk mode
vfio/mlx5: Pre-allocate chunks for the STOP_COPY phase
vfio/mlx5: Add support for SAVING in chunk mode
vfio/mlx5: Add support for READING in chunk mode
vfio/mlx5: Activate the chunk mode functionality
I didn't check in great depth but this looks OK to me
Reviewed-by: Jason Gunthorpe <jgg@xxxxxxxxxx>
Thanks Jason
I think this is a good design to start motivating more qmeu
improvements, eg using io_uring as we could go further in the driver
to optimize with that kind of support.
Jason
Alex,
Can we move forward with the series and send a PR for the first patch
that needs to go also to net/mlx5 ?
Thanks,
Yishai
