On Mon, 5 Dec 2022 16:48:34 +0200
Yishai Hadas <yishaih@xxxxxxxxxx> wrote:
> vfio precopy ioctl returns an estimation of data available for
> transferring from the device.
>
> Whenever a user is using VFIO_MIG_GET_PRECOPY_INFO, track the current
> state of the device, and if needed, append the dirty data to the
> transfer FD data. This is done by saving a middle state.
>
> As mlx5 runs the SAVE command asynchronously, make sure to query for
> incremental data only once there is no active save command.
> Running both in parallel, might end-up with a failure in the incremental
> query command on un-tracked vhca.
>
> Also, a middle state will be saved only after the previous state has
> finished its SAVE command and has been fully transferred, this prevents
> endless use resources.
>
> Co-developed-by: Shay Drory <shayd@xxxxxxxxxx>
> Signed-off-by: Shay Drory <shayd@xxxxxxxxxx>
> Reviewed-by: Jason Gunthorpe <jgg@xxxxxxxxxx>
> Signed-off-by: Yishai Hadas <yishaih@xxxxxxxxxx>
> ---
> drivers/vfio/pci/mlx5/cmd.c | 16 +++++
> drivers/vfio/pci/mlx5/main.c | 111 +++++++++++++++++++++++++++++++++++
> 2 files changed, 127 insertions(+)
>
> diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c
> index 160fa38fc78d..12e74ecebe64 100644
> --- a/drivers/vfio/pci/mlx5/cmd.c
> +++ b/drivers/vfio/pci/mlx5/cmd.c
> @@ -67,12 +67,25 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev,
> {
> u32 out[MLX5_ST_SZ_DW(query_vhca_migration_state_out)] = {};
> u32 in[MLX5_ST_SZ_DW(query_vhca_migration_state_in)] = {};
> + bool inc = query_flags & MLX5VF_QUERY_INC;
> int ret;
>
> lockdep_assert_held(&mvdev->state_mutex);
> if (mvdev->mdev_detach)
> return -ENOTCONN;
>
> + /*
> + * In case PRE_COPY is used, saving_migf is exposed while device is
> + * running. Make sure to run only once there is no active save command.
> + * Running both in parallel, might end-up with a failure in the
> + * incremental query command on un-tracked vhca.
> + */
> + if (inc) {
> + ret = wait_for_completion_interruptible(&mvdev->saving_migf->save_comp);
> + if (ret)
> + return ret;
> + }
> +
> MLX5_SET(query_vhca_migration_state_in, in, opcode,
> MLX5_CMD_OP_QUERY_VHCA_MIGRATION_STATE);
> MLX5_SET(query_vhca_migration_state_in, in, vhca_id, mvdev->vhca_id);
> @@ -82,6 +95,9 @@ int mlx5vf_cmd_query_vhca_migration_state(struct mlx5vf_pci_core_device *mvdev,
>
> ret = mlx5_cmd_exec_inout(mvdev->mdev, query_vhca_migration_state, in,
> out);
> + if (inc)
> + complete(&mvdev->saving_migf->save_comp);
> +
> if (ret)
> return ret;
>
> diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c
> index 9a36e36ec33b..08c7d96e92b7 100644
> --- a/drivers/vfio/pci/mlx5/main.c
> +++ b/drivers/vfio/pci/mlx5/main.c
> @@ -294,10 +294,121 @@ static void mlx5vf_mark_err(struct mlx5_vf_migration_file *migf)
> wake_up_interruptible(&migf->poll_wait);
> }
>
> +static ssize_t mlx5vf_precopy_ioctl(struct file *filp, unsigned int cmd,
> + unsigned long arg)
ssize_t is incompatible with file_operations.unlocked_ioctl in 32-bit
builds (i386):
drivers/vfio/pci/mlx5/main.c:419:27: error: initialization of ‘long int (*)(struct file *, unsigned int, long unsigned int)’ from incompatible pointer type ‘ssize_t (*)(struct file *, unsigned int, long unsigned int)’ {aka ‘int (*)(struct file *, unsigned int, long unsigned int)’} [-Werror=incompatible-pointer-types]
419 | .unlocked_ioctl = mlx5vf_precopy_ioctl,
| ^~~~~~~~~~~~~~~~~~~~
Thanks,
Alex
> static const struct file_operations mlx5vf_save_fops = {
> .owner = THIS_MODULE,
> .read = mlx5vf_save_read,
> .poll = mlx5vf_save_poll,
> + .unlocked_ioctl = mlx5vf_precopy_ioctl,
> + .compat_ioctl = compat_ptr_ioctl,
> .release = mlx5vf_release_file,
> .llseek = no_llseek,
> };
