On Thu, 2023-12-21 at 13:08 +0100, Eugenio Perez Martin wrote: > On Thu, Dec 21, 2023 at 12:52 PM Dragos Tatulea <dtatulea@xxxxxxxxxx> wrote: > > > > On Thu, 2023-12-21 at 08:46 +0100, Eugenio Perez Martin wrote: > > > On Thu, Dec 21, 2023 at 3:03 AM Jason Wang <jasowang@xxxxxxxxxx> wrote: > > > > > > > > On Wed, Dec 20, 2023 at 9:32 PM Eugenio Perez Martin > > > > <eperezma@xxxxxxxxxx> wrote: > > > > > > > > > > On Wed, Dec 20, 2023 at 5:06 AM Jason Wang <jasowang@xxxxxxxxxx> wrote: > > > > > > > > > > > > On Wed, Dec 20, 2023 at 11:46 AM Jason Wang <jasowang@xxxxxxxxxx> wrote: > > > > > > > > > > > > > > On Wed, Dec 20, 2023 at 2:09 AM Dragos Tatulea <dtatulea@xxxxxxxxxx> wrote: > > > > > > > > > > > > > > > > The virtio spec doesn't allow changing virtqueue addresses after > > > > > > > > DRIVER_OK. Some devices do support this operation when the device is > > > > > > > > suspended. The VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND flag > > > > > > > > advertises this support as a backend features. > > > > > > > > > > > > > > There's an ongoing effort in virtio spec to introduce the suspend state. > > > > > > > > > > > > > > So I wonder if it's better to just allow such behaviour? > > > > > > > > > > > > Actually I mean, allow drivers to modify the parameters during suspend > > > > > > without a new feature. > > > > > > > > > > > > > > > > That would be ideal, but how do userland checks if it can suspend + > > > > > change properties + resume? > > > > > > > > As discussed, it looks to me the only device that supports suspend is > > > > simulator and it supports change properties. > > > > > > > > E.g: > > > > > > > > static int vdpasim_set_vq_address(struct vdpa_device *vdpa, u16 idx, > > > > u64 desc_area, u64 driver_area, > > > > u64 device_area) > > > > { > > > > struct vdpasim *vdpasim = vdpa_to_sim(vdpa); > > > > struct vdpasim_virtqueue *vq = &vdpasim->vqs[idx]; > > > > > > > > vq->desc_addr = desc_area; > > > > vq->driver_addr = driver_area; > > > > vq->device_addr = device_area; > > > > > > > > return 0; > > > > } > > > > > > > > > > So in the current kernel master it is valid to set a different vq > > > address while the device is suspended in vdpa_sim. But it is not valid > > > in mlx5, as the FW will not be updated in resume (Dragos, please > > > correct me if I'm wrong). Both of them return success. > > > > > In the current state, there is no resume. HW Virtqueues will just get re-created > > with the new address. > > > > Oh, then all of this is effectively transparent to the userspace > except for the time it takes? > Not quite: mlx5_vdpa_set_vq_address will save the vq address only on the SW vq representation. Only later will it will call into the FW to update the FW. Later means: - On DRIVER_OK state, when the VQs get created. - On .set_map when the VQs get re-created (before this series) / updated (after this series) - On .resume (after this series). So if the .set_vq_address is called when the VQ is in DRIVER_OK but not suspended those addresses will be set later for later. > In that case you're right, we don't need feature flags. But I think it > would be great to also move the error return in case userspace tries > to modify vq parameters out of suspend state. > On the driver side or on the core side? Thanks > Thanks! > > > > > How can we know in the destination QEMU if it is valid to suspend & > > > set address? Should we handle this as a bugfix and backport the > > > change? > > > > > > > > > > > > > The only way that comes to my mind is to make sure all parents return > > > > > error if userland tries to do it, and then fallback in userland. > > > > > > > > Yes. > > > > > > > > > I'm > > > > > ok with that, but I'm not sure if the current master & previous kernel > > > > > has a coherent behavior. Do they return error? Or return success > > > > > without changing address / vq state? > > > > > > > > We probably don't need to worry too much here, as e.g set_vq_address > > > > could fail even without suspend (just at uAPI level). > > > > > > > > > > I don't get this, sorry. I rephrased my point with an example earlier > > > in the mail. > > > > > >