On Thu, Dec 17, 2020 at 11:30:18AM +0800, Jason Wang wrote: > > On 2020/12/16 下午5:47, Michael S. Tsirkin wrote: > > On Wed, Dec 16, 2020 at 02:47:57PM +0800, Jason Wang wrote: > > > Hi All: > > > > > > This series tries to add the support for control virtqueue in vDPA. > > > > > > Control virtqueue is used by networking device for accepting various > > > commands from the driver. It's a must to support multiqueue and other > > > configurations. > > > > > > When used by vhost-vDPA bus driver for VM, the control virtqueue > > > should be shadowed via userspace VMM (Qemu) instead of being assigned > > > directly to Guest. This is because Qemu needs to know the device state > > > in order to start and stop device correctly (e.g for Live Migration). > > > > > > This requies to isolate the memory mapping for control virtqueue > > > presented by vhost-vDPA to prevent guest from accesing it directly. > > > To achieve this, vDPA introduce two new abstractions: > > > > > > - address space: identified through address space id (ASID) and a set > > > of memory mapping in maintained > > > - virtqueue group: the minimal set of virtqueues that must share an > > > address space > > How will this support the pretty common case where control vq > > is programmed by the kernel through the PF, and others by the VFs? > > > In this case, the VF parent need to provide a software control vq and decode > the command then send them to VF. But how does that tie to the address space infrastructure? > > > > > > > I actually thought the way to support it is by exposing > > something like an "inject buffers" API which sends data to a given VQ. > > Maybe an ioctl, and maybe down the road uio ring can support batching > > these .... > > > So the virtuqueue allows the request to be processed asynchronously (e.g > driver may choose to use interrupt for control vq). This means we need to > support that in uAPI level. I don't think we need to make it async, just a regular ioctl will do. In fact no guest uses the asynchronous property. > And if we manage to do that, it's just another > type of virtqueue. > > For virtio-vDPA, this also means the extensions for queue processing which > is a functional duplication. I don't see why, just send it to the actual control vq :) > Using what proposed in this series, we don't > need any changes for kernel virtio drivers. > > What's more important, this series could be used for future features that > requires DMA isolation between virtqueues: > > - report dirty pages via virtqueue > - sub function level device slicing I agree these are nice to have, but I am not sure basic control vq must be tied to that. > ... > > Thanks > > > > > > > > > Device needs to advertise the following attributes to vDPA: > > > > > > - the number of address spaces supported in the device > > > - the number of virtqueue groups supported in the device > > > - the mappings from a specific virtqueue to its virtqueue groups > > > > > > The mappings from virtqueue to virtqueue groups is fixed and defined > > > by vDPA device driver. E.g: > > > > > > - For the device that has hardware ASID support, it can simply > > > advertise a per virtqueue virtqueue group. > > > - For the device that does not have hardware ASID support, it can > > > simply advertise a single virtqueue group that contains all > > > virtqueues. Or if it wants a software emulated control virtqueue, it > > > can advertise two virtqueue groups, one is for cvq, another is for > > > the rest virtqueues. > > > > > > vDPA also allow to change the association between virtqueue group and > > > address space. So in the case of control virtqueue, userspace > > > VMM(Qemu) may use a dedicated address space for the control virtqueue > > > group to isolate the memory mapping. > > > > > > The vhost/vhost-vDPA is also extend for the userspace to: > > > > > > - query the number of virtqueue groups and address spaces supported by > > > the device > > > - query the virtqueue group for a specific virtqueue > > > - assocaite a virtqueue group with an address space > > > - send ASID based IOTLB commands > > > > > > This will help userspace VMM(Qemu) to detect whether the control vq > > > could be supported and isolate memory mappings of control virtqueue > > > from the others. > > > > > > To demonstrate the usage, vDPA simulator is extended to support > > > setting MAC address via a emulated control virtqueue. > > > > > > Please review. > > > > > > Changes since RFC: > > > > > > - tweak vhost uAPI documentation > > > - switch to use device specific IOTLB really in patch 4 > > > - tweak the commit log > > > - fix that ASID in vhost is claimed to be 32 actually but 16bit > > > actually > > > - fix use after free when using ASID with IOTLB batching requests > > > - switch to use Stefano's patch for having separated iov > > > - remove unused "used_as" variable > > > - fix the iotlb/asid checking in vhost_vdpa_unmap() > > > > > > Thanks > > > > > > Jason Wang (20): > > > vhost: move the backend feature bits to vhost_types.h > > > virtio-vdpa: don't set callback if virtio doesn't need it > > > vhost-vdpa: passing iotlb to IOMMU mapping helpers > > > vhost-vdpa: switch to use vhost-vdpa specific IOTLB > > > vdpa: add the missing comment for nvqs in struct vdpa_device > > > vdpa: introduce virtqueue groups > > > vdpa: multiple address spaces support > > > vdpa: introduce config operations for associating ASID to a virtqueue > > > group > > > vhost_iotlb: split out IOTLB initialization > > > vhost: support ASID in IOTLB API > > > vhost-vdpa: introduce asid based IOTLB > > > vhost-vdpa: introduce uAPI to get the number of virtqueue groups > > > vhost-vdpa: introduce uAPI to get the number of address spaces > > > vhost-vdpa: uAPI to get virtqueue group id > > > vhost-vdpa: introduce uAPI to set group ASID > > > vhost-vdpa: support ASID based IOTLB API > > > vdpa_sim: advertise VIRTIO_NET_F_MTU > > > vdpa_sim: factor out buffer completion logic > > > vdpa_sim: filter destination mac address > > > vdpasim: control virtqueue support > > > > > > Stefano Garzarella (1): > > > vdpa_sim: split vdpasim_virtqueue's iov field in out_iov and in_iov > > > > > > drivers/vdpa/ifcvf/ifcvf_main.c | 9 +- > > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 11 +- > > > drivers/vdpa/vdpa.c | 8 +- > > > drivers/vdpa/vdpa_sim/vdpa_sim.c | 292 ++++++++++++++++++++++++------ > > > drivers/vhost/iotlb.c | 23 ++- > > > drivers/vhost/vdpa.c | 246 ++++++++++++++++++++----- > > > drivers/vhost/vhost.c | 23 ++- > > > drivers/vhost/vhost.h | 4 +- > > > drivers/virtio/virtio_vdpa.c | 2 +- > > > include/linux/vdpa.h | 42 ++++- > > > include/linux/vhost_iotlb.h | 2 + > > > include/uapi/linux/vhost.h | 25 ++- > > > include/uapi/linux/vhost_types.h | 10 +- > > > 13 files changed, 561 insertions(+), 136 deletions(-) > > > > > > -- > > > 2.25.1 _______________________________________________ Virtualization mailing list Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx https://lists.linuxfoundation.org/mailman/listinfo/virtualization
