On Fri, Dec 25, 2020 at 2:58 PM Jason Wang <jasowang@xxxxxxxxxx> wrote: > > > On 2020/12/24 下午3:37, Yongji Xie wrote: > > On Thu, Dec 24, 2020 at 10:41 AM Jason Wang <jasowang@xxxxxxxxxx> wrote: > >> > >> On 2020/12/23 下午8:14, Yongji Xie wrote: > >>> On Wed, Dec 23, 2020 at 5:05 PM Jason Wang <jasowang@xxxxxxxxxx> wrote: > >>>> On 2020/12/22 下午10:52, Xie Yongji wrote: > >>>>> To support vhost-vdpa bus driver, we need a way to share the > >>>>> vhost-vdpa backend process's memory with the userspace VDUSE process. > >>>>> > >>>>> This patch tries to make use of the vhost iotlb message to achieve > >>>>> that. We will get the shm file from the iotlb message and pass it > >>>>> to the userspace VDUSE process. > >>>>> > >>>>> Signed-off-by: Xie Yongji <xieyongji@xxxxxxxxxxxxx> > >>>>> --- > >>>>> Documentation/driver-api/vduse.rst | 15 +++- > >>>>> drivers/vdpa/vdpa_user/vduse_dev.c | 147 ++++++++++++++++++++++++++++++++++++- > >>>>> include/uapi/linux/vduse.h | 11 +++ > >>>>> 3 files changed, 171 insertions(+), 2 deletions(-) > >>>>> > >>>>> diff --git a/Documentation/driver-api/vduse.rst b/Documentation/driver-api/vduse.rst > >>>>> index 623f7b040ccf..48e4b1ba353f 100644 > >>>>> --- a/Documentation/driver-api/vduse.rst > >>>>> +++ b/Documentation/driver-api/vduse.rst > >>>>> @@ -46,13 +46,26 @@ The following types of messages are provided by the VDUSE framework now: > >>>>> > >>>>> - VDUSE_GET_CONFIG: Read from device specific configuration space > >>>>> > >>>>> +- VDUSE_UPDATE_IOTLB: Update the memory mapping in device IOTLB > >>>>> + > >>>>> +- VDUSE_INVALIDATE_IOTLB: Invalidate the memory mapping in device IOTLB > >>>>> + > >>>>> Please see include/linux/vdpa.h for details. > >>>>> > >>>>> -In the data path, VDUSE framework implements a MMU-based on-chip IOMMU > >>>>> +The data path of userspace vDPA device is implemented in different ways > >>>>> +depending on the vdpa bus to which it is attached. > >>>>> + > >>>>> +In virtio-vdpa case, VDUSE framework implements a MMU-based on-chip IOMMU > >>>>> driver which supports mapping the kernel dma buffer to a userspace iova > >>>>> region dynamically. The userspace iova region can be created by passing > >>>>> the userspace vDPA device fd to mmap(2). > >>>>> > >>>>> +In vhost-vdpa case, the dma buffer is reside in a userspace memory region > >>>>> +which will be shared to the VDUSE userspace processs via the file > >>>>> +descriptor in VDUSE_UPDATE_IOTLB message. And the corresponding address > >>>>> +mapping (IOVA of dma buffer <-> VA of the memory region) is also included > >>>>> +in this message. > >>>>> + > >>>>> Besides, the eventfd mechanism is used to trigger interrupt callbacks and > >>>>> receive virtqueue kicks in userspace. The following ioctls on the userspace > >>>>> vDPA device fd are provided to support that: > >>>>> diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c > >>>>> index b974333ed4e9..d24aaacb6008 100644 > >>>>> --- a/drivers/vdpa/vdpa_user/vduse_dev.c > >>>>> +++ b/drivers/vdpa/vdpa_user/vduse_dev.c > >>>>> @@ -34,6 +34,7 @@ > >>>>> > >>>>> struct vduse_dev_msg { > >>>>> struct vduse_dev_request req; > >>>>> + struct file *iotlb_file; > >>>>> struct vduse_dev_response resp; > >>>>> struct list_head list; > >>>>> wait_queue_head_t waitq; > >>>>> @@ -325,12 +326,80 @@ static int vduse_dev_set_vq_state(struct vduse_dev *dev, > >>>>> return ret; > >>>>> } > >>>>> > >>>>> +static int vduse_dev_update_iotlb(struct vduse_dev *dev, struct file *file, > >>>>> + u64 offset, u64 iova, u64 size, u8 perm) > >>>>> +{ > >>>>> + struct vduse_dev_msg *msg; > >>>>> + int ret; > >>>>> + > >>>>> + if (!size) > >>>>> + return -EINVAL; > >>>>> + > >>>>> + msg = vduse_dev_new_msg(dev, VDUSE_UPDATE_IOTLB); > >>>>> + msg->req.size = sizeof(struct vduse_iotlb); > >>>>> + msg->req.iotlb.offset = offset; > >>>>> + msg->req.iotlb.iova = iova; > >>>>> + msg->req.iotlb.size = size; > >>>>> + msg->req.iotlb.perm = perm; > >>>>> + msg->req.iotlb.fd = -1; > >>>>> + msg->iotlb_file = get_file(file); > >>>>> + > >>>>> + ret = vduse_dev_msg_sync(dev, msg); > >>>> My feeling is that we should provide consistent API for the userspace > >>>> device to use. > >>>> > >>>> E.g we'd better carry the IOTLB message for both virtio/vhost drivers. > >>>> > >>>> It looks to me for virtio drivers we can still use UPDAT_IOTLB message > >>>> by using VDUSE file as msg->iotlb_file here. > >>>> > >>> It's OK for me. One problem is when to transfer the UPDATE_IOTLB > >>> message in virtio cases. > >> > >> Instead of generating IOTLB messages for userspace. > >> > >> How about record the mappings (which is a common case for device have > >> on-chip IOMMU e.g mlx5e and vdpa simlator), then we can introduce ioctl > >> for userspace to query? > >> > > If so, the IOTLB UPDATE is actually triggered by ioctl, but > > IOTLB_INVALIDATE is triggered by the message. Is it a little odd? > > > Good point. > > Some questions here: > > 1) Userspace think the IOTLB was flushed after IOTLB_INVALIDATE syscall > is returned. But this patch doesn't have this guarantee. Can this lead > some issues? I'm not sure. But should it be guaranteed in VDUSE userspace? Just like what vhost-user backend process does. > 2) I think after VDUSE userspace receives IOTLB_INVALIDATE, it needs to > issue a munmap(). What if it doesn't do that? > Yes, the munmap() is needed. If it doesn't do that, VDUSE userspace could still access the corresponding guest memory region. Thanks, Yongji