On Fri, Dec 25, 2020 at 3:02 PM Jason Wang <jasowang@xxxxxxxxxx> wrote: > > > On 2020/12/25 上午10:37, Yongji Xie wrote: > > On Thu, Dec 24, 2020 at 3:37 PM Yongji Xie <xieyongji@xxxxxxxxxxxxx> wrote: > >> On Thu, Dec 24, 2020 at 10:41 AM Jason Wang <jasowang@xxxxxxxxxx> wrote: > >>> > >>> On 2020/12/23 下午8:14, Yongji Xie wrote: > >>>> On Wed, Dec 23, 2020 at 5:05 PM Jason Wang <jasowang@xxxxxxxxxx> wrote: > >>>>> On 2020/12/22 下午10:52, Xie Yongji wrote: > >>>>>> To support vhost-vdpa bus driver, we need a way to share the > >>>>>> vhost-vdpa backend process's memory with the userspace VDUSE process. > >>>>>> > >>>>>> This patch tries to make use of the vhost iotlb message to achieve > >>>>>> that. We will get the shm file from the iotlb message and pass it > >>>>>> to the userspace VDUSE process. > >>>>>> > >>>>>> Signed-off-by: Xie Yongji <xieyongji@xxxxxxxxxxxxx> > >>>>>> --- > >>>>>> Documentation/driver-api/vduse.rst | 15 +++- > >>>>>> drivers/vdpa/vdpa_user/vduse_dev.c | 147 ++++++++++++++++++++++++++++++++++++- > >>>>>> include/uapi/linux/vduse.h | 11 +++ > >>>>>> 3 files changed, 171 insertions(+), 2 deletions(-) > >>>>>> > >>>>>> diff --git a/Documentation/driver-api/vduse.rst b/Documentation/driver-api/vduse.rst > >>>>>> index 623f7b040ccf..48e4b1ba353f 100644 > >>>>>> --- a/Documentation/driver-api/vduse.rst > >>>>>> +++ b/Documentation/driver-api/vduse.rst > >>>>>> @@ -46,13 +46,26 @@ The following types of messages are provided by the VDUSE framework now: > >>>>>> > >>>>>> - VDUSE_GET_CONFIG: Read from device specific configuration space > >>>>>> > >>>>>> +- VDUSE_UPDATE_IOTLB: Update the memory mapping in device IOTLB > >>>>>> + > >>>>>> +- VDUSE_INVALIDATE_IOTLB: Invalidate the memory mapping in device IOTLB > >>>>>> + > >>>>>> Please see include/linux/vdpa.h for details. > >>>>>> > >>>>>> -In the data path, VDUSE framework implements a MMU-based on-chip IOMMU > >>>>>> +The data path of userspace vDPA device is implemented in different ways > >>>>>> +depending on the vdpa bus to which it is attached. > >>>>>> + > >>>>>> +In virtio-vdpa case, VDUSE framework implements a MMU-based on-chip IOMMU > >>>>>> driver which supports mapping the kernel dma buffer to a userspace iova > >>>>>> region dynamically. The userspace iova region can be created by passing > >>>>>> the userspace vDPA device fd to mmap(2). > >>>>>> > >>>>>> +In vhost-vdpa case, the dma buffer is reside in a userspace memory region > >>>>>> +which will be shared to the VDUSE userspace processs via the file > >>>>>> +descriptor in VDUSE_UPDATE_IOTLB message. And the corresponding address > >>>>>> +mapping (IOVA of dma buffer <-> VA of the memory region) is also included > >>>>>> +in this message. > >>>>>> + > >>>>>> Besides, the eventfd mechanism is used to trigger interrupt callbacks and > >>>>>> receive virtqueue kicks in userspace. The following ioctls on the userspace > >>>>>> vDPA device fd are provided to support that: > >>>>>> diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c > >>>>>> index b974333ed4e9..d24aaacb6008 100644 > >>>>>> --- a/drivers/vdpa/vdpa_user/vduse_dev.c > >>>>>> +++ b/drivers/vdpa/vdpa_user/vduse_dev.c > >>>>>> @@ -34,6 +34,7 @@ > >>>>>> > >>>>>> struct vduse_dev_msg { > >>>>>> struct vduse_dev_request req; > >>>>>> + struct file *iotlb_file; > >>>>>> struct vduse_dev_response resp; > >>>>>> struct list_head list; > >>>>>> wait_queue_head_t waitq; > >>>>>> @@ -325,12 +326,80 @@ static int vduse_dev_set_vq_state(struct vduse_dev *dev, > >>>>>> return ret; > >>>>>> } > >>>>>> > >>>>>> +static int vduse_dev_update_iotlb(struct vduse_dev *dev, struct file *file, > >>>>>> + u64 offset, u64 iova, u64 size, u8 perm) > >>>>>> +{ > >>>>>> + struct vduse_dev_msg *msg; > >>>>>> + int ret; > >>>>>> + > >>>>>> + if (!size) > >>>>>> + return -EINVAL; > >>>>>> + > >>>>>> + msg = vduse_dev_new_msg(dev, VDUSE_UPDATE_IOTLB); > >>>>>> + msg->req.size = sizeof(struct vduse_iotlb); > >>>>>> + msg->req.iotlb.offset = offset; > >>>>>> + msg->req.iotlb.iova = iova; > >>>>>> + msg->req.iotlb.size = size; > >>>>>> + msg->req.iotlb.perm = perm; > >>>>>> + msg->req.iotlb.fd = -1; > >>>>>> + msg->iotlb_file = get_file(file); > >>>>>> + > >>>>>> + ret = vduse_dev_msg_sync(dev, msg); > >>>>> My feeling is that we should provide consistent API for the userspace > >>>>> device to use. > >>>>> > >>>>> E.g we'd better carry the IOTLB message for both virtio/vhost drivers. > >>>>> > >>>>> It looks to me for virtio drivers we can still use UPDAT_IOTLB message > >>>>> by using VDUSE file as msg->iotlb_file here. > >>>>> > >>>> It's OK for me. One problem is when to transfer the UPDATE_IOTLB > >>>> message in virtio cases. > >>> > >>> Instead of generating IOTLB messages for userspace. > >>> > >>> How about record the mappings (which is a common case for device have > >>> on-chip IOMMU e.g mlx5e and vdpa simlator), then we can introduce ioctl > >>> for userspace to query? > >>> > >> If so, the IOTLB UPDATE is actually triggered by ioctl, but > >> IOTLB_INVALIDATE is triggered by the message. Is it a little odd? Or > >> how about trigger it when userspace call mmap() on the device fd? > >> > > Oh sorry, looks like mmap() needs to be called in IOTLB UPDATE message > > handler. Is it possible for the vdpa device to know which vdpa bus it > > is attached to? > > > We'd better not. It's kind of layer violation. > OK. Now I think both ioctl and message are needed. The ioctl is useful when VDUSE userspace daemon reboot. And the IOTLB_UPDATE message could be generated during the first DMA mapping in the virtio-vdpa case. Thanks, Yongji