On Mon, Feb 13, 2023 at 02:13:59PM -0500, Stefan Hajnoczi wrote: > On Mon, Feb 13, 2023 at 11:47:31AM +0800, Ming Lei wrote: > > On Wed, Feb 08, 2023 at 07:17:10AM -0500, Stefan Hajnoczi wrote: > > > On Wed, Feb 08, 2023 at 10:12:19AM +0800, Ming Lei wrote: > > > > On Mon, Feb 06, 2023 at 03:27:09PM -0500, Stefan Hajnoczi wrote: > > > > > On Mon, Feb 06, 2023 at 11:00:27PM +0800, Ming Lei wrote: > > > > > > Hello, > > > > > > > > > > > > So far UBLK is only used for implementing virtual block device from > > > > > > userspace, such as loop, nbd, qcow2, ...[1]. > > > > > > > > > > I won't be at LSF/MM so here are my thoughts: > > > > > > > > Thanks for the thoughts, :-) > > > > > > > > > > > > > > > > > > > > > It could be useful for UBLK to cover real storage hardware too: > > > > > > > > > > > > - for fast prototype or performance evaluation > > > > > > > > > > > > - some network storages are attached to host, such as iscsi and nvme-tcp, > > > > > > the current UBLK interface doesn't support such devices, since it needs > > > > > > all LUNs/Namespaces to share host resources(such as tag) > > > > > > > > > > Can you explain this in more detail? It seems like an iSCSI or > > > > > NVMe-over-TCP initiator could be implemented as a ublk server today. > > > > > What am I missing? > > > > > > > > The current ublk can't do that yet, because the interface doesn't > > > > support multiple ublk disks sharing single host, which is exactly > > > > the case of scsi and nvme. > > > > > > Can you give an example that shows exactly where a problem is hit? > > > > > > I took a quick look at the ublk source code and didn't spot a place > > > where it prevents a single ublk server process from handling multiple > > > devices. > > > > > > Regarding "host resources(such as tag)", can the ublk server deal with > > > that in userspace? The Linux block layer doesn't have the concept of a > > > "host", that would come in at the SCSI/NVMe level that's implemented in > > > userspace. > > > > > > I don't understand yet... > > > > blk_mq_tag_set is embedded into driver host structure, and referred by queue > > via q->tag_set, both scsi and nvme allocates tag in host/queue wide, > > that said all LUNs/NSs share host/queue tags, current every ublk > > device is independent, and can't shard tags. > > Does this actually prevent ublk servers with multiple ublk devices or is > it just sub-optimal? It is former, ublk can't support multiple devices which share single host because duplicated tag can be seen in host side, then io is failed. > > Also, is this specific to real storage hardware? I guess userspace > NVMe-over-TCP or iSCSI initiators would be affected regardless of > whether they simply use the Sockets API (software) or userspace device > drivers (hardware). > > Sorry for all these questions, I think I'm a little confused because you > said "doesn't support such devices" and I thought this discussion was > about real storage hardware. Neither of these seem to apply to the > tag_set issue. The reality is that both scsi and nvme(either virt or real hardware) supports multi LUNs/NSs, so tag_set issue has to be solved, or multi-LUNs/NSs has to be supported. > > > > > > > > > > > > > > > > > > > > > > > > > > > - SPDK has supported user space driver for real hardware > > > > > > > > > > I think this could already be implemented today. There will be extra > > > > > memory copies because SPDK won't have access to the application's memory > > > > > pages. > > > > > > > > Here I proposed zero copy, and current SPDK nvme-pci implementation haven't > > > > such extra copy per my understanding. > > > > > > > > > > > > > > > > > > > > > So propose to extend UBLK for supporting real hardware device: > > > > > > > > > > > > 1) extend UBLK ABI interface to support disks attached to host, such > > > > > > as SCSI Luns/NVME Namespaces > > > > > > > > > > > > 2) the followings are related with operating hardware from userspace, > > > > > > so userspace driver has to be trusted, and root is required, and > > > > > > can't support unprivileged UBLK device > > > > > > > > > > Linux VFIO provides a safe userspace API for userspace device drivers. > > > > > That means memory and interrupts are isolated. Neither userspace nor the > > > > > hardware device can access memory or interrupts that the userspace > > > > > process is not allowed to access. > > > > > > > > > > I think there are still limitations like all memory pages exposed to the > > > > > device need to be pinned. So effectively you might still need privileges > > > > > to get the mlock resource limits. > > > > > > > > > > But overall I think what you're saying about root and unprivileged ublk > > > > > devices is not true. Hardware support should be developed with the goal > > > > > of supporting unprivileged userspace ublk servers. > > > > > > > > > > Those unprivileged userspace ublk servers cannot claim any PCI device > > > > > they want. The user/admin will need to give them permission to open a > > > > > network card, SCSI HBA, etc. > > > > > > > > It depends on implementation, please see > > > > > > > > https://spdk.io/doc/userspace.html > > > > > > > > ``` > > > > The SPDK NVMe Driver, for instance, maps the BAR for the NVMe device and > > > > then follows along with the NVMe Specification to initialize the device, > > > > create queue pairs, and ultimately send I/O. > > > > ``` > > > > > > > > The above way needs userspace to operating hardware by the mapped BAR, > > > > which can't be allowed for unprivileged user. > > > > > > From https://spdk.io/doc/system_configuration.html: > > > > > > Running SPDK as non-privileged user > > > > > > One of the benefits of using the VFIO Linux kernel driver is the > > > ability to perform DMA operations with peripheral devices as > > > unprivileged user. The permissions to access particular devices still > > > need to be granted by the system administrator, but only on a one-time > > > basis. Note that this functionality is supported with DPDK starting > > > from version 18.11. > > > > > > This is what I had described in my previous reply. > > > > My reference on spdk were mostly from spdk/nvme doc. > > Just take quick look at spdk code, looks both vfio and direct > > programming hardware are supported: > > > > 1) lib/nvme/nvme_vfio_user.c > > const struct spdk_nvme_transport_ops vfio_ops { > > .qpair_submit_request = nvme_pcie_qpair_submit_request, > > Ignore this, it's the userspace vfio-user UNIX domain socket protocol > support. It's not kernel VFIO and is unrelated to what we're discussing. > More info on vfio-user: https://spdk.io/news/2021/05/04/vfio-user/ Not sure, why does .qpair_submit_request point to nvme_pcie_qpair_submit_request? > > > > > > > 2) lib/nvme/nvme_pcie.c > > const struct spdk_nvme_transport_ops pcie_ops = { > > .qpair_submit_request = nvme_pcie_qpair_submit_request > > nvme_pcie_qpair_submit_tracker > > nvme_pcie_qpair_submit_tracker > > nvme_pcie_qpair_ring_sq_doorbell > > > > but vfio dma isn't used in nvme_pcie_qpair_submit_request, and simply > > write/read mmaped mmio. > > I have only a small amount of SPDK code experienced, so this might be Me too. > wrong, but I think the NVMe PCI driver code does not need to directly > call VFIO APIs. That is handled by DPDK/SPDK's EAL operating system > abstractions and device driver APIs. > > DMA memory is mapped permanently so the device driver doesn't need to > perform individual map/unmap operations in the data path. NVMe PCI > request submission builds the NVMe command structures containing device > addresses (i.e. IOVAs when IOMMU is enabled). If IOMMU isn't used, it is physical address of memory. Then I guess you may understand why I said this way can't be done by un-privileged user, cause driver is writing memory physical address to device register directly. But other driver can follow this approach if the way is accepted. > > This code probably supports both IOMMU (VFIO) and non-IOMMU operation. > > > > > > > > > > > > > > > > > > > > > > > > > > > 3) how to operating hardware memory space > > > > > > - unbind kernel driver and rebind with uio/vfio > > > > > > - map PCI BAR into userspace[2], then userspace can operate hardware > > > > > > with mapped user address via MMIO > > > > > > > > > > > > 4) DMA > > > > > > - DMA requires physical memory address, UBLK driver actually has > > > > > > block request pages, so can we export request SG list(each segment > > > > > > physical address, offset, len) into userspace? If the max_segments > > > > > > limit is not too big(<=64), the needed buffer for holding SG list > > > > > > can be small enough. > > > > > > > > > > DMA with an IOMMU requires an I/O Virtual Address, not a CPU physical > > > > > address. The IOVA space is defined by the IOMMU page tables. Userspace > > > > > controls the IOMMU page tables via Linux VFIO ioctls. > > > > > > > > > > For example, <linux/vfio.h> struct vfio_iommu_type1_dma_map defines the > > > > > IOMMU mapping that makes a range of userspace virtual addresses > > > > > available at a given IOVA. > > > > > > > > > > Mapping and unmapping operations are not free. Similar to mmap(2), the > > > > > program will be slow if it does this frequently. > > > > > > > > Yeah, but SPDK shouldn't use vfio DMA interface, see: > > > > > > > > https://spdk.io/doc/memory.html > > > > > > > > they just programs DMA directly with physical address of pinned hugepages. > > > > > > From the page you linked: > > > > > > IOMMU Support > > > > > > ... > > > > > > This is a future-proof, hardware-accelerated solution for performing > > > DMA operations into and out of a user space process and forms the > > > long-term foundation for SPDK and DPDK's memory management strategy. > > > We highly recommend that applications are deployed using vfio and the > > > IOMMU enabled, which is fully supported today. > > > > > > Yes, SPDK supports running without IOMMU, but they recommend running > > > with the IOMMU. > > > > > > > > > > > > > > > > > I think it's effectively the same problem as ublk zero-copy. We want to > > > > > give the ublk server access to just the I/O buffers that it currently > > > > > needs, but doing so would be expensive :(. > > > > > > > > > > I think Linux has strategies for avoiding the expense like > > > > > iommu.strict=0 and swiotlb. The drawback is that in our case userspace > > > > > and/or the hardware device controller by userspace would still have > > > > > access to the memory pages after I/O has completed. This reduces memory > > > > > isolation :(. > > > > > > > > > > DPDK/SPDK and QEMU use long-lived Linux VFIO DMA mappings. > > > > > > > > Per the above SPDK links, the nvme-pci doesn't use vfio dma mapping. > > > > > > When using VFIO (recommended by the docs), SPDK uses long-lived DMA > > > mappings. Here are places in the SPDK/DPDK source code where VFIO DMA > > > mapping is used: > > > https://github.com/spdk/spdk/blob/master/lib/env_dpdk/memory.c#L1371 > > > https://github.com/spdk/dpdk/blob/e89c0845a60831864becc261cff48dd9321e7e79/lib/eal/linux/eal_vfio.c#L2164 > > > > I meant spdk nvme implementation. > > I did too. The NVMe PCI driver will use the PCI driver APIs and the EAL > (operating system abstraction) will deal with IOMMU APIs (VFIO) > transparently. > > > > > > > > > > > > > > > > > > > > What I'm trying to get at is that either memory isolation is compromised > > > > > or performance is reduced. It's hard to have good performance together > > > > > with memory isolation. > > > > > > > > > > I think ublk should follow the VFIO philosophy of being a safe > > > > > kernel/userspace interface. If userspace is malicious or buggy, the > > > > > kernel's and other process' memory should not be corrupted. > > > > > > > > It is tradeoff between performance and isolation, that is why I mention > > > > that directing programming hardware in userspace can be done by root > > > > only. > > > > > > Yes, there is a trade-off. Over the years the use of unsafe approaches > > > has been discouraged and replaced (/dev/kmem, uio -> VFIO, etc). As > > > secure boot, integrity architecture, and stuff like that becomes more > > > widely used, it's harder to include features that break memory isolation > > > in software in mainstream distros. There can be an option to sacrifice > > > memory isolation for performance and some users may be willing to accept > > > the trade-off. I think it should be an option feature though. > > > > > > I did want to point out that the statement that "direct programming > > > hardware in userspace can be done by root only" is false (see VFIO). > > > > Unfortunately not see vfio is used when spdk/nvme is operating hardware > > mmio. > > I think my responses above answered this, but just to be clear: with > VFIO PCI userspace mmaps the BARs and performs direct accesses to them > (load/store instructions). No VFIO API wrappers are necessary for MMIO > accesses, so the code you posted works fine with VFIO. > > > > > > > > > > > > > > > > > > > > > > > - small amount of physical memory for using as DMA descriptor can be > > > > > > pre-allocated from userspace, and ask kernel to pin pages, then still > > > > > > return physical address to userspace for programming DMA > > > > > > > > > > I think this is possible today. The ublk server owns the I/O buffers. It > > > > > can mlock them and DMA map them via VFIO. ublk doesn't need to know > > > > > anything about this. > > > > > > > > It depends on if such VFIO DMA mapping is required for each IO. If it > > > > is required, that won't help one high performance driver. > > > > > > It is not necessary to perform a DMA mapping for each IO. ublk's > > > existing model is sufficient: > > > 1. ublk server allocates I/O buffers and VFIO DMA maps them on startup. > > > 2. At runtime the ublk server provides these I/O buffers to the kernel, > > > no further DMA mapping is required. > > > > > > Unfortunately there's still the kernel<->userspace copy that existing > > > ublk applications have, but there's no new overhead related to VFIO. > > > > We are working on ublk zero copy for avoiding the copy. > > I'm curious if it's possible to come up with a solution that doesn't > break memory isolation. Userspace controls the IOMMU with Linux VFIO, so > if kernel pages are exposed to the device, then userspace will also be > able to access them (e.g. by submitting a request that gets the device > to DMA those pages). spdk nvme already exposes physical address of memory and uses the physical address to program hardware directly. And I think it can't be done by un-trusted user. But I agree with you that this way should be avoided as far as possible. > > > > > > > > > > > > > > > > > - this way is still zero copy > > > > > > > > > > True zero-copy would be when an application does O_DIRECT I/O and the > > > > > hardware device DMAs to/from the application's memory pages. ublk > > > > > doesn't do that today and when combined with VFIO it doesn't get any > > > > > easier. I don't think it's possible because you cannot allow userspace > > > > > to control a hardware device and grant DMA access to pages that > > > > > userspace isn't allowed to access. A malicious userspace will program > > > > > the device to access those pages :). > > > > > > > > But that should be what SPDK nvme/pci is doing per the above links, :-) > > > > > > Sure, it's possible to break memory isolation. Breaking memory isolation > > > isn't specific to ublk servers that access hardware. The same unsafe > > > zero-copy approach would probably also work for regular ublk servers. > > > This is basically bringing back /dev/kmem :). > > > > > > > > > > > > > > > > > > > > > > > > 5) notification from hardware: interrupt or polling > > > > > > - SPDK applies userspace polling, this way is doable, but > > > > > > eat CPU, so it is only one choice > > > > > > > > > > > > - io_uring command has been proved as very efficient, if io_uring > > > > > > command is applied(similar way with UBLK for forwarding blk io > > > > > > command from kernel to userspace) to uio/vfio for delivering interrupt, > > > > > > which should be efficient too, given batching processes are done after > > > > > > the io_uring command is completed > > > > > > > > > > I wonder how much difference there is between the new io_uring command > > > > > for receiving VFIO irqs that you are suggesting compared to the existing > > > > > io_uring approach IORING_OP_READ eventfd. > > > > > > > > eventfd needs extra read/write on the event fd, so more syscalls are > > > > required. > > > > > > No extra syscall is required because IORING_OP_READ is used to read the > > > eventfd, but maybe you were referring to bypassing the > > > file->f_op->read() code path? > > > > OK, missed that, it is usually done in the following way: > > > > io_uring_prep_poll_add(sqe, evfd, POLLIN) > > sqe->flags |= IOSQE_IO_LINK; > > ... > > sqe = io_uring_get_sqe(&ring); > > io_uring_prep_readv(sqe, evfd, &vec, 1, 0); > > sqe->flags |= IOSQE_IO_LINK; > > > > When I get time, will compare the two and see which one performs better. > > That would be really interesting. Anyway, interrupt notification looks not one big deal. Thanks, Ming
