On 2/6/23 16:00, Ming Lei wrote:
Hello, So far UBLK is only used for implementing virtual block device from userspace, such as loop, nbd, qcow2, ...[1]. It could be useful for UBLK to cover real storage hardware too: - for fast prototype or performance evaluation - some network storages are attached to host, such as iscsi and nvme-tcp, the current UBLK interface doesn't support such devices, since it needs all LUNs/Namespaces to share host resources(such as tag) - SPDK has supported user space driver for real hardware So propose to extend UBLK for supporting real hardware device: 1) extend UBLK ABI interface to support disks attached to host, such as SCSI Luns/NVME Namespaces 2) the followings are related with operating hardware from userspace, so userspace driver has to be trusted, and root is required, and can't support unprivileged UBLK device 3) how to operating hardware memory space - unbind kernel driver and rebind with uio/vfio - map PCI BAR into userspace[2], then userspace can operate hardware with mapped user address via MMIO 4) DMA - DMA requires physical memory address, UBLK driver actually has block request pages, so can we export request SG list(each segment physical address, offset, len) into userspace? If the max_segments limit is not too big(<=64), the needed buffer for holding SG list can be small enough. - small amount of physical memory for using as DMA descriptor can be pre-allocated from userspace, and ask kernel to pin pages, then still return physical address to userspace for programming DMA - this way is still zero copy 5) notification from hardware: interrupt or polling - SPDK applies userspace polling, this way is doable, but eat CPU, so it is only one choice - io_uring command has been proved as very efficient, if io_uring command is applied(similar way with UBLK for forwarding blk io command from kernel to userspace) to uio/vfio for delivering interrupt, which should be efficient too, given batching processes are done after the io_uring command is completed - or it could be flexible by hybrid interrupt & polling, given userspace single pthread/queue implementation can retrieve all kinds of inflight IO info in very cheap way, and maybe it is likely to apply some ML model to learn & predict when IO will be completed 6) others?
Good idea. I'd love to have this discussion. Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@xxxxxxx +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
