On Wed, Mar 08, 2023 at 09:50:53AM +0100, Hans Holmberg wrote: > This is a great topic, so I'd like to be part of it as well. > > It would be great to figure out what latency overhead we could expect > of ublk in the future, clarifying what use cases ublk could cater for. > This will help a lot in making decisions on what to implement > in-kernel vs user space. If the zero copy patchset[1] can be accepted, the main overhead should be in io_uring command communication. I just run one quick test on my laptop between ublk/null(2 queues, depth 64, with zero copy) and null_blk(2 queues, depth 64), and run single job fio (128 qd, batch 16, libaio, 4k randread). IOPS on ublk is less than ~13% than null_blk. So looks the difference isn't bad, cause IOPS level has been reached Million level(1.29M vs. 1.46M). This basically shows the communication overhead. However, ublk userspace can handle io in lockless way, and minimize context switch & maximize io parallel by coroutine, that is ublk's advantage, and hard or impossible to do in kernel. In the ublksrv[2] project, we implemented loop, nbd & qcow2, so far, in my previous IOPS test result: 1) kernel loop(dio) vs. ublk/loop: the two are close 2) kernel nbd vs. ublk/nbd: ublk/nbd is a bit better than kernel nbd 3) qmeu-nbd based qcow2 vs. ublk/qcow2: ublk/qcow2 is much better All the three just works, not run further optimization yet. Also ublk may perform bad if io isn't handled in batch, such as, single queue depth io submission. But ublk is still very young, and there can be lots of optimization in future, such as: 1) applying polling for reducing communication overhead for both io command and io handle, and this way should improve latency for low QD workload 2) apply kind of ML model for predicating IO completion, and improve io polling, meantime reducing cpu utilization. 3) improve io_uring command for reducing communication overhead IMO, ublk is one generic userspace block device approach, especially good at: 1) handle complicated io logic, such as, btree is applied in io mapping, cause userspace has more weapons for this stuff 2) virtual device, such as all network based storage, or logical volume management 3) quick prototype development 4) flexible storage simulation for test purpose [1] https://lore.kernel.org/linux-block/ZAff9usDuyXxIPt9@xxxxxxxxxxxxxxxxxxxxxxxxx/T/#t [2] https://github.com/ming1/ubdsrv Thanks, Ming
