hi,
Sagi Grimberg <sagi@xxxxxxxxxxx> writes:
Actually, I'd rather have something like an 'inverse io_uring', where
an application creates a memory region separated into several 'ring'
for submission and completion.
Then the kernel could write/map the incoming data onto the rings, and
application can read from there.
Maybe it'll be worthwhile to look at virtio here.
There is lio loopback backed by tcmu... I'm assuming that nvmet can
hook into the same/similar interface. nvmet is pretty lean, and we
can probably help tcmu/equivalent scale better if that is a concern...
Sagi,
I looked at tcmu prior to starting this work. Other than the tcmu
overhead, one concern was the complexity of a scsi device interface
versus sending block requests to userspace.
Yeah, Some of our costumers have tried to use tcmu and found obvious
overhead, which
impact io throughput tremendously, especially it lacks zero-copy and
multi-queue support.
Previously I have sent a report to tcmu community:
https://www.spinics.net/lists/target-devel/msg21121.html
And currently I have implemented a zero-copy prototype for tcmu(not sent
out yet), which
increases user's io throughput from 3.6GB to 11.5GB/s, fio 4 jobs, 8
iodepth, io size 256kb.
This prototype uses remap_pfn_range() to map io requests' sg pages to
user space, but
remap_pfn_range() have obvious overhead while intel pat is enabled. I
also sent a mail to
mm community:
https://lore.kernel.org/linux-mm/c5526629-5ce4-1f99-e9af-36da2876b258@xxxxxxxxxxxxxxxxx/T/#u
About how to map sg pages to use space correctly, but there's no
response yet.
If anybody is familiar with my question, may kindly give help, thanks.
Regards,
Xiaoguang Wang
What would be the advantage of doing it as a nvme target over delivering
directly to userspace as a block driver?
Also, when considering the case where userspace wants to just look at the IO
descriptor, without actually sending data to userspace, I'm not sure
that would be doable with tcmu?
Another attempt to do the same thing here, now with device-mapper:
https://patchwork.kernel.org/project/dm-devel/patch/20201203215859.2719888-4-palmer@xxxxxxxxxxx/
