Ming Lei <ming.lei@xxxxxxxxxx> writes: > On Thu, Feb 16, 2023 at 10:44:02AM +0100, Andreas Hindborg wrote: >> >> Hi Ming, >> >> Ming Lei <ming.lei@xxxxxxxxxx> writes: >> >> > On Mon, Feb 13, 2023 at 02:13:59PM -0500, Stefan Hajnoczi wrote: >> >> On Mon, Feb 13, 2023 at 11:47:31AM +0800, Ming Lei wrote: >> >> > On Wed, Feb 08, 2023 at 07:17:10AM -0500, Stefan Hajnoczi wrote: >> >> > > On Wed, Feb 08, 2023 at 10:12:19AM +0800, Ming Lei wrote: >> >> > > > On Mon, Feb 06, 2023 at 03:27:09PM -0500, Stefan Hajnoczi wrote: >> >> > > > > On Mon, Feb 06, 2023 at 11:00:27PM +0800, Ming Lei wrote: >> >> > > > > > Hello, >> >> > > > > > >> >> > > > > > So far UBLK is only used for implementing virtual block device from >> >> > > > > > userspace, such as loop, nbd, qcow2, ...[1]. >> >> > > > > >> >> > > > > I won't be at LSF/MM so here are my thoughts: >> >> > > > >> >> > > > Thanks for the thoughts, :-) >> >> > > > >> >> > > > > >> >> > > > > > >> >> > > > > > It could be useful for UBLK to cover real storage hardware too: >> >> > > > > > >> >> > > > > > - for fast prototype or performance evaluation >> >> > > > > > >> >> > > > > > - some network storages are attached to host, such as iscsi and nvme-tcp, >> >> > > > > > the current UBLK interface doesn't support such devices, since it needs >> >> > > > > > all LUNs/Namespaces to share host resources(such as tag) >> >> > > > > >> >> > > > > Can you explain this in more detail? It seems like an iSCSI or >> >> > > > > NVMe-over-TCP initiator could be implemented as a ublk server today. >> >> > > > > What am I missing? >> >> > > > >> >> > > > The current ublk can't do that yet, because the interface doesn't >> >> > > > support multiple ublk disks sharing single host, which is exactly >> >> > > > the case of scsi and nvme. >> >> > > >> >> > > Can you give an example that shows exactly where a problem is hit? >> >> > > >> >> > > I took a quick look at the ublk source code and didn't spot a place >> >> > > where it prevents a single ublk server process from handling multiple >> >> > > devices. >> >> > > >> >> > > Regarding "host resources(such as tag)", can the ublk server deal with >> >> > > that in userspace? The Linux block layer doesn't have the concept of a >> >> > > "host", that would come in at the SCSI/NVMe level that's implemented in >> >> > > userspace. >> >> > > >> >> > > I don't understand yet... >> >> > >> >> > blk_mq_tag_set is embedded into driver host structure, and referred by queue >> >> > via q->tag_set, both scsi and nvme allocates tag in host/queue wide, >> >> > that said all LUNs/NSs share host/queue tags, current every ublk >> >> > device is independent, and can't shard tags. >> >> >> >> Does this actually prevent ublk servers with multiple ublk devices or is >> >> it just sub-optimal? >> > >> > It is former, ublk can't support multiple devices which share single host >> > because duplicated tag can be seen in host side, then io is failed. >> > >> >> I have trouble following this discussion. Why can we not handle multiple >> block devices in a single ublk user space process? >> >> From this conversation it seems that the limiting factor is allocation >> of the tag set of the virtual device in the kernel? But as far as I can >> tell, the tag sets are allocated per virtual block device in >> `ublk_ctrl_add_dev()`? >> >> It seems to me that a single ublk user space process shuld be able to >> connect to multiple storage devices (for instance nvme-of) and then >> create a ublk device for each namespace, all from a single ublk process. >> >> Could you elaborate on why this is not possible? > > If the multiple storages devices are independent, the current ublk can > handle them just fine. > > But if these storage devices(such as luns in iscsi, or NSs in nvme-tcp) > share single host, and use host-wide tagset, the current interface can't > work as expected, because tags is shared among all these devices. The > current ublk interface needs to be extended for covering this case. Thanks for clarifying, that is very helpful. Follow up question: What would the implications be if one tried to expose (through ublk) each nvme namespace of an nvme-of controller with an independent tag set? What are the benefits of sharing a tagset across all namespaces of a controller? Best regards, Andreas
