On 12/8/21 8:43 PM, Xiaoguang Wang wrote:
hi，
I'm a newcomer to tcmu or iscsi subsystem, and have spent several days
to learn
iSCSI and tcmu, so if my question looks fool, forgive me :)
One of our customers is using tcmu to visit remote distributed
filesystem and finds
that there's obvious copy overhead in tcmu while doing read
operations, so I spent
time to find the reason and see whether can optimize a bit. Now
according to my
understanding about tcmu kernel codes, tcmu allocates internal data
pages for data
area, use these data pages as temporary storage between user-space
backstore and
tcmu. For iSCSI initiator's write request, tcmu first copy sg page's
content to internal
data pages, then user-space backstore moves mmaped data area for these
data pages
to backstore; for iSCSI initiator's read request, tcmu also allocates
internal data pages,
backstore copies distributed filesystem's data to these data pages,
later tcmu copy
data pages' content to sg's pages. That means for both read and write
requests, it
exists one extra data copy.
So my question is that whether we don't allocate internal data pages
in tcmu, just make
sg's pages to be mmaped in data area, so we can reduce one extra copy,
which I think
it can improve throughput. Or is there any special security issues
that we can not do
this way? Thanks.
Yeah, this is a good start to improve this. Currently the tcmu-runner
could benifit from the mapped temporary data pages and won't do the
extra copy in userspace.
I think you can do the dynamic map/unmap for the data pages for each
SCSI cmd in the LIO's ringbuffer, but this should be opaque to the user
space daemons. And at the same time the LIO needs to tell tcmu-runner
the offset from/to where should it read/write the data contents in the
mapped ringbuffer via each tcmu cmd entry. Currently the tcmu cmd entry
will be followed by its data contents.
But I didn't investigate whether could we make the sg's pages, not sure
whether will they have any limitation to do this.
BRs
Xiubo
Regards,
Xiaoguang Wang
