On 08.12.21 14:32, Xiubo Li wrote:
On 12/8/21 8:43 PM, Xiaoguang Wang wrote:
hi，
I'm a newcomer to tcmu or iscsi subsystem, and have spent several days
to learn
iSCSI and tcmu, so if my question looks fool, forgive me :)
One of our customers is using tcmu to visit remote distributed
filesystem and finds
that there's obvious copy overhead in tcmu while doing read
operations, so I spent
time to find the reason and see whether can optimize a bit. Now
according to my
understanding about tcmu kernel codes, tcmu allocates internal data
pages for data
area, use these data pages as temporary storage between user-space
backstore and
tcmu. For iSCSI initiator's write request, tcmu first copy sg page's
content to internal
data pages, then user-space backstore moves mmaped data area for these
data pages
to backstore; for iSCSI initiator's read request, tcmu also allocates
internal data pages,
backstore copies distributed filesystem's data to these data pages,
later tcmu copy
data pages' content to sg's pages. That means for both read and write
requests, it
exists one extra data copy.
So my question is that whether we don't allocate internal data pages
in tcmu, just make
sg's pages to be mmaped in data area, so we can reduce one extra copy,
which I think
it can improve throughput. Or is there any special security issues
that we can not do
this way? Thanks.
Yeah, this is a good start to improve this. Currently the tcmu-runner
could benifit from the mapped temporary data pages and won't do the
extra copy in userspace.
I think the idea is to avoid data copy in tcmu, not userspace.
What extra copy in userspace are you talking about?
I think you can do the dynamic map/unmap for the data pages for each
SCSI cmd in the LIO's ringbuffer, but this should be opaque to the user
space daemons. And at the same time the LIO needs to tell tcmu-runner
the offset from/to where should it read/write the data contents in the
mapped ringbuffer via each tcmu cmd entry. Currently the tcmu cmd entry
will be followed by its data contents.
Unfortunately that's not true. tcmu does not store data in the cmd ring.
Data area is behind the cmd ring in tcmu's uio dev mapping.
So cmd entry in cmd ring already contains an array of struct iovec to
specify the pieces of the data area used for the cmd's in or out data.
But I didn't investigate whether could we make the sg's pages, not sure
whether will they have any limitation to do this.
BRs
Xiubo
Regards,
Xiaoguang Wang
