Re: [RFC 0/3] Add zero copy feature for tcmu

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 18.03.22 10:55, Xiaoguang Wang wrote:
The core idea to implement tcmu zero copy feature is really straight,
which just maps block device io request's sgl pages to tcmu user space
backstore, then we can avoid extra copy overhead between sgl pages and
tcmu internal data area(which really impacts io throughput), please see
https://www.spinics.net/lists/target-devel/msg21121.html for detailed
info.


Can you please tell us, how big the performance improvement is and
which configuration you are using for measurenments?

Initially I use remap_pfn_range or vm_insert_pages to map sgl pages to
user space, but both of them have limits:
1)  Use vm_insert_pages
which is like tcp getsockopt(TCP_ZEROCOPY_RECEIVE), but there're two
restrictions:
   1. anonymous pages can not be mmaped to user spacea.
     ==> vm_insert_pages
     ====> insert_pages
     ======> insert_page_in_batch_locked
     ========> validate_page_before_insert
     In validate_page_before_insert(), it shows that anonymous page can not
     be mapped to use space, we know that if issuing direct io to block
     device, io request's sgl pages mostly comes from anonymous page.
         if (PageAnon(page) || PageSlab(page) || page_has_type(page))
             return -EINVAL;
     I'm not sure why there is such restriction? for safety reasons ?

   2. warn_on triggered in __folio_mark_dirty
     When calling zap_page_range in tcmu user space backstore when io
     completes, there is a warn_on triggered in __folio_mark_dirty:
        if (folio->mapping) {   /* Race with truncate? */
            WARN_ON_ONCE(warn && !folio_test_uptodate(folio));

     I'm not familiar with folio yet, but I think the reason is that when
     issuing a buffered read to tcmu block device, it's page cache mapped
     to user space, backstore write this page and pte will be dirtied. but
     initially it's newly allocated, hence page_update flag not set.
     In zap_pte_range(), there is such codes:
        if (!PageAnon(page)) {
            if (pte_dirty(ptent)) {
                force_flush = 1;
                set_page_dirty(page);
            }
    So this warn_on is reasonable.
    Indeed what I want is just to map io request sgl pages to tcmu user
    space backstore, then backstore can read or write data to mapped area,
    I don't want to care about page or its mapping status, so I choose to
    use remap_pfn_range.

2) Use remap_pfn_range()
   remap_pfn_range works well, but it has somewhat obvious overhead. For a
   512kb io request, it has 128 pages, and usually this 128 page's pfn are
   not consecutive, so in worst cases, for a 512kb io request, I'd need to
   issue 128 calls to remap_pfn_range, it's horrible. And in remap_pfn_range,
   if x86 page attribute table feature is enabled, lookup_memtype called by
   track_pfn_remap() also introduces obvious overhead.

Finally in order to solve these problems, Xu Yu helps to implment a new
helper, which accepts an array of pages as parameter, anonymous pages can
be mapped to user space, pages would be treated as special pte(pte_special
returns true), so vm_normal_page returns NULL, above folio warn_on won't
trigger.

Thanks.

Xiaoguang Wang (2):
   mm: export zap_page_range()
   scsi: target: tcmu: Support zero copy

Xu Yu (1):
   mm/memory.c: introduce vm_insert_page(s)_mkspecial

  drivers/target/target_core_user.c | 257 +++++++++++++++++++++++++++++++++-----
  include/linux/mm.h                |   2 +
  mm/memory.c                       | 183 +++++++++++++++++++++++++++
  3 files changed, 414 insertions(+), 28 deletions(-)





[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux