User space memory is mapped in kernel in form of pages array. These pages
are iterated and added to BIO. In process, pages are also checked for
contiguity and merged.
When mTHP is enabled the pages generally belong to larger order folio. This
patch series enables adding large folio to bio. It fetches folio for
page in the page array. The page might start from an offset in the folio
which could be multiples of PAGE_SIZE. Subsequent pages in page array
might belong to same folio. Using the length of folio, folio_offset and
remaining size, determine length in folio which can be added to the bio.
Check if pages are contiguous and belong to same folio. If yes then skip
further processing for the contiguous pages.
This complete scheme reduces the overhead of iterating through pages.
perf diff before and after this change(with mTHP enabled):
Perf diff for write I/O with 128K block size:
1.24% -0.20% [kernel.kallsyms] [k] bio_iov_iter_get_pages
1.71% [kernel.kallsyms] [k] bvec_try_merge_page
Perf diff for read I/O with 128K block size:
4.03% -1.59% [kernel.kallsyms] [k] bio_iov_iter_get_pages
5.14% [kernel.kallsyms] [k] bvec_try_merge_page
Patch 1: Adds folio-ized version of bvec_try_merge_hw_page()
Patch 2: Adds folio-ized version of bio_add_hw_page()
Patch 3: Adds bigger size from folio to BIO
Patch 4: Adds mm function to release n pages of folio
Patch 5: Unpin user pages belonging to folio at once using the mm
function
Changes since v7:
- corrected issue found by kernel robot, fixed by avoiding updating
same_page for different pages of folio
- changed folio-lise to folio-ize
- corrected order with wrapper definition after the guts
- removed check for BIO_PAGE_PINNED
- separated mm related implementation in different patch
Changes since v6:
- folio-lize bvec_try_merge_hw_page() to bvec_try_merge_hw_folio()
- restructured the code as per comments
- typecast with size_t while calculating the offset in folio.
Changes since v5:
- Made offset and len as size_t in function bio_add_hw_folio()
- Avoid unpinning skipped pages at submission, rather unpin all pages at
once on IO completion
Changes since v4:
- folio-lize bio_add_hw_page() to bio_add_hw_folio()
- make bio_add_hw_page() as a wrapper around bio_add_hw_folio()
- made a helper function to check for contiguous pages of folio
- changed &folio->page to folio_page(folio, 0)
- reworded comments
Changes since v3:
- Added change to see if pages are contiguous and belong to same folio.
If not then avoid skipping of pages.(Suggested by Matthew Wilcox)
Changes since v2:
- Made separate patches
- Corrected code as per kernel coding style
- Removed size_folio variable
Changes since v1:
- Changed functions bio_iov_add_page() and bio_iov_add_zone_append_page()
to accept a folio
- Removed branch and calculate folio_offset and len in same fashion for
both 0 order and larger folios
- Added change in NVMe driver to use nvme_setup_prp_simple() by
ignoring multiples of PAGE_SIZE in offset
- Added a change to unpin_user_pages which were added as folios. Also
stopped the unpin of pages one by one from __bio_release_pages()
(Suggested by Keith)
Kundan Kumar (5):
block: Added folio-ized version of bvec_try_merge_hw_page()
block: Added folio-ized version of bio_add_hw_page()
block: introduce folio awareness and add a bigger size from folio
mm: release number of pages of a folio
block: unpin user pages belonging to a folio at once
block/bio.c | 156 ++++++++++++++++++++++++++++++++++-----------
block/blk.h | 14 ++++
include/linux/mm.h | 1 +
mm/gup.c | 13 ++++
4 files changed, 147 insertions(+), 37 deletions(-)
--
2.25.1
