User space memory is mapped in kernel in form of pages array. These pages
are iterated and added to BIO. In process, pages are also checked for
contiguity and merged.
When mTHP is enabled the pages generally belong to larger order folio. This
patch series enables adding large folio to bio. It fetches folio for
page in the page array. The page might start from an offset in the folio
which could be multiples of PAGE_SIZE. Subsequent pages in page array
might belong to same folio. Using the length of folio, folio_offset and
remaining size, determine length in folio which can be added to the bio.
Check if pages are contiguous and belong to same folio. If yes then skip
further processing for the contiguous pages.
This complete scheme reduces the overhead of iterating through pages.
perf diff before and after this change:
Perf diff for write I/O with 128K block size:
1.16% -0.18% [kernel.kallsyms] [k] bio_iov_iter_get_pages
1.72% [kernel.kallsyms] [k] bvec_try_merge_page
Perf diff for read I/O with 128K block size:
3.76% -1.25% [kernel.kallsyms] [k] bio_iov_iter_get_pages
4.96% [kernel.kallsyms] [k] bvec_try_merge_page
Patch 1: Adds folio-lized version of bio_add_hw_page()
Patch 2: Adds changes to add larger order folio to BIO.
Patch 3: If a large folio gets added, the subsequent pages of the folio are
released. This helps to avoid calculations at I/O completion.
Changes since v4:
- folio-lize bio_add_hw_page() to bio_add_hw_folio()
- make bio_add_hw_page() as a wrapper around bio_add_hw_folio()
- make new functions bio_release_folio() and unpin_user_folio()
- made a helper function to check for contiguous pages of folio
- changed &folio->page to folio_page(folio, 0)
- reworded comments
Changes since v3:
- Added change to see if pages are contiguous and belong to same folio.
If not then avoid skipping of pages.(Suggested by Matthew Wilcox)
Changes since v2:
- Made separate patches
- Corrected code as per kernel coding style
- Removed size_folio variable
Changes since v1:
- Changed functions bio_iov_add_page() and bio_iov_add_zone_append_page()
to accept a folio
- Removed branch and calculate folio_offset and len in same fashion for
both 0 order and larger folios
- Added change in NVMe driver to use nvme_setup_prp_simple() by
ignoring multiples of PAGE_SIZE in offset
- Added a change to unpin_user_pages which were added as folios. Also
stopped the unpin of pages one by one from __bio_release_pages()
(Suggested by Keith)
Kundan Kumar (3):
block: Added folio-lized version of bio_add_hw_page()
block: add folio awareness instead of looping through pages
block: unpin user pages belonging to a folio
block/bio.c | 123 +++++++++++++++++++++++++++++++++------------
block/blk.h | 11 ++++
include/linux/mm.h | 1 +
mm/gup.c | 13 +++++
4 files changed, 116 insertions(+), 32 deletions(-)
--
2.25.1
