User space memory is mapped in kernel in form of pages array. These pages
are iterated and added to BIO. In process, pages are also checked for
contiguity and merged.
When mTHP is enabled the pages generally belong to larger order folio. The
pages from larger order folio need not be checked for contiguity and can be
added directly as a big chunk.
This patch series enables adding large folio to bio. It fetches folio for
page in the page array. The page might start from an offset in the folio
which could be multiples of PAGE_SIZE. Subsequent pages in page array
might belong to same folio. Using the length of folio, folio_offset and
remaining size, determine length in folio which shall be added to the
bio. Further processing is skipped for these pages. This reduces the
overhead of iterating through pages.
perf diff before and after this change:
Perf diff for write I/O with 128K block size:
1.26% -1.04% [kernel.kallsyms] [k] bio_iov_iter_get_pages
1.74% [kernel.kallsyms] [k] bvec_try_merge_page
Perf diff for read I/O with 128K block size:
4.40% -3.63% [kernel.kallsyms] [k] bio_iov_iter_get_pages
5.60% [kernel.kallsyms] [k] bvec_try_merge_page
Patch 1: Ignores bigger offset and helps driver to keep using PRP for I/Os
if length is one or two PRPs
Patch 2: Adds changes to add larger order folio to BIO.
Patch 3: If a large folio gets added, the subsequent pages of the folio are
released. This helps to avoid calculations at I/O completion.
Previous Postings
-----------------
v1: https://lore.kernel.org/all/20240419091721.1790-1-kundan.kumar@xxxxxxxxxxx/
v2: https://lore.kernel.org/linux-block/33717b97-8986-4d6e-aa10-47393b810ea2@xxxxxxx/T/#m31564a9c6ae30e32e560dd8c76d26a3290a875aa
Changes since v2:
- Made separate patches
- Corrected code as per kernel coding style
- Removed size_folio variable
Changes since v1:
- Changed functions bio_iov_add_page() and bio_iov_add_zone_append_page() to
accept a folio
- Removed branch and calculate folio_offset and len in same fashion for both
0 order and larger folios
- Added change in NVMe driver to use nvme_setup_prp_simple() by
ignoring multiples of PAGE_SIZE in offset
- Added a change to unpin_user_pages which were added as folios. Also stopped
the unpin of pages one by one from __bio_release_pages()(Suggested by
Keith)
Kundan Kumar (3):
nvme: adjust multiples of NVME_CTRL_PAGE_SIZE in offset
block: add folio awareness instead of looping through pages
block: unpin user pages belonging to a folio
block/bio.c | 50 +++++++++++++++++++++++------------------
drivers/nvme/host/pci.c | 3 ++-
2 files changed, 30 insertions(+), 23 deletions(-)
--
2.25.1
