Registered buffers are stored and processed in the form of bvec array,
each bvec element typically points to a PAGE_SIZE page but can also work
with hugepages. Specifically, a buffer consisting of a hugepage is
coalesced to use only one hugepage bvec entry during registration.
This coalescing feature helps to save both the space and DMA-mapping time.
However, currently the coalescing feature doesn't work for multi-hugepage
buffers. For a buffer with several 2M hugepages, we still split it into
thousands of 4K page bvec entries while in fact, we can just use a
handful of hugepage bvecs.
This patch series enables coalescing registered buffers with more than
one hugepages. It optimizes the DMA-mapping time and saves memory for
these kind of buffers.
Testing:
The hugepage fixed buffer I/O can be tested using fio without
modification. The fio command used in the following test is given
in [1]. There's also a liburing testcase in [2]. Also, the system
should have enough hugepages available before testing.
Perf diff of 8M(4 * 2M hugepages) fio randread test:
Before After Symbol
.....................................................
5.88% [k] __blk_rq_map_sg
3.98% -3.95% [k] dma_direct_map_sg
2.47% [k] dma_pool_alloc
1.37% -1.36% [k] sg_next
+0.28% [k] dma_map_page_attrs
Perf diff of 8M fio randwrite test:
Before After Symbol
......................................................
2.80% [k] __blk_rq_map_sg
1.74% [k] dma_direct_map_sg
1.61% [k] dma_pool_alloc
0.67% [k] sg_next
+0.04% [k] dma_map_page_attrs
The first patch prepares for adding the multi-hugepage coalescing
by storing folio_shift and folio_mask into imu, the 2nd patch
enables the feature.
---
Changes since v7:
- Rebase to io_uring-6.11
v7 : https://lore.kernel.org/io-uring/54ef8248-29ac-4c0e-b224-8515b4c604cc@xxxxxxxxx/T/#t
Changes since v6:
- Remove the restriction on non-border-aligned single hugepage.
- Code style issue.
v6 : https://lore.kernel.org/io-uring/20240716060807.2707-1-cliang01.li@xxxxxxxxxxx/T/#t
[1]
fio -iodepth=64 -rw=randread -direct=1 -ioengine=io_uring \
-bs=8M -numjobs=1 -group_reporting -mem=shmhuge -fixedbufs -hugepage-size=2M \
-filename=/dev/nvme0n1 -runtime=10s -name=test1
[2]
https://lore.kernel.org/io-uring/20240514051343.582556-1-cliang01.li@xxxxxxxxxxx/T/#u
Chenliang Li (2):
io_uring/rsrc: store folio shift and mask into imu
io_uring/rsrc: enable multi-hugepage buffer coalescing
io_uring/rsrc.c | 149 +++++++++++++++++++++++++++++++++++-------------
io_uring/rsrc.h | 10 ++++
2 files changed, 118 insertions(+), 41 deletions(-)
base-commit: c3fca4fb83f7c84cd1e1aa9fe3a0e220ce8f30fb
--
2.34.1
