Registered buffers are stored and processed in the form of bvec array,
each bvec element typically points to a PAGE_SIZE page but can also work
with hugepages. Specifically, a buffer consisting of a hugepage is
coalesced to use only one hugepage bvec entry during registration.
This coalescing feature helps to save both the space and DMA-mapping time.
However, currently the coalescing feature doesn't work for multi-hugepage
buffers. For a buffer with several 2M hugepages, we still split it into
thousands of 4K page bvec entries while in fact, we can just use a
handful of hugepage bvecs.
This patch series enables coalescing registered buffers with more than
one hugepages. It optimizes the DMA-mapping time and saves memory for
these kind of buffers.
Testing:
The hugepage fixed buffer I/O can be tested using fio without
modification. The fio command used in the following test is given
in [1]. There's also a liburing testcase in [2]. Also, the system
should have enough hugepages available before testing.
Perf diff of 8M(4 * 2M hugepages) fio randread test:
Before After Symbol
.....................................................
5.88% [k] __blk_rq_map_sg
3.98% -3.95% [k] dma_direct_map_sg
2.47% [k] dma_pool_alloc
1.37% -1.36% [k] sg_next
+0.28% [k] dma_map_page_attrs
Perf diff of 8M fio randwrite test:
Before After Symbol
......................................................
2.80% [k] __blk_rq_map_sg
1.74% [k] dma_direct_map_sg
1.61% [k] dma_pool_alloc
0.67% [k] sg_next
+0.04% [k] dma_map_page_attrs
The first patch prepares for adding the multi-hugepage coalescing
by storing folio_shift and folio_mask into imu, the 2nd patch
enables the feature.
---
Changes since v5:
- Reshuffle the patchset to avoid unused funtion warnings.
- Store head page of the folio and use folio offset to get rid of
branching on bvec setups.
- Add restrictions for non border-aligned folios.
- Remove unnecessary folio_size field in io_imu_folio_data struct.
v5 : https://lore.kernel.org/io-uring/20240708021426.2217-1-cliang01.li@xxxxxxxxxxx/T/#t
[1]
fio -iodepth=64 -rw=randread -direct=1 -ioengine=io_uring \
-bs=8M -numjobs=1 -group_reporting -mem=shmhuge -fixedbufs -hugepage-size=2M \
-filename=/dev/nvme0n1 -runtime=10s -name=test1
[2]
https://lore.kernel.org/io-uring/20240514051343.582556-1-cliang01.li@xxxxxxxxxxx/T/#u
Chenliang Li (2):
io_uring/rsrc: store folio shift and mask into imu
io_uring/rsrc: enable multi-hugepage buffer coalescing
io_uring/rsrc.c | 154 +++++++++++++++++++++++++++++++++++-------------
io_uring/rsrc.h | 10 ++++
2 files changed, 123 insertions(+), 41 deletions(-)
base-commit: ad00e629145b2b9f0d78aa46e204a9df7d628978
--
2.34.1
