IAA Decompression Batching: =========================== This patch-series applies over [1], the IAA compress batching patch-series. [1] https://patchwork.kernel.org/project/linux-mm/list/?series=900537 This RFC patch-series introduces the use of the Intel Analytics Accelerator (IAA) for parallel decompression of 4K folios prefetched by swapin_readahead(). We have developed zswap batched loading of these prefetched folios, that deploys the use of parallel decompressions by IAA. swapin_readahead() provides a natural batching interface because it adapts to the usefulness of prior prefetches, to adjust the readahead window. Hence, it allows the page-cluster to be set based on workload characteristics. For workloads that are prefetching friendly, this can form the basis for reading ahead up to 32 folios with zswap load batching to significantly reduce swap in latency, major page-faults and systime; thereby improving workload performance. The patch-series builds upon the IAA compress batching patch-series [1], and is organized as follows: 1) A centralized batch decompression API that can be used by swap modules. 2) "struct folio_batch" modifications, e.g., PAGEVEC_SIZE is increased to 2^5. 3) Addition of "zswap_batch" and "non_zswap_batch" folio_batches in swap_read_folio() to serve the purposes of a plug. 4) swap_read_zswap_batch_unplug() API in page_io.c to process a read batch of entries found in zswap. 5) zswap API to add a swap entry to a load batch, init/reinit the batch, process the batch using the batch decompression API. 6) Modifications to the swapin_readahead() functions, swap_vma_readahead() and swap_cluster_readahead() to: a) Call swap_read_folio() to add prefetch swap entries to "zswap_batch" and "non_zswap_batch" folio_batches. b) Process the two readahead folio batches: "non_zswap_batch" folios will be read sequentially; "zswap_batch" folios will be batch decompressed with IAA. 7) Modifications to do_swap_page() to invoke swapin_readahead() from both, the single-mapped SWP_SYNCHRONOUS_IO and shared/non-SWP_SYNCHRONOUS_IO branches. In the former path, we call swapin_readahead() only in the !zswap_never_enabled() case. a) This causes folios to be read into the swapcache in both paths. This design choice was motivated by stability: to handle race conditions with say, process 1 faulting in a single-mapped folio; however, process 2 could be simultaneously prefetching it as a "readahead" folio. b) If the single-mapped folio was successfully read and the race did not occur, there are checks added to free the swapcache entry for the folio, before do_swap_page() returns. 8) Finally, for IAA batching, we reduce SWAP_BATCH to 16 and modify the swap slots cache thresholds to alleviate lock contention on the swap_info_struct lock due to reduced swap page-fault latencies. IAA decompress batching can be enabled only on platforms that have IAA, by setting this config variable: CONFIG_ZSWAP_LOAD_BATCHING_ENABLED="y" A new swap parameter "singlemapped_ra_enabled" (false by default) is added for use on platforms that have IAA. If zswap_load_batching_enabled() is true, this is intended to give the user the option to run experiments with IAA and with software compressors for zswap. These are the recommended settings for "singlemapped_ra_enabled", which takes effect only in the do_swap_page() single-mapped SWP_SYNCHRONOUS_IO path: For IAA: echo true > /sys/kernel/mm/swap/singlemapped_ra_enabled For software compressors: echo false > /sys/kernel/mm/swap/singlemapped_ra_enabled If "singlemapped_ra_enabled" is set to false, swapin_readahead() will skip prefetching folios in the "single-mapped SWP_SYNCHRONOUS_IO" do_swap_page() path. IAA decompress batching performance testing was done using the kernel compilation test "allmodconfig" run in tmpfs, which demonstrates a significant amount of readahead activity. vm-scalability usemem is not ideal for decompress batching because there is very little readahead activity even with page-cluster of 5 (swap_ra is < 150 with 4k/16k/32k/64k folios). The kernel compilation experiments with decompress batching demonstrate significant latency reductions with kernel compilation: up to 4% lower elapsed time, 14% lower sys time than mm-unstable/zstd. When combined with compress batching, we see a reduction of 5% in elapsed time and 20% in sys time as compared to mm-unstable commit 817952b8be34 with zstd. Our internal validation of IAA compress/decompress batching in highly contended Sapphire Rapids server setups with workloads running on 72 cores for ~25 minutes under stringent memory limit constraints have shown up to 50% reduction in sys time and 3.5% reduction in workload run time as compared to software compressors. System setup for testing: ========================= Testing of this patch-series was done with mm-unstable as of 10-16-2024, commit 817952b8be34, without and with this patch-series ("this patch-series" includes [1]). Data was gathered on an Intel Sapphire Rapids server, dual-socket 56 cores per socket, 4 IAA devices per socket, 503 GiB RAM and 525G SSD disk partition swap. Core frequency was fixed at 2500MHz. The kernel compilation test with run in tmpfs, using the "allmodconfig", so that significant swapout and readahead activity can be observed to quantify decompress batching. Other kernel configuration parameters: zswap compressor : deflate-iaa zswap allocator : zsmalloc vm.page-cluster : 3,4 IAA "compression verification" is disabled and the async poll acomp interface is used in the iaa_crypto driver (the defaults with this series). Performance testing (Kernel compilation): ========================================= As mentioned earlier, for workloads that see a lot of swapout activity, we can benefit from configuring 2 WQs per IAA device, with compress jobs from all same-socket cores being distributed toothe wq.1 of all IAAs on the socket, with the "global_wq" developed in this patch-series. Although this data includes IAA decompress batching, which will be submitted as a separate RFC patch-series, I am listing it here to quantify the benefit of distributing compress jobs among all IAAs. The kernel compilation test with "allmodconfig" is able to quantify this well: 4K folios: deflate-iaa: kernel compilation ========================================== ------------------------------------------------------------------------------ mm-unstable-10-16-2024 zswap_load_batch with IAA decompress batching ------------------------------------------------------------------------------ zswap compressor zstd deflate-iaa vm.compress-batchsize n/a 1 vm.page-cluster 3 3 ------------------------------------------------------------------------------ real_sec 783.87 752.99 user_sec 15,750.07 15,746.37 sys_sec 6,522.32 5,638.16 Max_Res_Set_Size_KB 1,872,640 1,872,640 ------------------------------------------------------------------------------ zswpout 82,364,991 105,190,461 zswpin 21,303,393 29,684,653 pswpout 13 1 pswpin 12 1 pgmajfault 17,114,339 24,034,146 swap_ra 4,596,035 6,219,484 swap_ra_hit 2,903,249 3,876,195 ------------------------------------------------------------------------------ Progression of kernel compilation latency improvements with compress/decompress batching: ============================================================ ------------------------------------------------------------------------------- mm-unstable-10-16-2024 shrink_folio_ zswap_load_batch list() w/ IAA decompress batching batching of folios ------------------------------------------------------------------------------- zswap compr zstd deflate-iaa deflate-iaa deflate-iaa deflate-iaa vm.compress- n/a n/a 32 1 32 batchsize vm.page- 3 3 3 3 3 cluster ------------------------------------------------------------------------------- real_sec 783.87 761.69 747.32 752.99 749.25 user_sec 15,750.07 15,716.69 15,728.39 15,746.37 15,741.71 sys_sec 6,522.32 5,725.28 5,399.44 5,638.16 5,482.12 Max_RSS_KB 1,872,640 1,870,848 1,874,432 1,872,640 1,872,640 zswpout 82,364,991 97,739,600 102,780,612 105,190,461 106,729,372 zswpin 21,303,393 27,684,166 29,016,252 29,684,653 30,717,819 pswpout 13 222 213 1 12 pswpin 12 209 202 1 8 pgmajfault 17,114,339 22,421,211 23,378,161 24,034,146 24,852,985 swap_ra 4,596,035 5,840,082 6,231,646 6,219,484 6,504,878 swap_ra_hit 2,903,249 3,682,444 3,940,420 3,876,195 4,092,852 ------------------------------------------------------------------------------- The last 2 columns of the latency reduction progression are as follows: IAA decompress batching combined with distributing compress jobs to all same-socket IAA devices: ======================================================================= ------------------------------------------------------------------------------ IAA shrink_folio_list() compress batching and swapin_readahead() decompress batching 1WQ 2WQ (distribute compress jobs) 1 local WQ (wq.0) 1 local WQ (wq.0) + per IAA 1 global WQ (wq.1) per IAA ------------------------------------------------------------------------------ zswap compressor deflate-iaa deflate-iaa vm.compress-batchsize 32 32 vm.page-cluster 4 4 ------------------------------------------------------------------------------ real_sec 746.77 745.42 user_sec 15,732.66 15,738.85 sys_sec 5,384.14 5,247.86 Max_Res_Set_Size_KB 1,874,432 1,872,640 ------------------------------------------------------------------------------ zswpout 101,648,460 104,882,982 zswpin 27,418,319 29,428,515 pswpout 213 22 pswpin 207 6 pgmajfault 21,896,616 23,629,768 swap_ra 6,054,409 6,385,080 swap_ra_hit 3,791,628 3,985,141 ------------------------------------------------------------------------------ I would greatly appreciate code review comments for this RFC series! [1] https://patchwork.kernel.org/project/linux-mm/list/?series=900537 Thanks, Kanchana Kanchana P Sridhar (7): mm: zswap: Config variable to enable zswap loads with decompress batching. mm: swap: Add IAA batch decompression API swap_crypto_acomp_decompress_batch(). pagevec: struct folio_batch changes for decompress batching interface. mm: swap: swap_read_folio() can add a folio to a folio_batch if it is in zswap. mm: swap, zswap: zswap folio_batch processing with IAA decompression batching. mm: do_swap_page() calls swapin_readahead() zswap load batching interface. mm: For IAA batching, reduce SWAP_BATCH and modify swap slot cache thresholds. include/linux/pagevec.h | 13 +- include/linux/swap.h | 7 + include/linux/swap_slots.h | 7 + include/linux/zswap.h | 65 +++++++++ mm/Kconfig | 13 ++ mm/memory.c | 187 +++++++++++++++++++------ mm/page_io.c | 61 ++++++++- mm/shmem.c | 2 +- mm/swap.h | 102 ++++++++++++-- mm/swap_state.c | 272 ++++++++++++++++++++++++++++++++++--- mm/swapfile.c | 2 +- mm/zswap.c | 272 +++++++++++++++++++++++++++++++++++++ 12 files changed, 927 insertions(+), 76 deletions(-) -- 2.27.0
