Re: [PATCH v8 0/8] mm: zswap swap-out of large folios

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Sep 27, 2024 at 7:16 PM Kanchana P Sridhar
<kanchana.p.sridhar@xxxxxxxxx> wrote:
>
> Hi All,
>
> This patch-series enables zswap_store() to accept and store large
> folios. The most significant contribution in this series is from the
> earlier RFC submitted by Ryan Roberts [1]. Ryan's original RFC has been
> migrated to mm-unstable as of 9-27-2024 in patch 6 of this series, and
> adapted based on code review comments received for v7 of the current
> patch-series.
>
> [1]: [RFC PATCH v1] mm: zswap: Store large folios without splitting
>      https://lore.kernel.org/linux-mm/20231019110543.3284654-1-ryan.roberts@xxxxxxx/T/#u
>
> The first few patches do the prep work for supporting large folios in
> zswap_store. Patch 6 provides the main functionality to swap-out large
> folios in zswap. Patch 7 adds sysfs per-order hugepages "zswpout" counters
> that get incremented upon successful zswap_store of large folios:
>
> /sys/kernel/mm/transparent_hugepage/hugepages-*kB/stats/zswpout
>
> Patch 8 updates the documentation for the new sysfs "zswpout" counters.
>
> This patch-series is a pre-requisite for zswap compress batching of large
> folio swap-out and decompress batching of swap-ins based on
> swapin_readahead(), using Intel IAA hardware acceleration, which we would
> like to submit in subsequent patch-series, with performance improvement
> data.
>
> Thanks to Ying Huang for pre-posting review feedback and suggestions!
>
> Thanks also to Nhat, Yosry, Johannes, Barry, Chengming, Usama and Ying for
> their helpful feedback, data reviews and suggestions!
>
> Co-development signoff request:
> ===============================
> I would like to thank Ryan Roberts for his original RFC [1] and request
> his co-developer signoff on patch 6 in this series. Thanks Ryan!


Ryan, could you help Kanchana out with a Signed-off-by please :)

>
>
>
> System setup for testing:
> =========================
> Testing of this patch-series was done with mm-unstable as of 9-27-2024,
> commit de2fbaa6d9c3576ec7133ed02a370ec9376bf000. Data was gathered
> without/with this patch-series, on an Intel Sapphire Rapids server,
> dual-socket 56 cores per socket, 4 IAA devices per socket, 503 GiB RAM and
> 525G SSD disk partition swap. Core frequency was fixed at 2500MHz.
>
> The vm-scalability "usemem" test was run in a cgroup whose memory.high
> was fixed at 150G. The is no swap limit set for the cgroup. 30 usemem
> processes were run, each allocating and writing 10G of memory, and sleeping
> for 10 sec before exiting:
>
> usemem --init-time -w -O -s 10 -n 30 10g
>
> Other kernel configuration parameters:
>
>     zswap compressors : zstd, deflate-iaa
>     zswap allocator   : zsmalloc
>     vm.page-cluster   : 2
>
> In the experiments where "deflate-iaa" is used as the zswap compressor,
> IAA "compression verification" is enabled by default
> (cat /sys/bus/dsa/drivers/crypto/verify_compress). Hence each IAA
> compression will be decompressed internally by the "iaa_crypto" driver, the
> crc-s returned by the hardware will be compared and errors reported in case
> of mismatches. Thus "deflate-iaa" helps ensure better data integrity as
> compared to the software compressors, and the experimental data listed
> below is with verify_compress set to "1".
>
> Total and average throughput are derived from the individual 30 processes'
> throughputs reported by usemem. elapsed/sys times are measured with perf.
>
> The vm stats and sysfs hugepages stats included with the performance data
> provide details on the swapout activity to zswap/swap device.
>
>
> Testing labels used in data summaries:
> ======================================
> The data refers to these test configurations and the before/after
> comparisons that they do:
>
>  before-case1:
>  -------------
>  mm-unstable 9-27-2024, CONFIG_THP_SWAP=N (compares zswap 4K vs. zswap 64K)
>
>  In this scenario, CONFIG_THP_SWAP=N results in 64K/2M folios to be split
>  into 4K folios that get processed by zswap.
>
>  before-case2:
>  -------------
>  mm-unstable 9-27-2024, CONFIG_THP_SWAP=Y (compares SSD swap large folios vs. zswap large folios)
>
>  In this scenario, CONFIG_THP_SWAP=Y results in zswap rejecting large
>  folios, which will then be stored by the SSD swap device.
>
>  after:
>  ------
>  v8 of this patch-series, CONFIG_THP_SWAP=Y
>
>  The "after" is CONFIG_THP_SWAP=Y and v8 of this patch-series, that results
>  in 64K/2M folios to not be split, and to be processed by zswap_store.
>
>
> Regression Testing:
> ===================
> I ran vm-scalability usemem without large folios, i.e., only 4K folios with
> mm-unstable and this patch-series. The main goal was to make sure that
> there is no functional or performance regression wrt the earlier zswap
> behavior for 4K folios, now that 4K folios will be processed by the new
> zswap_store() code.
>
> The data indicates there is no significant regression.
>
>  -------------------------------------------------------------------------------
>  4K folios:
>  ==========
>
>  zswap compressor                zstd          zstd        zstd  zstd v8 zstd v8
>                          before-case1  before-case2       after      vs.     vs.
>                                                                    case1   case2
>  -------------------------------------------------------------------------------
>  Total throughput (KB/s)    4,793,363     4,880,978   4,813,151       0%     -1%
>  Average throughput (KB/s)    159,778       162,699     160,438       0%     -1%
>  elapsed time (sec)            130.14        123.17      127.21       2%     -3%
>  sys time (sec)              3,135.53      2,985.64    3,110.53       1%     -4%
>
>  memcg_high                   446,826       444,626     448,231
>  memcg_swap_fail                    0             0           0
>  pswpout                            0             0           0
>  pswpin                             0             0           0
>  zswpout                   48,932,107    48,931,971  48,931,584
>  zswpin                           383           386         388
>  thp_swpout                         0             0           0
>  thp_swpout_fallback                0             0           0
>  64kB-mthp_swpout_fallback          0             0           0
>  pgmajfault                     3,063         3,077       3,082
>  swap_ra                           93            94          93
>  swap_ra_hit                       47            47          47
>  ZSWPOUT-64kB                     n/a           n/a           0
>  SWPOUT-64kB                        0             0           0
>  -------------------------------------------------------------------------------
>
>
> Performance Testing:
> ====================
>
> We list the data for 64K folios with before/after data per-compressor,
> followed by the same for 2M pmd-mappable folios.
>
>
>  -------------------------------------------------------------------------------
>  64K folios: zstd:
>  =================
>
>  zswap compressor                zstd          zstd         zstd      zstd v8
>                          before-case1  before-case2        after     vs.    vs.
>                                                                    case1  case2
>  -------------------------------------------------------------------------------
>  Total throughput (KB/s)    5,222,213     1,076,611    6,227,367     19%    478%
>  Average throughput (KB/s)    174,073        35,887      207,578     19%    478%
>  elapsed time (sec)            120.50        347.16       109.21      9%     69%


The diff here is supposed to be negative, right?
(Same for the below results)

Otherwise the results are looking really good, we have come a long way
since the first version :)

Thanks for working on this! I will look at individual patches later
today or early next week.

>
>  sys time (sec)              2,930.33        248.16     2,609.22     11%   -951%
>  memcg_high                   416,773       552,200      482,703
>  memcg_swap_fail            3,192,906         1,293          944
>  pswpout                            0    40,778,448            0
>  pswpin                             0            16            0
>  zswpout                   48,931,583        20,903   48,931,271
>  zswpin                           384           363          392
>  thp_swpout                         0             0            0
>  thp_swpout_fallback                0             0            0
>  64kB-mthp_swpout_fallback  3,192,906         1,293          944
>  pgmajfault                     3,452         3,072        3,095
>  swap_ra                           90            87          100
>  swap_ra_hit                       42            43           56
>  ZSWPOUT-64kB                     n/a           n/a    3,057,260
>  SWPOUT-64kB                        0     2,548,653            0
>  -------------------------------------------------------------------------------
>
>
>  -------------------------------------------------------------------------------
>  64K folios: deflate-iaa:
>  ========================
>
>  zswap compressor         deflate-iaa   deflate-iaa  deflate-iaa  deflate-iaa v8
>                          before-case1  before-case2        after      vs.    vs.
>                                                                    case1   case2
>  -------------------------------------------------------------------------------
>  Total throughput (KB/s)    5,652,608     1,089,180    6,315,000     12%    480%
>  Average throughput (KB/s)    188,420        36,306      210,500     12%    480%
>  elapsed time (sec)            102.90        343.35        91.11     11%     73%
>
>
>  sys time (sec)              2,246.86        213.53     1,939.31     14%   -808%
>  memcg_high                   576,104       502,907      612,505
>  memcg_swap_fail            4,016,117         1,407        1,660
>  pswpout                            0    40,862,080            0
>  pswpin                             0            20            0
>  zswpout                   61,163,423        22,444   57,317,607
>  zswpin                           401           368          449
>  thp_swpout                         0             0            0
>  thp_swpout_fallback                0             0            0
>  64kB-mthp_swpout_fallback  4,016,117         1,407        1,660
>  pgmajfault                     3,063         3,153        3,167
>  swap_ra                           96            93          149
>  swap_ra_hit                       46            45           89
>  ZSWPOUT-64kB                     n/a           n/a    3,580,673
>  SWPOUT-64kB                        0     2,553,880            0
>  -------------------------------------------------------------------------------
>
>
>  -------------------------------------------------------------------------------
>  2M folios: zstd:
>  ================
>
>  zswap compressor                zstd          zstd         zstd      zstd v8
>                          before-case1  before-case2        after     vs.    vs.
>                                                                    case1  case2
>  -------------------------------------------------------------------------------
>  Total throughput (KB/s)    5,895,500     1,109,694     6,460,111    10%    482%
>  Average throughput (KB/s)    196,516        36,989       215,337    10%    482%
>  elapsed time (sec)            108.77        334.28        105.92     3%     68%
>
>
>  sys time (sec)              2,657.14         94.88      2,436.24     8%  -2468%
>  memcg_high                    64,200        66,316        60,300
>  memcg_swap_fail              101,182            70            30
>  pswpout                            0    40,166,400             0
>  pswpin                             0             0             0
>  zswpout                   48,931,499        36,507    48,869,236
>  zswpin                           380           379           397
>  thp_swpout                         0        78,450             0
>  thp_swpout_fallback          101,182            70            30
>  2MB-mthp_swpout_fallback           0             0             0
>  pgmajfault                     3,067         3,417         4,765
>  swap_ra                           91            90         5,073
>  swap_ra_hit                       45            45         5,024
>  ZSWPOUT-2MB                      n/a           n/a        95,408
>  SWPOUT-2MB                         0        78,450             0
>  -------------------------------------------------------------------------------
>
>
>  -------------------------------------------------------------------------------
>  2M folios: deflate-iaa:
>  =======================
>
>  zswap compressor         deflate-iaa   deflate-iaa  deflate-iaa  deflate-iaa v8
>                          before-case1  before-case2        after      vs.    vs.
>                                                                    case1   case2
>  -------------------------------------------------------------------------------
>  Total throughput (KB/s)   6,286,587      1,126,785     7,569,560    20%    572%
>  Average throughput (KB/s)   209,552         37,559       252,318    20%    572%
>  elapsed time (sec)            96.19         333.03         81.96    15%     75%
>
>  sys time (sec)             2,141.44          99.96      1,768.41    17%  -1669%
>  memcg_high                   99,253         64,666        75,139
>  memcg_swap_fail             129,074             53            73
>  pswpout                           0     40,048,128             0
>  pswpin                            0              0             0
>  zswpout                  61,312,794         28,321    57,083,119
>  zswpin                          383            406           447
>  thp_swpout                        0         78,219             0
>  thp_swpout_fallback         129,074             53            73
>  2MB-mthp_swpout_fallback          0              0             0
>  pgmajfault                    3,430          3,077         7,133
>  swap_ra                          91            103        11,978
>  swap_ra_hit                      47             46        11,920
>  ZSWPOUT-2MB                     n/a            n/a       111,390
>  SWPOUT-2MB                        0         78,219             0
>  -------------------------------------------------------------------------------
>
> And finally, this is a comparison of deflate-iaa vs. zstd with v8 of this
> patch-series:
>
>  ---------------------------------------------
>                    zswap_store large folios v8
>                   Impr w/ deflate-iaa vs. zstd
>
>                        64K folios    2M folios
>  ---------------------------------------------
>  Throughput (KB/s)             1%          17%
>  elapsed time (sec)           17%          23%
>  sys time (sec)               26%          27%
>  ---------------------------------------------
>
>
> Conclusions based on the performance results:
> =============================================
>
>  v8 wrt before-case1:
>  --------------------
>  We see significant improvements in throughput, elapsed and sys time for
>  zstd and deflate-iaa, when comparing before-case1 (THP_SWAP=N) vs. after
>  (THP_SWAP=Y) with zswap_store large folios.
>
>  v8 wrt before-case2:
>  --------------------
>  We see even more significant improvements in throughput and elapsed time
>  for zstd and deflate-iaa, when comparing before-case2 (large-folio-SSD)
>  vs. after (large-folio-zswap). The sys time increases with
>  large-folio-zswap as expected, due to the CPU compression time
>  vs. asynchronous disk write times, as pointed out by Ying and Yosry.
>
>  In before-case2, when zswap does not store large folios, only allocations
>  and cgroup charging due to 4K folio zswap stores count towards the cgroup
>  memory limit. However, in the after scenario, with the introduction of
>  zswap_store() of large folios, there is an added component of the zswap
>  compressed pool usage from large folio stores from potentially all 30
>  processes, that gets counted towards the memory limit. As a result, we see
>  higher swapout activity in the "after" data.
>
>
> Summary:
> ========
> The v8 data presented above shows that zswap_store of large folios
> demonstrates good throughput/performance improvements compared to
> conventional SSD swap of large folios with a sufficiently large 525G SSD
> swap device. Hence, it seems reasonable for zswap_store to support large
> folios, so that further performance improvements can be implemented.
>
> In the experimental setup used in this patchset, we have enabled IAA
> compress verification to ensure additional hardware data integrity CRC
> checks not currently done by the software compressors. We see good
> throughput/latency improvements with deflate-iaa vs. zstd with zswap_store
> of large folios.
>
> Some of the ideas for further reducing latency that have shown promise in
> our experiments, are:
>
> 1) IAA compress/decompress batching.
> 2) Distributing compress jobs across all IAA devices on the socket.
>
> The tests run for this patchset are using only 1 IAA device per core, that
> avails of 2 compress engines on the device. In our experiments with IAA
> batching, we distribute compress jobs from all cores to the 8 compress
> engines available per socket. We further compress the pages in each folio
> in parallel in the accelerator. As a result, we improve compress latency
> and reclaim throughput.
>
> In decompress batching, we use swapin_readahead to generate a prefetch
> batch of 4K folios that we decompress in parallel in IAA.
>
>  ------------------------------------------------------------------------------
>                           IAA compress/decompress batching
>               Further improvements wrt v8 zswap_store Sequential
>                           subpage store using "deflate-iaa":
>
>                       "deflate-iaa" Batching  "deflate-iaa-canned" [2] Batching
>                           Additional Impr               Additional Impr
>                      64K folios    2M folios     64K folios    2M folios
>  ------------------------------------------------------------------------------
>  Throughput (KB/s)          35%          34%            44%          44%
>  elapsed time (sec)          9%          10%            14%          17%
>  sys time (sec)            0.4%           4%             8%          15%
>  ------------------------------------------------------------------------------
>
>
> With zswap IAA compress/decompress batching, we are able to demonstrate
> significant performance improvements and memory savings in server
> scalability experiments in highly contended system scenarios under
> significant memory pressure; as compared to software compressors. We hope
> to submit this work in subsequent patch series. The current patch-series is
> a prequisite for these future submissions.
>
> Thanks,
> Kanchana
>
>
> [1] https://lore.kernel.org/linux-mm/20231019110543.3284654-1-ryan.roberts@xxxxxxx/T/#u
> [2] https://patchwork.kernel.org/project/linux-crypto/cover/cover.1710969449.git.andre.glover@xxxxxxxxxxxxxxx/
>
>
> Changes since v7:
> =================
> 1) Rebased to mm-unstable as of 9-27-2024,
>    commit de2fbaa6d9c3576ec7133ed02a370ec9376bf000.
> 2) Added Nhat's 'Reviewed-by' to patches 1 and 2. Thanks Nhat!
> 3) Implemented one-time obj_cgroup_may_zswap and zswap_check_limits at the
>    start of zswap_store. Implemented one-time batch updates to cgroup zswap
>    charging (with total compressed bytes), zswap_stored_pages and the
>    memcg/vm zswpout event stats (with folio_nr_pages()) only for successful
>    stores at the end of zswap_store. Thanks Yosry and Johannes for guidance
>    on this!
> 4) Changed the existing zswap_pool_get() to zswap_pool_tryget(). Modified
>    zswap_pool_current_get() and zswap_pool_find_get() to call
>    zswap_pool_tryget(). Furthermore, zswap_store() obtains a reference to a
>    valid zswap_pool upfront by calling zswap_pool_tryget(), and errors out
>    if the tryget fails. Added a new zswap_pool_get() that calls
>    "percpu_ref_get(&pool->ref)" and is called in zswap_store_page(), as
>    suggested by Johannes & Yosry. Thanks both!
> 5) Provided a new count_objcg_events() API for batch event updates.
> 6) Changed "zswap_stored_pages" to atomic_long_t to support adding
>    folio_nr_pages() to it once a large folio is stored successfully.
> 7) Deleted the refactoring done in v7 for the xarray updates in
>    zswap_store_page(); and unwinding of stored offsets in zswap_store() in
>    case of errors, as suggested by Johannes.
> 8) Deleted the CONFIG_ZSWAP_STORE_THP_DEFAULT_ON config option and
>    "zswap_mthp_enabled" tunable, as recommended by Yosry, Johannes and
>    Nhat.
> 9) Replaced references to "mTHP" with "large folios"; organized
>    before/after data per-compressor for easier visual comparisons;
>    incorporated Nhat's feedback in the documentation updates; moved
>    changelog to the end. Thanks Johannes, Yosry and Nhat!
> 10) Moved the usemem testing configuration to 30 processes, each allocating
>     10G within a 150G memory-limit constrained cgroup, maintaining the
>     allocated memory for 10 sec before exiting. Thanks Ying for this
>     suggestion!
>
> Changes since v6:
> =================
> 1) Rebased to mm-unstable as of 9-23-2024,
>    commit acfabf7e197f7a5bedf4749dac1f39551417b049.
> 2) Refactored into smaller commits, as suggested by Yosry and
>    Chengming. Thanks both!
> 3) Reworded the commit log for patches 5 and 6 as per Yosry's
>    suggestion. Thanks Yosry!
> 4) Gathered data on a Sapphire Rapids server that has 823GiB SSD swap disk
>    partition. Also, all experiments are run with usemem --sleep 10, so that
>    the memory allocated by the 70 processes remains in memory
>    longer. Posted elapsed and sys times. Thanks to Yosry, Nhat and Ying for
>    their help with refining the performance characterization methodology.
> 5) Updated Documentation/admin-guide/mm/transhuge.rst as suggested by
>    Nhat. Thanks Nhat!
>
> Changes since v5:
> =================
> 1) Rebased to mm-unstable as of 8/29/2024,
>    commit 9287e4adbc6ab8fa04d25eb82e097fed877a4642.
> 2) Added CONFIG_ZSWAP_STORE_THP_DEFAULT_ON (off by default) to
>    enable/disable zswap_store() of mTHP folios. Thanks Nhat for the
>    suggestion to add a knob by which users can enable/disable this
>    change. Nhat, I hope this is along the lines of what you were
>    thinking.
> 3) Added vm-scalability usemem data with 4K folios with
>    CONFIG_ZSWAP_STORE_THP_DEFAULT_ON off, that I gathered to make sure
>    there is no regression with this change.
> 4) Added data with usemem with 64K and 2M THP for an alternate view of
>    before/after, as suggested by Yosry, so we can understand the impact
>    of when mTHPs are split into 4K folios in shrink_folio_list()
>    (CONFIG_THP_SWAP off) vs. not split (CONFIG_THP_SWAP on) and stored
>    in zswap. Thanks Yosry for this suggestion.
>
> Changes since v4:
> =================
> 1) Published before/after data with zstd, as suggested by Nhat (Thanks
>    Nhat for the data reviews!).
> 2) Rebased to mm-unstable from 8/27/2024,
>    commit b659edec079c90012cf8d05624e312d1062b8b87.
> 3) Incorporated the change in memcontrol.h that defines obj_cgroup_get() if
>    CONFIG_MEMCG is not defined, to resolve build errors reported by kernel
>    robot; as per Nhat's and Michal's suggestion to not require a separate
>    patch to fix the build errors (thanks both!).
> 4) Deleted all same-filled folio processing in zswap_store() of mTHP, as
>    suggested by Yosry (Thanks Yosry!).
> 5) Squashed the commits that define new mthp zswpout stat counters, and
>    invoke count_mthp_stat() after successful zswap_store()s; into a single
>    commit. Thanks Yosry for this suggestion!
>
> Changes since v3:
> =================
> 1) Rebased to mm-unstable commit 8c0b4f7b65fd1ca7af01267f491e815a40d77444.
>    Thanks to Barry for suggesting aligning with Ryan Roberts' latest
>    changes to count_mthp_stat() so that it's always defined, even when THP
>    is disabled. Barry, I have also made one other change in page_io.c
>    where count_mthp_stat() is called by count_swpout_vm_event(). I would
>    appreciate it if you can review this. Thanks!
>    Hopefully this should resolve the kernel robot build errors.
>
> Changes since v2:
> =================
> 1) Gathered usemem data using SSD as the backing swap device for zswap,
>    as suggested by Ying Huang. Ying, I would appreciate it if you can
>    review the latest data. Thanks!
> 2) Generated the base commit info in the patches to attempt to address
>    the kernel test robot build errors.
> 3) No code changes to the individual patches themselves.
>
> Changes since RFC v1:
> =====================
>
> 1) Use sysfs for zswpout mTHP stats, as per Barry Song's suggestion.
>    Thanks Barry!
> 2) Addressed some of the code review comments that Nhat Pham provided in
>    Ryan's initial RFC [1]:
>    - Added a comment about the cgroup zswap limit checks occuring once per
>      folio at the beginning of zswap_store().
>      Nhat, Ryan, please do let me know if the comments convey the summary
>      from the RFC discussion. Thanks!
>    - Posted data on running the cgroup suite's zswap kselftest.
> 3) Rebased to v6.11-rc3.
> 4) Gathered performance data with usemem and the rebased patch-series.
>
>
>
> Kanchana P Sridhar (8):
>   mm: Define obj_cgroup_get() if CONFIG_MEMCG is not defined.
>   mm: zswap: Modify zswap_compress() to accept a page instead of a
>     folio.
>   mm: zswap: Rename zswap_pool_get() to zswap_pool_tryget().
>   mm: Provide a new count_objcg_events() API for batch event updates.
>   mm: zswap: Modify zswap_stored_pages to be atomic_long_t.
>   mm: zswap: Support large folios in zswap_store().
>   mm: swap: Count successful large folio zswap stores in hugepage
>     zswpout stats.
>   mm: Document the newly added sysfs large folios zswpout stats.
>
>  Documentation/admin-guide/mm/transhuge.rst |   8 +-
>  fs/proc/meminfo.c                          |   2 +-
>  include/linux/huge_mm.h                    |   1 +
>  include/linux/memcontrol.h                 |  24 ++
>  include/linux/zswap.h                      |   2 +-
>  mm/huge_memory.c                           |   3 +
>  mm/page_io.c                               |   1 +
>  mm/zswap.c                                 | 254 +++++++++++++++------
>  8 files changed, 219 insertions(+), 76 deletions(-)
>
>
> base-commit: de2fbaa6d9c3576ec7133ed02a370ec9376bf000
> --
> 2.27.0
>





[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux