On 13/11/2023 17:01, Zi Yan wrote: > From: Zi Yan <ziy@xxxxxxxxxx> > > Hi all, > > This patchset enables >0 order folio memory compaction, which is one of > the prerequisitions for large folio support[1]. It is on top of > mm-everything-2023-10-21-02-40 (also applies to mm-everything-2023-11-10-20-42 > without conflicts). > > Per patch vm-scalability results on a 8-CPU (Intel Xeon E5-2650 v4 @ 2.20GHz) > 16G VM are at the bottom and based on mm-everything-2023-10-21-02-40. > Each has 5 runs. Highlights: > > 1. mmap-pread-seq-mt: -18.3% if split folios during compaction and -7.7% > if migrating folios during compaction. > > 2. lru-file-readtwice: +41.5% if split folios during compaction and > +30.4% if migrating folios during compaction. > > 3. lru-file-mmap-read: +35.1% if split folios during compaction and > +32.3% if migrating folios during compaction. > > I will run experiments on bare metal and report results later. Hi Zi, I've taken this series into a backport on v6.1 that I maintain for my folios stuff - I had to do a fair bit of massaging to get it ported, so its been a good way to review the code. With the exception of the issue that Baolin has already pointed out, it looks good to me! So, assuming the forthcoming perf numbers don't show any problem: Reviewed-by: Ryan Roberts <ryan.roberts@xxxxxxx> > > > Changelog > === > > From RFC: > 1. Enabled >0 order folio compaction in the first patch by splitting all > to-be-migrated folios. (per Huang, Ying) > > 2. Stopped isolating compound pages with order greater than cc->order > to avoid wasting effort, since cc->order gives a hint that no free pages > with order greater than it exist, thus migrating the compound pages will fail. > (per Baolin Wang) > > 3. Retained the folio check within lru lock. (per Baolin Wang) > > 4. Made isolate_freepages_block() generate order-sorted multi lists. > (per Johannes Weiner) > > > Overview > === > > To support >0 order folio compaction, the patchset changes how free pages used > for migration are kept during compaction. Free pages used to be split into > order-0 pages that are post allocation processed (i.e., PageBuddy flag cleared, > page order stored in page->private is zeroed, and page reference is set to 1). > Now all free pages are kept in a MAX_ORDER+1 array of page lists based > on their order without post allocation process. When migrate_pages() asks for > a new page, one of the free pages, based on the requested page order, is > then processed and given out. > > > Optimizations > === > > 1. Free page split is added to increase migration success rate in case > a source page does not have a matched free page in the free page lists. > Free page merge is possible but not implemented, since existing > PFN-based buddy page merge algorithm requires the identification of > buddy pages, but free pages kept for memory compaction cannot have > PageBuddy set to avoid confusing other PFN scanners. > > 2. Sort source pages in ascending order before migration is added to > reduce free page split. Otherwise, high order free pages might be > prematurely split, causing undesired high order folio migration failures. > > > TODOs > === > > 1. Refactor free page post allocation and free page preparation code so > that compaction_alloc() and compaction_free() can call functions instead > of hard coding. > > 2. Make isolate_migratepages_block() produce a sorted list for > compaction to avoid sort_folio_by_order() in Patch 4. > > > Feel free to give comments and ask questions. > > Thanks. > > vm-scalability results > === > > ========================================================================================= > compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/qemu-vm/mmap-xread-seq-mt/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 12896955 +2.7% 13249322 -4.0% 12385175 ± 5% +1.1% 13033951 -0.4% 12845698 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 14626665 -5.5% 13825048 -1.7% 14384837 -1.9% 14345958 -2.0% 14338834 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/qemu-vm/mmap-pread-seq-mt/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 14314960 -18.3% 11691969 ± 2% -7.7% 13217795 -8.8% 13057395 -10.8% 12764204 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/qemu-vm/lru-file-readtwice/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 6498203 ± 6% +41.5% 9193356 +30.4% 8472024 +30.0% 8449054 ± 3% +32.3% 8596781 ± 2% vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 5440732 ± 10% +35.1% 7352631 ± 9% +32.3% 7200622 ± 5% +27.0% 6909673 ± 4% +25.9% 6849852 ± 5% vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/qemu-vm/lru-file-mmap-read-rand/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 1627989 +0.3% 1632141 +1.8% 1656877 +0.1% 1629816 -0.2% 1625213 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/8T/qemu-vm/anon-wx-seq-mt/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 6738043 +1.4% 6835096 +1.4% 6834329 +1.0% 6805401 +1.2% 6820637 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/8T/qemu-vm/anon-w-seq-mt/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 7644894 +0.2% 7661425 +0.4% 7676929 +1.0% 7719218 +0.3% 7664658 vm-scalability.throughput > > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-wx-rand-mt/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 521941 +3.3% 539254 +3.3% 539213 +1.0% 527048 +3.4% 539588 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 481028 +4.3% 501876 +2.4% 492713 +0.4% 482829 ± 2% +3.5% 497649 ± 2% vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-w-rand-mt/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 477063 +4.0% 496373 +4.0% 496283 +1.4% 483966 +3.2% 492216 ± 2% vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 458073 +3.3% 473056 +2.6% 469835 +0.3% 459266 +3.0% 471735 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/512G/qemu-vm/anon-cow-rand-mt/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 468158 +0.8% 472115 +1.4% 474547 +1.1% 473307 +2.1% 478173 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-xread-seq/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 3489946 +1.4% 3540036 +0.4% 3505413 +1.7% 3549852 +1.4% 3539309 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-xread-seq-mt/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 2621783 +1.0% 2647480 +0.6% 2636399 +0.7% 2641150 +0.1% 2624301 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/2T/qemu-vm/shm-pread-seq/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 3504293 +1.5% 3557664 -0.4% 3489639 +0.4% 3516842 +0.8% 3533944 vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/256G/qemu-vm/msync/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 919378 ± 6% +3.8% 954028 ± 10% +7.2% 985920 ± 6% +7.4% 987077 ± 5% +14.2% 1049473 ± 2% vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/256G/qemu-vm/msync-mt/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 2398063 +2.0% 2446844 -1.5% 2361096 ± 2% -3.1% 2324346 ± 4% -1.0% 2375055 ± 4% vm-scalability.throughput > > ========================================================================================= > compiler/kconfig/rootfs/runtime/size/tbox_group/test/testcase: > gcc-13/defconfig/debian/300s/128G/qemu-vm/truncate/vm-scalability > > commit: > 6.6.0-rc4-mm-everything-2023-10-21-02-40+ > 6.6.0-rc4-split-folio-in-compaction+ > 6.6.0-rc4-folio-migration-in-compaction+ > 6.6.0-rc4-folio-migration-free-page-split+ > 6.6.0-rc4-folio-migration-free-page-split-sort-src+ > > 6.6.0-rc4-mm-eve 6.6.0-rc4-split-folio-in-co 6.6.0-rc4-folio-migration-i 6.6.0-rc4-folio-migration-f 6.6.0-rc4-folio-migration-f > ---------------- --------------------------- --------------------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev %change %stddev %change %stddev > \ | \ | \ | \ | \ > 2.084e+08 ± 2% +1.5% 2.114e+08 ± 3% +6.5% 2.22e+08 ± 3% -2.3% 2.035e+08 ± 2% -2.1% 2.04e+08 ± 5% vm-scalability.throughput > > > > [1] https://lore.kernel.org/linux-mm/f8d47176-03a8-99bf-a813-b5942830fd73@xxxxxxx/ > > Zi Yan (4): > mm/compaction: enable compacting >0 order folios. > mm/compaction: add support for >0 order folio memory compaction. > mm/compaction: optimize >0 order folio compaction with free page > split. > mm/compaction: optimize >0 order folio compaction by sorting source > pages. > > mm/compaction.c | 280 ++++++++++++++++++++++++++++++++++++------------ > mm/internal.h | 7 +- > 2 files changed, 217 insertions(+), 70 deletions(-) >
