On Wed, Jan 10, 2024 at 12:16 PM Kairui Song <ryncsn@xxxxxxxxx> wrote: > > Yu Zhao <yuzhao@xxxxxxxxxx> 于2023年12月26日周二 06:01写道： > > > > On Mon, Dec 25, 2023 at 2:52 PM Yu Zhao <yuzhao@xxxxxxxxxx> wrote: > > > > > > On Mon, Dec 25, 2023 at 5:03 AM Kairui Song <ryncsn@xxxxxxxxx> wrote: > > > > > > > > Yu Zhao <yuzhao@xxxxxxxxxx> 于2023年12月25日周一 14:30写道： > > > > > > > > > > On Wed, Dec 20, 2023 at 1:24 AM Kairui Song <ryncsn@xxxxxxxxx> wrote: > > > > > > > > > > > > Yu Zhao <yuzhao@xxxxxxxxxx> 于2023年12月20日周三 16:17写道： > > > > > > > > > > > > > > On Tue, Dec 19, 2023 at 11:38 PM Yu Zhao <yuzhao@xxxxxxxxxx> wrote: > > > > > > > > > > > > > > > > On Tue, Dec 19, 2023 at 11:58 AM Kairui Song <ryncsn@xxxxxxxxx> wrote: > > > > > > > > > > > > > > > > > > Yu Zhao <yuzhao@xxxxxxxxxx> 于2023年12月19日周二 11:45写道： > > > > > > > > > > > > > > > > > > > > On Mon, Dec 18, 2023 at 8:21 PM Yu Zhao <yuzhao@xxxxxxxxxx> wrote: > > > > > > > > > > > > > > > > > > > > > > On Mon, Dec 18, 2023 at 11:05 AM Kairui Song <ryncsn@xxxxxxxxx> wrote: > > > > > > > > > > > > > > > > > > > > > > > > Yu Zhao <yuzhao@xxxxxxxxxx> 于2023年12月15日周五 12:56写道： > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Dec 14, 2023 at 04:51:00PM -0700, Yu Zhao wrote: > > > > > > > > > > > > > > On Thu, Dec 14, 2023 at 11:38 AM Kairui Song <ryncsn@xxxxxxxxx> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yu Zhao <yuzhao@xxxxxxxxxx> 于2023年12月14日周四 11:09写道： > > > > > > > > > > > > > > > > On Wed, Dec 13, 2023 at 12:59:14AM -0700, Yu Zhao wrote: > > > > > > > > > > > > > > > > > On Tue, Dec 12, 2023 at 8:03 PM Kairui Song <ryncsn@xxxxxxxxx> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Kairui Song <ryncsn@xxxxxxxxx> 于2023年12月12日周二 14:52写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yu Zhao <yuzhao@xxxxxxxxxx> 于2023年12月12日周二 06:07写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Dec 8, 2023 at 1:24 AM Kairui Song <ryncsn@xxxxxxxxx> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yu Zhao <yuzhao@xxxxxxxxxx> 于2023年12月8日周五 14:14写道： > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unmapped folios accessed through file descriptors can be > > > > > > > > > > > > > > > > > > > > > > underprotected. Those folios are added to the oldest generation based > > > > > > > > > > > > > > > > > > > > > > on: > > > > > > > > > > > > > > > > > > > > > > 1. The fact that they are less costly to reclaim (no need to walk the > > > > > > > > > > > > > > > > > > > > > > rmap and flush the TLB) and have less impact on performance (don't > > > > > > > > > > > > > > > > > > > > > > cause major PFs and can be non-blocking if needed again). > > > > > > > > > > > > > > > > > > > > > > 2. The observation that they are likely to be single-use. E.g., for > > > > > > > > > > > > > > > > > > > > > > client use cases like Android, its apps parse configuration files > > > > > > > > > > > > > > > > > > > > > > and store the data in heap (anon); for server use cases like MySQL, > > > > > > > > > > > > > > > > > > > > > > it reads from InnoDB files and holds the cached data for tables in > > > > > > > > > > > > > > > > > > > > > > buffer pools (anon). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > However, the oldest generation can be very short lived, and if so, it > > > > > > > > > > > > > > > > > > > > > > doesn't provide the PID controller with enough time to respond to a > > > > > > > > > > > > > > > > > > > > > > surge of refaults. (Note that the PID controller uses weighted > > > > > > > > > > > > > > > > > > > > > > refaults and those from evicted generations only take a half of the > > > > > > > > > > > > > > > > > > > > > > whole weight.) In other words, for a short lived generation, the > > > > > > > > > > > > > > > > > > > > > > moving average smooths out the spike quickly. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To fix the problem: > > > > > > > > > > > > > > > > > > > > > > 1. For folios that are already on LRU, if they can be beyond the > > > > > > > > > > > > > > > > > > > > > > tracking range of tiers, i.e., five accesses through file > > > > > > > > > > > > > > > > > > > > > > descriptors, move them to the second oldest generation to give them > > > > > > > > > > > > > > > > > > > > > > more time to age. (Note that tiers are used by the PID controller > > > > > > > > > > > > > > > > > > > > > > to statistically determine whether folios accessed multiple times > > > > > > > > > > > > > > > > > > > > > > through file descriptors are worth protecting.) > > > > > > > > > > > > > > > > > > > > > > 2. When adding unmapped folios to LRU, adjust the placement of them so > > > > > > > > > > > > > > > > > > > > > > that they are not too close to the tail. The effect of this is > > > > > > > > > > > > > > > > > > > > > > similar to the above. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Android, launching 55 apps sequentially: > > > > > > > > > > > > > > > > > > > > > > Before After Change > > > > > > > > > > > > > > > > > > > > > > workingset_refault_anon 25641024 25598972 0% > > > > > > > > > > > > > > > > > > > > > > workingset_refault_file 115016834 106178438 -8% > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yu, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks you for your amazing works on MGLRU. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I believe this is the similar issue I was trying to resolve previously: > > > > > > > > > > > > > > > > > > > > > https://lwn.net/Articles/945266/ > > > > > > > > > > > > > > > > > > > > > The idea is to use refault distance to decide if the page should be > > > > > > > > > > > > > > > > > > > > > place in oldest generation or some other gen, which per my test, > > > > > > > > > > > > > > > > > > > > > worked very well, and we have been using refault distance for MGLRU in > > > > > > > > > > > > > > > > > > > > > multiple workloads. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are a few issues left in my previous RFC series, like anon pages > > > > > > > > > > > > > > > > > > > > > in MGLRU shouldn't be considered, I wanted to collect feedback or test > > > > > > > > > > > > > > > > > > > > > cases, but unfortunately it seems didn't get too much attention > > > > > > > > > > > > > > > > > > > > > upstream. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think both this patch and my previous series are for solving the > > > > > > > > > > > > > > > > > > > > > file pages underpertected issue, and I did a quick test using this > > > > > > > > > > > > > > > > > > > > > series, for mongodb test, refault distance seems still a better > > > > > > > > > > > > > > > > > > > > > solution (I'm not saying these two optimization are mutually exclusive > > > > > > > > > > > > > > > > > > > > > though, just they do have some conflicts in implementation and solving > > > > > > > > > > > > > > > > > > > > > similar problem): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Previous result: > > > > > > > > > > > > > > > > > > > > > ================================================================== > > > > > > > > > > > > > > > > > > > > > Execution Results after 905 seconds > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > > > > > > > > > > > Executed Time (µs) Rate > > > > > > > > > > > > > > > > > > > > > STOCK_LEVEL 2542 27121571486.2 0.09 txn/s > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > > > > > > > > > > > TOTAL 2542 27121571486.2 0.09 txn/s > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > This patch: > > > > > > > > > > > > > > > > > > > > > ================================================================== > > > > > > > > > > > > > > > > > > > > > Execution Results after 900 seconds > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > > > > > > > > > > > Executed Time (µs) Rate > > > > > > > > > > > > > > > > > > > > > STOCK_LEVEL 1594 27061522574.4 0.06 txn/s > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > > > > > > > > > > > TOTAL 1594 27061522574.4 0.06 txn/s > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unpatched version is always around ~500. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the test results! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think there are a few points here: > > > > > > > > > > > > > > > > > > > > > - Refault distance make use of page shadow so it can better > > > > > > > > > > > > > > > > > > > > > distinguish evicted pages of different access pattern (re-access > > > > > > > > > > > > > > > > > > > > > distance). > > > > > > > > > > > > > > > > > > > > > - Throttled refault distance can help hold part of workingset when > > > > > > > > > > > > > > > > > > > > > memory is too small to hold the whole workingset. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So maybe part of this patch and the bits of previous series can be > > > > > > > > > > > > > > > > > > > > > combined to work better on this issue, how do you think? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'll try to find some time this week to look at your RFC. It'd be a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yu, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm working on V4 of the RFC now, which just update some comments, and > > > > > > > > > > > > > > > > > > skip anon page re-activation in refault path for mglru which was not > > > > > > > > > > > > > > > > > > very helpful, only some tiny adjustment. > > > > > > > > > > > > > > > > > > And I found it easier to test with fio, using following test script: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > #!/bin/bash > > > > > > > > > > > > > > > > > > swapoff -a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > modprobe brd rd_nr=1 rd_size=16777216 > > > > > > > > > > > > > > > > > > mkfs.ext4 /dev/ram0 > > > > > > > > > > > > > > > > > > mount /dev/ram0 /mnt > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > mkdir -p /sys/fs/cgroup/benchmark > > > > > > > > > > > > > > > > > > cd /sys/fs/cgroup/benchmark > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > echo 4G > memory.max > > > > > > > > > > > > > > > > > > echo $$ > cgroup.procs > > > > > > > > > > > > > > > > > > echo 3 > /proc/sys/vm/drop_caches > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fio -name=mglru --numjobs=12 --directory=/mnt --size=1024m \ > > > > > > > > > > > > > > > > > > --buffered=1 --ioengine=io_uring --iodepth=128 \ > > > > > > > > > > > > > > > > > > --iodepth_batch_submit=32 --iodepth_batch_complete=32 \ > > > > > > > > > > > > > > > > > > --rw=randread --random_distribution=zipf:0.5 --norandommap \ > > > > > > > > > > > > > > > > > > --time_based --ramp_time=5m --runtime=5m --group_reporting > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > zipf:0.5 is used here to simulate a cached read with slight bias > > > > > > > > > > > > > > > > > > towards certain pages. > > > > > > > > > > > > > > > > > > Unpatched 6.7-rc4: > > > > > > > > > > > > > > > > > > Run status group 0 (all jobs): > > > > > > > > > > > > > > > > > > READ: bw=6548MiB/s (6866MB/s), 6548MiB/s-6548MiB/s > > > > > > > > > > > > > > > > > > (6866MB/s-6866MB/s), io=1918GiB (2060GB), run=300001-300001msec > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Patched with RFC v4: > > > > > > > > > > > > > > > > > > Run status group 0 (all jobs): > > > > > > > > > > > > > > > > > > READ: bw=7270MiB/s (7623MB/s), 7270MiB/s-7270MiB/s > > > > > > > > > > > > > > > > > > (7623MB/s-7623MB/s), io=2130GiB (2287GB), run=300001-300001msec > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Patched with this series: > > > > > > > > > > > > > > > > > > Run status group 0 (all jobs): > > > > > > > > > > > > > > > > > > READ: bw=7098MiB/s (7442MB/s), 7098MiB/s-7098MiB/s > > > > > > > > > > > > > > > > > > (7442MB/s-7442MB/s), io=2079GiB (2233GB), run=300002-300002msec > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > MGLRU off: > > > > > > > > > > > > > > > > > > Run status group 0 (all jobs): > > > > > > > > > > > > > > > > > > READ: bw=6525MiB/s (6842MB/s), 6525MiB/s-6525MiB/s > > > > > > > > > > > > > > > > > > (6842MB/s-6842MB/s), io=1912GiB (2052GB), run=300002-300002msec > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - If I change zipf:0.5 to random: > > > > > > > > > > > > > > > > > > Unpatched 6.7-rc4: > > > > > > > > > > > > > > > > > > Patched with this series: > > > > > > > > > > > > > > > > > > Run status group 0 (all jobs): > > > > > > > > > > > > > > > > > > READ: bw=5975MiB/s (6265MB/s), 5975MiB/s-5975MiB/s > > > > > > > > > > > > > > > > > > (6265MB/s-6265MB/s), io=1750GiB (1879GB), run=300002-300002msec > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Patched with RFC v4: > > > > > > > > > > > > > > > > > > Run status group 0 (all jobs): > > > > > > > > > > > > > > > > > > READ: bw=5987MiB/s (6278MB/s), 5987MiB/s-5987MiB/s > > > > > > > > > > > > > > > > > > (6278MB/s-6278MB/s), io=1754GiB (1883GB), run=300001-300001msec > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Patched with this series: > > > > > > > > > > > > > > > > > > Run status group 0 (all jobs): > > > > > > > > > > > > > > > > > > READ: bw=5839MiB/s (6123MB/s), 5839MiB/s-5839MiB/s > > > > > > > > > > > > > > > > > > (6123MB/s-6123MB/s), io=1711GiB (1837GB), run=300001-300001msec > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > MGLRU off: > > > > > > > > > > > > > > > > > > Run status group 0 (all jobs): > > > > > > > > > > > > > > > > > > READ: bw=5689MiB/s (5965MB/s), 5689MiB/s-5689MiB/s > > > > > > > > > > > > > > > > > > (5965MB/s-5965MB/s), io=1667GiB (1790GB), run=300003-300003msec > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > fio uses ramdisk so LRU accuracy will have smaller impact. The Mongodb > > > > > > > > > > > > > > > > > > test I provided before uses a SATA SSD so it will have a much higher > > > > > > > > > > > > > > > > > > impact. I'll provides a script to setup the test case and run it, it's > > > > > > > > > > > > > > > > > > more complex to setup than fio since involving setting up multiple > > > > > > > > > > > > > > > > > > replicas and auth and hundreds of GB of test fixtures, I'm currently > > > > > > > > > > > > > > > > > > occupied by some other tasks but will try best to send them out as > > > > > > > > > > > > > > > > > > soon as possible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks! Apparently your RFC did show better IOPS with both access > > > > > > > > > > > > > > > > > patterns, which was a surprise to me because it had higher refaults > > > > > > > > > > > > > > > > > and usually higher refautls result in worse performance. > > > > > > > > > > > > > > > > > > > > > > > > > > And thanks for providing the refaults I requested for -- your data > > > > > > > > > > > > > below confirms what I mentioned above: > > > > > > > > > > > > > > > > > > > > > > > > > > For fio: > > > > > > > > > > > > > Your RFC This series Change > > > > > > > > > > > > > workingset_refault_file 628192729 596790506 -5% > > > > > > > > > > > > > IOPS 1862k 1830k -2% > > > > > > > > > > > > > > > > > > > > > > > > > > For MongoDB: > > > > > > > > > > > > > Your RFC This series Change > > > > > > > > > > > > > workingset_refault_anon 10512 35277 +30% > > > > > > > > > > > > > workingset_refault_file 22751782 20335355 -11% > > > > > > > > > > > > > total 22762294 20370632 -11% > > > > > > > > > > > > > TPS 0.09 0.06 -33% > > > > > > > > > > > > > > > > > > > > > > > > > > For MongoDB, this series should be a big win (but apparently it's not), > > > > > > > > > > > > > especially when using zram, since an anon refault should be a lot > > > > > > > > > > > > > cheaper than a file refault. > > > > > > > > > > > > > > > > > > > > > > > > > > So, I'm baffled... > > > > > > > > > > > > > > > > > > > > > > > > > > One important detail I forgot to mention: based on your data from > > > > > > > > > > > > > lru_gen_full, I think there is another difference between our Kconfigs: > > > > > > > > > > > > > > > > > > > > > > > > > > Your Kconfig My Kconfig Max possible > > > > > > > > > > > > > LRU_REFS_WIDTH 1 2 2 > > > > > > > > > > > > > > > > > > > > > > > > Hi Yu, > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the info, my fault, I forgot to update my config as I was > > > > > > > > > > > > testing some other features. > > > > > > > > > > > > Buf after I changed LRU_REFS_WIDTH to 2 by disabling IDLE_PAGE, thing > > > > > > > > > > > > got much worse for MongoDB test: > > > > > > > > > > > > > > > > > > > > > > > > With LRU_REFS_WIDTH == 2: > > > > > > > > > > > > > > > > > > > > > > > > This patch: > > > > > > > > > > > > ================================================================== > > > > > > > > > > > > Execution Results after 919 seconds > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > > Executed Time (µs) Rate > > > > > > > > > > > > STOCK_LEVEL 488 27598136201.9 0.02 txn/s > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > > TOTAL 488 27598136201.9 0.02 txn/s > > > > > > > > > > > > > > > > > > > > > > > > memcg 86 /system.slice/docker-1c3a90be9f0a072f5719332419550cd0e1455f2cd5863bc2780ca4d3f913ece5.scope > > > > > > > > > > > > node 0 > > > > > > > > > > > > 1 948187 0x 0x > > > > > > > > > > > > 0 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 1 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 2 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 3 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 2 948187 0 6051788· > > > > > > > > > > > > 0 0r 0e 0p 11916r > > > > > > > > > > > > 66442e 0p > > > > > > > > > > > > 1 0r 0e 0p 903r > > > > > > > > > > > > 16888e 0p > > > > > > > > > > > > 2 0r 0e 0p 459r > > > > > > > > > > > > 9764e 0p > > > > > > > > > > > > 3 0r 0e 0p 0r > > > > > > > > > > > > 0e 2874p > > > > > > > > > > > > 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 3 948187 1353160 6351· > > > > > > > > > > > > 0 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 1 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 2 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 3 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 4 73045 23573 12· > > > > > > > > > > > > 0 0R 0T 0 3498607R > > > > > > > > > > > > 4868605T 0· > > > > > > > > > > > > 1 0R 0T 0 3012246R > > > > > > > > > > > > 3270261T 0· > > > > > > > > > > > > 2 0R 0T 0 2498608R > > > > > > > > > > > > 2839104T 0· > > > > > > > > > > > > 3 0R 0T 0 0R > > > > > > > > > > > > 1983947T 0· > > > > > > > > > > > > 1486579L 0O 1380614Y 2945N > > > > > > > > > > > > 2945F 2734A > > > > > > > > > > > > > > > > > > > > > > > > workingset_refault_anon 0 > > > > > > > > > > > > workingset_refault_file 18130598 > > > > > > > > > > > > > > > > > > > > > > > > total used free shared buff/cache available > > > > > > > > > > > > Mem: 31978 6705 312 20 24960 24786 > > > > > > > > > > > > Swap: 31977 4 31973 > > > > > > > > > > > > > > > > > > > > > > > > RFC: > > > > > > > > > > > > ================================================================== > > > > > > > > > > > > Execution Results after 908 seconds > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > > Executed Time (µs) Rate > > > > > > > > > > > > STOCK_LEVEL 2252 27159962888.2 0.08 txn/s > > > > > > > > > > > > ------------------------------------------------------------------ > > > > > > > > > > > > TOTAL 2252 27159962888.2 0.08 txn/s > > > > > > > > > > > > > > > > > > > > > > > > workingset_refault_anon 22585 > > > > > > > > > > > > workingset_refault_file 22715256 > > > > > > > > > > > > > > > > > > > > > > > > memcg 66 /system.slice/docker-0989446ff78106e32d3f400a0cf371c9a703281bded86d6d6bb1af706ebb25da.scope > > > > > > > > > > > > node 0 > > > > > > > > > > > > 22 563007 2274 1198225· > > > > > > > > > > > > 0 0r 1e 0p 0r > > > > > > > > > > > > 697076e 0p > > > > > > > > > > > > 1 0r 0e 0p 0r > > > > > > > > > > > > 0e 325661p > > > > > > > > > > > > 2 0r 0e 0p 0r > > > > > > > > > > > > 0e 888728p > > > > > > > > > > > > 3 0r 0e 0p 0r > > > > > > > > > > > > 0e 3602238p > > > > > > > > > > > > 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 23 532222 7525 4948747· > > > > > > > > > > > > 0 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 1 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 2 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 3 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 24 500367 1214667 3292· > > > > > > > > > > > > 0 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 1 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 2 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 3 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 0 0 0 0 > > > > > > > > > > > > 0 0· > > > > > > > > > > > > 25 469692 40797 466· > > > > > > > > > > > > 0 0R 271T 0 0R > > > > > > > > > > > > 1162165T 0· > > > > > > > > > > > > 1 0R 0T 0 774028R > > > > > > > > > > > > 1205332T 0· > > > > > > > > > > > > 2 0R 0T 0 0R > > > > > > > > > > > > 932484T 0· > > > > > > > > > > > > 3 0R 1T 0 0R > > > > > > > > > > > > 4252158T 0· > > > > > > > > > > > > 25178380L 156515O 23953602Y 59234N > > > > > > > > > > > > 49391F 48664A > > > > > > > > > > > > > > > > > > > > > > > > total used free shared buff/cache available > > > > > > > > > > > > Mem: 31978 6968 338 5 24671 24555 > > > > > > > > > > > > Swap: 31977 1533 30444 > > > > > > > > > > > > > > > > > > > > > > > > Using same mongodb config (a 3 replica cluster using the same config): > > > > > > > > > > > > { > > > > > > > > > > > > "net": { > > > > > > > > > > > > "bindIpAll": true, > > > > > > > > > > > > "ipv6": false, > > > > > > > > > > > > "maxIncomingConnections": 10000, > > > > > > > > > > > > }, > > > > > > > > > > > > "setParameter": { > > > > > > > > > > > > "disabledSecureAllocatorDomains": "*" > > > > > > > > > > > > }, > > > > > > > > > > > > "replication": { > > > > > > > > > > > > "oplogSizeMB": 10480, > > > > > > > > > > > > "replSetName": "issa-tpcc_0" > > > > > > > > > > > > }, > > > > > > > > > > > > "security": { > > > > > > > > > > > > "keyFile": "/data/db/keyfile" > > > > > > > > > > > > }, > > > > > > > > > > > > "storage": { > > > > > > > > > > > > "dbPath": "/data/db/", > > > > > > > > > > > > "syncPeriodSecs": 60, > > > > > > > > > > > > "directoryPerDB": true, > > > > > > > > > > > > "wiredTiger": { > > > > > > > > > > > > "engineConfig": { > > > > > > > > > > > > "cacheSizeGB": 5 > > > > > > > > > > > > } > > > > > > > > > > > > } > > > > > > > > > > > > }, > > > > > > > > > > > > "systemLog": { > > > > > > > > > > > > "destination": "file", > > > > > > > > > > > > "logAppend": true, > > > > > > > > > > > > "logRotate": "rename", > > > > > > > > > > > > "path": "/data/db/mongod.log", > > > > > > > > > > > > "verbosity": 0 > > > > > > > > > > > > } > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > > > > The test environment have 32g memory and 16 core. > > > > > > > > > > > > > > > > > > > > > > > > Per my analyze, the access pattern for the mongodb test is that page > > > > > > > > > > > > will be re-access long after it's evicted so PID controller won't > > > > > > > > > > > > protect higher tier. That RFC will make use of the long existing > > > > > > > > > > > > shadow to do feedback to PID/Gen so the result will be much better. > > > > > > > > > > > > Still need more adjusting though, will try to do a rebase on top of > > > > > > > > > > > > mm-unstable which includes your patch. > > > > > > > > > > > > > > > > > > > > > > > > I've no idea why the workingset_refault_* is higher in the better > > > > > > > > > > > > case, this a clearly an IO bound workload, Memory and IO is busy while > > > > > > > > > > > > CPU is not full... > > > > > > > > > > > > > > > > > > > > > > > > I've uploaded my local reproducer here: > > > > > > > > > > > > https://github.com/ryncsn/emm-test-project/tree/master/mongo-cluster > > > > > > > > > > > > https://github.com/ryncsn/py-tpcc > > > > > > > > > > > > > > > > > > > > > > Thanks for the repos -- I'm trying them right now. Which MongoDB > > > > > > > > > > > version did you use? setup.sh didn't seem to install it. > > > > > > > > > > > > > > > > > > > > > > Also do you have a QEMU image? It'd be a lot easier for me to > > > > > > > > > > > duplicate the exact environment by looking into it. > > > > > > > > > > > > > > > > > > > > I ended up using docker.io/mongodb/mongodb-community-server:latest, > > > > > > > > > > and it's not working: > > > > > > > > > > > > > > > > > > > > # docker exec -it mongo-r1 mongosh --eval \ > > > > > > > > > > '"rs.initiate({ > > > > > > > > > > _id: "issa-tpcc_0", > > > > > > > > > > members: [ > > > > > > > > > > {_id: 0, host: "mongo-r1"}, > > > > > > > > > > {_id: 1, host: "mongo-r2"}, > > > > > > > > > > {_id: 2, host: "mongo-r3"} > > > > > > > > > > ] > > > > > > > > > > })"' > > > > > > > > > > Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. > > > > > > > > > > Error: can only create exec sessions on running containers: container > > > > > > > > > > state improper > > > > > > > > > > > > > > > > > > Hi Yu, > > > > > > > > > > > > > > > > > > I've updated the test repo: > > > > > > > > > https://github.com/ryncsn/emm-test-project/tree/master/mongo-cluster > > > > > > > > > > > > > > > > > > I've tested it on top of latest Fedora Cloud Image 39 and it worked > > > > > > > > > well for me, the README now contains detailed and not hard to follow > > > > > > > > > steps to reproduce this test. > > > > > > > > > > > > > > > > Thanks. I was following the instructions down to the letter and it > > > > > > > > fell apart again at line 46 (./tpcc.py). > > > > > > > > > > > > > > I think you just broke it by > > > > > > > https://github.com/ryncsn/py-tpcc/commit/7b9b380d636cb84faa5b11b5562e531f924eeb7e > > > > > > > > > > > > > > (But it's also possible you actually wanted me to use this latest > > > > > > > commit but forgot to account for it in your instructions.) > > > > > > > > > > > > > > > Were you able to successfully run the benchmark on a fresh VM by > > > > > > > > following the instructions? If not, I'd appreciate it if you could do > > > > > > > > so and document all the missing steps. > > > > > > > > > > > > Ah, you are right, I attempted to convert it to Python3 but found it > > > > > > only brought more trouble, so I gave up and the instruction is still > > > > > > using Python2. However I accidentally pushed the WIP python3 convert > > > > > > commit... I've reset the repo to > > > > > > https://github.com/ryncsn/py-tpcc/commit/86e862c5cf3b2d1f51e0297742fa837c7a99ebf8, > > > > > > this is working well. Sorry for the inconvenient. > > > > > > > > > > Thanks -- I was able to reproduce results similar to yours. > > > > > > > > > > > > > Hi Yu, > > > > > > > > Thanks for the testing, and merry xmas. > > > > > > > > > It turned out the mystery (fewer refaults but worse performance) was caused by > > > > > 13.89% 13.89% kswapd0 [kernel.vmlinux] [k] > > > > > __list_del_entry_valid_or_report > > > > > > > > I'm not sure about this, if the task is CPU bounded, this could > > > > explain. But it's not, the performance gap is larger when tested on > > > > slow IO device. > > > > > > > > The iostat output during my test run: > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > 7.40 0.00 2.42 83.37 0.00 6.80 > > > > Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s > > > > %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util > > > > vda 35.00 0.80 167.60 17.20 6.90 3.50 > > > > 16.47 81.40 0.47 1.62 0.02 4.79 21.50 0.63 2.27 > > > > vdb 5999.30 4.80 104433.60 84.00 0.00 8.30 > > > > 0.00 63.36 6.54 1.31 39.25 17.41 17.50 0.17 100.00 > > > > zram0 0.00 0.00 0.00 0.00 0.00 0.00 > > > > 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > > > > > > I ran the benchmark on the slowest bare metal I have that roughly > > > matches your CPU/DRAM configurations (ThinkPad P1 G4 > > > https://support.lenovo.com/us/en/solutions/pd031426). > > > > > > But it seems you used a VM (vda/vdb) -- I never run performance > > > benchmarks in VMs because the host and hypervisor can complicate > > > things, for example, in this case, is it possible the host page cache > > > cached your disk image containing the database files? > > > > > > > You can see CPU is waiting for IO, %user is always around 10%. > > > > The hotspot you posted only take up 13.89% of the runtime, which > > > > shouldn't cause so much performance drop. > > > > > > > > > > > > > > Apparently Fedora has CONFIG_DEBUG_LIST=y by default, and after I > > > > > turned it off (the only change I made), this series showed better TPS > > > > > (I used"--duration=10800" for more reliable results): > > > > > v6.7-rc6 RFC [1] change > > > > > total txns 25024 24672 +1% > > > > > workingset_refault_anon 573668 680248 -16% > > > > > workingset_refault_file 260631976 265808452 -2% > > > > > > > > I have disabled CONFIG_DEBUG_LIST when doing performance comparison test. > > > > Also I'd suggest we both use the same distro you shared with me and > > the default .config except CONFIG_DEBUG_LIST=n, and v6.7-rc6 for now. > > > > (I'm attaching the default .config based on /boot/config-6.5.6-300.fc39.x86_64.) > > > > Hi Yu > > I've been adapting and testing the refault distance series based on > latest 6.7. Also I found a serious bug in my previous V3, so I updated > it here with some importance changes (using a seperate refault > distance model, instead of glueing to active/inactive model): > https://github.com/ryncsn/linux/commits/kasong/devel/refault-distance-2024-1/ > > So far I can conclude that previous result is not caused by host > cache, I setup a baremetal test environment, strictly using your > config, I gathered some data (I also updated the refault distance > patch series, updated version in link above, and also the baremetal > hava a fast NVME so the performance gap wasn't so large but still > stably observable): > > With latest 6.7 (Your config): > ================================================================== > Execution Results after 905 seconds > ------------------------------------------------------------------ > Executed Time (µs) Rate > STOCK_LEVEL 4025 27162035181.5 0.15 txn/s > ------------------------------------------------------------------ > TOTAL 4025 27162035181.5 0.15 txn/s > > vmstat: > workingset_nodes 82996 > workingset_refault_anon 269371 > workingset_refault_file 37671044 > workingset_activate_anon 121061 > workingset_activate_file 8927227 > workingset_restore_anon 121061 > workingset_restore_file 2578269 > workingset_nodereclaim 62394 > > lru_gen_full: > memcg 67 /machine.slice/libpod-38b33777db34724cf8edfbef1ac2e4fd0621f14151e241bbf1430d397d3dee51.scope/container > node 0 > 34 60565 21248 1254331 > 0 0r 0e 0p 121186r > 169948e 0p > 1 0r 0e 0p 156224r > 222553e 0p > 2 0r 0e 0p 0r > 0e 4227858p > 3 0r 0e 0p 0r > 0e 0p > 0 0 0 0 > 0 0 > 35 41132 714504 4300280 > 0 0 0 0 0 > 0 0 > 1 0 0 0 0 > 0 0 > 2 0 0 0 0 > 0 0 > 3 0 0 0 0 > 0 0 > 0 0 0 0 > 0 0 > 36 20586 473476 2105 > 0 0 0 0 0 > 0 0 > 1 0 0 0 0 > 0 0 > 2 0 0 0 0 > 0 0 > 3 0 0 0 0 > 0 0 > 0 0 0 0 > 0 0 > 37 2035 817 876 > 0 6647R 9116T 0 166836R > 871850T 0 > 1 0R 0T 0 110807R > 296447T 0 > 2 0R 268T 0 0R > 4655276T 0 > 3 0R 0T 0 0R > 0T 0 > 12510062L 639646O 11048666Y 45512N > 24520F 23613A > iostat: > avg-cpu: %user %nice %system %iowait %steal %idle > 76.29 0.00 12.09 3.50 1.44 6.69 > > Device tps kB_read/s kB_wrtn/s kB_dscd/s > kB_read kB_wrtn kB_dscd > dm-0 16.12 684.50 36.93 0.00 > 649996 35070 0 > dm-1 0.05 1.10 0.00 0.00 > 1044 0 0 > nvme0n1 16.47 700.22 39.09 0.00 664922 > 37118 0 > nvme1n1 4905.93 205287.92 1030.70 0.00 > 194939353 978740 0 > zram0 4440.17 5356.90 12404.81 0.00 > 5086856 11779480 0 > > free -m: > total used free shared buff/cache available > Mem: 31830 9475 403 0 21951 21918 > Swap: 31829 6500 25329 > > With latest refault distance series (Your config): > ================================================================== > Execution Results after 902 seconds > ------------------------------------------------------------------ > Executed Time (µs) Rate > STOCK_LEVEL 4260 27065448172.8 0.16 txn/s > ------------------------------------------------------------------ > TOTAL 4260 27065448172.8 0.16 txn/s > > workingset_nodes 113824 > workingset_refault_anon 293426 > workingset_refault_file 42700484 > workingset_activate_anon 0 > workingset_activate_file 13410174 > workingset_restore_anon 289106 > workingset_restore_file 5592042 > workingset_nodereclaim 33249 > > memcg 67 /machine.slice/libpod-8eff6b7b65e34fe0497ff5c0c88c750f6896c43a06bb26e8cd6470de596be76e.scope/container > node 0 > 15 261222 266350 65081 > 0 0r 0e 0p 185212r > 2314329e 0p > 1 0r 0e 0p 40887r > 710312e 0p > 2 0r 0e 0p 0r > 0e 5026031p > 3 0r 0e 0p 0r > 0e 0p > 0 0 0 0 > 0 0 > 16 199341 267661 5034442 > 0 0 0 0 0 > 0 0 > 1 0 0 0 0 > 0 0 > 2 0 0 0 0 > 0 0 > 3 0 0 0 0 > 0 0 > 0 0 0 0 > 0 0 > 17 120655 547852 592 > 0 0 0 0 0 > 0 0 > 1 0 0 0 0 > 0 0 > 2 0 0 0 0 > 0 0 > 3 0 0 0 0 > 0 0 > 0 0 0 0 > 0 0 > 18 55172 127769 3855 > 0 1910R 2975T 0 1894614R > 4375361T 0 > 1 0R 0T 0 2099208R > 2861460T 0 > 2 0R 27T 0 446000R > 5569781T 0 > 3 0R 0T 0 0R > 0T 0 > 2817512L 35421O 2579377Y 10452N > 5517F 5414A > > avg-cpu: %user %nice %system %iowait %steal %idle > 76.34 0.00 11.25 4.22 1.29 6.90 > > Device tps kB_read/s kB_wrtn/s kB_dscd/s > kB_read kB_wrtn kB_dscd > dm-0 12.85 563.18 30.75 0.00 > 532390 29070 0 > dm-1 0.05 1.10 0.00 0.00 > 1044 0 0 > nvme0n1 13.22 578.97 32.92 0.00 > 547315 31119 0 > nvme1n1 5384.11 229164.12 1038.95 0.00 > 216635713 982152 0 > zram0 3590.88 4730.84 9633.71 0.00 > 4472204 9107032 0 > > total used free shared buff/cache available > Mem: 31830 10854 520 0 20455 20541 > Swap: 31829 4508 27321 > > You see actually refault distance is now protecting more anon page, > total IO on ZRAM is lower, It's mostly CPU bound, and NVME is fast > enough, and result in a better performance. > > Things get more interesting if I disable page idle flag (so refs bits > is extended, in your config, refs bit is only one bit, so it maybe > overprotect file pages): > > Latest 6.7 (You config with page idle flag disabled): > ================================================================== > Execution Results after 904 seconds > ------------------------------------------------------------------ > Executed Time (µs) Rate > STOCK_LEVEL 4016 27122163703.9 0.15 txn/s > ------------------------------------------------------------------ > TOTAL 4016 27122163703.9 0.15 txn/s > > workingset_nodes 99637 > workingset_refault_anon 309548 > workingset_refault_file 45896663 > workingset_activate_anon 129401 > workingset_activate_file 18461236 > workingset_restore_anon 129400 > workingset_restore_file 4963707 > workingset_nodereclaim 43970 > > memcg 67 /machine.slice/libpod-7546463bd2b257a9b799817ca11bee1389d7deec20032529098520a89a207d7e.scope/container > node 0 > 27 103004 328070 269733 > 0 0r 0e 0p 509949r > 1957117e 0p > 1 0r 0e 0p 141642r > 319695e 0p > 2 0r 0e 0p 777835r > 793518e 0p > 3 0r 0e 0p 0r > 0e 4333835p > 0 0 0 0 > 0 0 > 28 82361 24748 5192182 > 0 0 0 0 0 > 0 0 > 1 0 0 0 0 > 0 0 > 2 0 0 0 0 > 0 0 > 3 0 0 0 0 > 0 0 > 0 0 0 0 > 0 0 > 29 57025 786386 5681 > 0 0 0 0 0 > 0 0 > 1 0 0 0 0 > 0 0 > 2 0 0 0 0 > 0 0 > 3 0 0 0 0 > 0 0 > 0 0 0 0 > 0 0 > 30 18619 76289 1273 > 0 4295R 8888T 0 222326R > 1044601T 0 > 1 0R 0T 0 117646R > 301735T 0 > 2 0R 0T 0 433431R > 825516T 0 > 3 0R 1T 0 0R > 4076839T 0 > 13369819L 603360O 11981074Y 47388N > 26235F 25276A > > avg-cpu: %user %nice %system %iowait %steal %idle > 74.90 0.00 11.96 4.92 1.62 6.60 > > Device tps kB_read/s kB_wrtn/s kB_dscd/s > kB_read kB_wrtn kB_dscd > dm-0 14.93 645.44 36.54 0.00 > 610150 34540 0 > dm-1 0.05 1.10 0.00 0.00 > 1044 0 0 > nvme0n1 15.30 661.23 38.71 0.00 > 625076 36589 0 > nvme1n1 6352.42 240726.35 1036.47 0.00 > 227565845 979808 0 > zram0 4189.65 4883.27 11876.36 0.00 > 4616304 11227080 0 > > total used free shared buff/cache available > Mem: 31830 9529 509 0 21791 21867 > Swap: 31829 6441 25388 > > Refault distance seriese (Your config with page idle flag disabled): > ================================================================== > Execution Results after 901 seconds > ------------------------------------------------------------------ > Executed Time (µs) Rate > STOCK_LEVEL 4268 27060267967.7 0.16 txn/s > ------------------------------------------------------------------ > TOTAL 4268 27060267967.7 0.16 txn/s > > workingset_nodes 115394 > workingset_refault_anon 144774 > workingset_refault_file 41055081 > workingset_activate_anon 8 > workingset_activate_file 13194460 > workingset_restore_anon 144629 > workingset_restore_file 187419 > workingset_nodereclaim 19598 > > memcg 66 /machine.slice/libpod-4866051af817731602b37017b0e71feb2a8f2cbaa949f577e0444af01b4f3b0c.scope/container > node 0 > 12 213402 18054 1287510 > 0 0r 0e 0p 0r > 15755e 0p > 1 0r 0e 0p 0r > 4771e 0p > 2 0r 0e 0p 908r > 6810e 0p > 3 0r 0e 0p 0r > 0e 3533888p > 0 0 0 0 > 0 0 > 13 141209 10690 3571958 > 0 0 0 0 0 > 0 0 > 1 0 0 0 0 > 0 0 > 2 0 0 0 0 > 0 0 > 3 0 0 0 0 > 0 0 > 0 0 0 0 > 0 0 > 14 69327 1165064 34657 > 0 0 0 0 0 > 0 0 > 1 0 0 0 0 > 0 0 > 2 0 0 0 0 > 0 0 > 3 0 0 0 0 > 0 0 > 0 0 0 0 > 0 0 > 15 6404 21574 3363 > 0 953R 1773T 0 1263395R > 3816639T 0 > 1 0R 0T 0 1164069R > 1936973T 0 > 2 0R 0T 0 350041R > 409121T 0 > 3 0R 3T 0 12305R > 4767303T 0 > 3622197L 36916O 3338446Y 10409N > 7120F 6945A > > avg-cpu: %user %nice %system %iowait %steal %idle > 75.79 0.00 10.68 3.91 1.18 8.44 > > Device tps kB_read/s kB_wrtn/s kB_dscd/s > kB_read kB_wrtn kB_dscd > dm-0 12.66 547.71 38.73 0.00 > 526782 37248 0 > dm-1 0.05 1.09 0.00 0.00 > 1044 0 0 > nvme0n1 13.02 563.23 40.86 0.00 > 541708 39297 0 > nvme1n1 4916.00 217529.48 1018.04 0.00 > 209217677 979136 0 > zram0 1744.90 1970.86 5009.75 0.00 > 1895556 4818328 0 > > total used free shared buff/cache available > Mem: 31830 11713 485 0 19630 19684 > Swap: 31829 2847 28982 > > And still refault distance series is better, and refault is also lower > for both anon/file pages. > > ------ > I did some more test using MySQL and other workflow, no performance > drop observed so far. > And with a loop MongoDB test (keep running 900s test in loop) using my > previous VM env > (the SATA SSD vdb is using cache bypass so not a host cache issue here) > I found one thing interesting (refs bit is set to 2 in config): > > Loop test using 6.7: > STOCK_LEVEL 874 27246011368.3 0.03 txn/s > STOCK_LEVEL 1533 27023181579.6 0.06 txn/s > STOCK_LEVEL 1122 28044867987.6 0.04 txn/s > STOCK_LEVEL 1032 27378070931.9 0.04 txn/s > STOCK_LEVEL 1021 27612530579.1 0.04 txn/s > STOCK_LEVEL 750 28076187896.3 0.03 txn/s > STOCK_LEVEL 780 27519993034.8 0.03 txn/s > Refault stat here: > workingset_refault_anon 126369 > workingset_refault_file 170389428 > STOCK_LEVEL 750 27464016123.5 0.03 txn/s > STOCK_LEVEL 780 27529550313.0 0.03 txn/s > STOCK_LEVEL 750 28296286486.1 0.03 txn/s > STOCK_LEVEL 690 27504193850.3 0.03 txn/s > STOCK_LEVEL 716 28089360754.5 0.03 txn/s > STOCK_LEVEL 607 27852180474.3 0.02 txn/s > STOCK_LEVEL 689 27703367075.4 0.02 txn/s > STOCK_LEVEL 630 28184685482.7 0.02 txn/s > STOCK_LEVEL 450 28667721196.2 0.02 txn/s > STOCK_LEVEL 450 28047985314.4 0.02 txn/s > STOCK_LEVEL 450 28125609857.3 0.02 txn/s > STOCK_LEVEL 420 27393478488.0 0.02 txn/s > STOCK_LEVEL 420 27435537312.3 0.02 txn/s > STOCK_LEVEL 420 29060748699.2 0.01 txn/s > STOCK_LEVEL 420 28155584095.2 0.01 txn/s > STOCK_LEVEL 420 27888635407.0 0.02 txn/s > STOCK_LEVEL 420 27307856858.5 0.02 txn/s > STOCK_LEVEL 420 28842280889.0 0.01 txn/s > STOCK_LEVEL 390 27640696814.1 0.01 txn/s > STOCK_LEVEL 420 28471605716.7 0.01 txn/s > STOCK_LEVEL 420 27648174237.5 0.02 txn/s > STOCK_LEVEL 420 27848217938.7 0.02 txn/s > STOCK_LEVEL 420 27344698602.2 0.02 txn/s > STOCK_LEVEL 420 27046819537.2 0.02 txn/s > STOCK_LEVEL 420 27855626843.2 0.02 txn/s > STOCK_LEVEL 420 27971873627.9 0.02 txn/s > STOCK_LEVEL 420 28007014046.4 0.01 txn/s > STOCK_LEVEL 420 28445164626.1 0.01 txn/s > STOCK_LEVEL 420 27902621006.5 0.02 txn/s > STOCK_LEVEL 420 28282574433.3 0.01 txn/s > STOCK_LEVEL 390 27161599608.7 0.01 txn/s > > Using refault distance seriese: > STOCK_LEVEL 2605 27120667462.8 0.10 txn/s > STOCK_LEVEL 3000 27106854857.2 0.11 txn/s > STOCK_LEVEL 2925 27066601064.4 0.11 txn/s > STOCK_LEVEL 2757 27035248005.2 0.10 txn/s > STOCK_LEVEL 1325 28053716046.8 0.05 txn/s > STOCK_LEVEL 717 27455091366.3 0.03 txn/s > STOCK_LEVEL 967 27404085208.2 0.04 txn/s > Refault stat here: > workingset_refault_anon 109337 > workingset_refault_file 191249716 > STOCK_LEVEL 716 27448213557.2 0.03 txn/s > STOCK_LEVEL 807 28607974517.8 0.03 txn/s > STOCK_LEVEL 760 28081442513.2 0.03 txn/s > STOCK_LEVEL 745 28594555797.6 0.03 txn/s > STOCK_LEVEL 450 27999536348.3 0.02 txn/s > STOCK_LEVEL 598 27095531895.4 0.02 txn/s > STOCK_LEVEL 711 27623112841.1 0.03 txn/s > STOCK_LEVEL 540 28358770820.6 0.02 txn/s > STOCK_LEVEL 480 27734277554.5 0.02 txn/s > STOCK_LEVEL 450 27313906125.3 0.02 txn/s > STOCK_LEVEL 480 27487299100.4 0.02 txn/s > STOCK_LEVEL 480 27804589683.5 0.02 txn/s > STOCK_LEVEL 480 28242205820.8 0.02 txn/s > STOCK_LEVEL 480 27540680102.3 0.02 txn/s > STOCK_LEVEL 450 27428645816.8 0.02 txn/s > STOCK_LEVEL 480 27946866129.2 0.02 txn/s > STOCK_LEVEL 480 27266068262.3 0.02 txn/s > STOCK_LEVEL 450 27267487051.5 0.02 txn/s > STOCK_LEVEL 480 27896369224.8 0.02 txn/s > STOCK_LEVEL 480 28784662706.1 0.02 txn/s > STOCK_LEVEL 450 27179853217.8 0.02 txn/s > STOCK_LEVEL 480 28170594101.7 0.02 txn/s > STOCK_LEVEL 450 28084651341.0 0.02 txn/s > STOCK_LEVEL 480 27901608868.6 0.02 txn/s > STOCK_LEVEL 480 27323790886.6 0.02 txn/s > STOCK_LEVEL 480 28891008895.4 0.02 txn/s > STOCK_LEVEL 480 27964563148.0 0.02 txn/s > STOCK_LEVEL 450 27942421198.4 0.02 txn/s > STOCK_LEVEL 480 28833968825.8 0.02 txn/s > STOCK_LEVEL 480 28090975437.9 0.02 txn/s > STOCK_LEVEL 480 27915246877.4 0.02 txn/s > > It seems the performance will drain as the test keep running (might be > caused by MongoDB anon usage rising or DB internal caching/logging), > that explains why for a long term test the performance gap seem to be > smaller. The VM have a poor IO performance so the test run speed is > much slower too, take a long time to warm up. > > But I think it's clear that refault distance series will boost the > performance during warm up, and for long time workload it's also > looking better, especially for low IO performance machines. > > I still can't explain about why workingset_refault is higher for the > better case in the VM environment... I can resetup/reboot/randomize > the test the the performance is same here. My guess is maybe related > to readahead or some kernel space IO path issue? The actual IO usage > is lower when refault distance series is applied. > > I notices a slight performance regression (1 - 3%) for pure in-mem FIO > though, the "bulk series" I sent previous can help improve it. > > There is a bug in my previous V3 that will cause PID controller to > lost control in long term (due to a bugged bit operation, my bad), > which I've fixed in link above, I can send out new series if you think > it's acceptable. Could you try the attached patch on the mainline v6.7 and see how it compares with the results above? Thanks.
Attachment:
mglru-6.7.patch
Description: Binary data
