On Thu, Jun 8, 2023 at 6:59 PM Yu Zhao <yuzhao@xxxxxxxxxx> wrote: > > TLDR > ==== > Multichase in 64 microVMs achieved 6% more total samples (in ~4 hours) after this patchset [1]. > > Hardware > ======== > HOST $ lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Address sizes: 43 bits physical, 48 bits virtual > Byte Order: Little Endian > CPU(s): 128 > On-line CPU(s) list: 0-127 > Vendor ID: AuthenticAMD > Model name: AMD Ryzen Threadripper PRO 3995WX 64-Cores > CPU family: 23 > Model: 49 > Thread(s) per core: 2 > Core(s) per socket: 64 > Socket(s): 1 > Stepping: 0 > Frequency boost: disabled > CPU max MHz: 4308.3979 > CPU min MHz: 2200.0000 > BogoMIPS: 5390.20 > Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 > ... > Virtualization features: > Virtualization: AMD-V > Caches (sum of all): > L1d: 2 MiB (64 instances) > L1i: 2 MiB (64 instances) > L2: 32 MiB (64 instances) > L3: 256 MiB (16 instances) > NUMA: > NUMA node(s): 1 > NUMA node0 CPU(s): 0-127 > Vulnerabilities: > Itlb multihit: Not affected > L1tf: Not affected > Mds: Not affected > Meltdown: Not affected > Mmio stale data: Not affected > Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection > Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl > Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization > Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected > Srbds: Not affected > Tsx async abort: Not affected > > HOST $ numactl -H > available: 1 nodes (0) > node 0 cpus: 0-127 > node 0 size: 257542 MB > node 0 free: 224855 MB > node distances: > node 0 > 0: 10 > > HOST $ cat /sys/class/nvme/nvme0/model > INTEL SSDPF21Q800GB > > HOST $ cat /sys/class/nvme/nvme0/numa_node > 0 > > Software > ======== > HOST $ cat /etc/lsb-release > DISTRIB_ID=Ubuntu > DISTRIB_RELEASE=22.04 > DISTRIB_CODENAME=jammy > DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS" > > HOST $ uname -a > Linux x86 6.4.0-rc5+ #1 SMP PREEMPT_DYNAMIC Wed Jun 7 22:17:47 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux > > HOST $ cat /proc/swaps > Filename Type Size Used Priority > /dev/nvme0n1p2 partition 466838356 0 -2 > > HOST $ cat /sys/kernel/mm/lru_gen/enabled > 0x000f > > HOST $ cat /sys/kernel/mm/transparent_hugepage/enabled > always madvise [never] > > HOST $ cat /sys/kernel/mm/transparent_hugepage/defrag > always defer defer+madvise madvise [never] > > Procedure > ========= > HOST $ git clone https://github.com/google/multichase > > HOST $ <Build multichase> > HOST $ <Unpack /boot/initrd.img into ./initrd/> > > HOST $ cp multichase/multichase ./initrd/bin/ > HOST $ sed -i \ > "/^maybe_break top$/i multichase -t 2 -m 4g -n 28800; poweroff" \ I was reminded that I missed one parameter above, i.e., "/^maybe_break top$/i multichase -N -t 2 -m 4g -n 28800; poweroff" \ ^^ > ./initrd/init > > HOST $ <Pack ./initrd/ into ./initrd.img> > > HOST $ cat run_microvms.sh > memcgs=64 > > run() { > path=/sys/fs/cgroup/memcg$1 > > mkdir $path > echo $BASHPID >$path/cgroup.procs And one line here: echo 4000m >$path/memory.min # or the largest size that doesn't cause OOM kills > qemu-system-x86_64 -M microvm,accel=kvm -cpu host -smp 2 -m 6g \ > -nographic -kernel /boot/vmlinuz -initrd ./initrd.img \ > -append "console=ttyS0 loglevel=0" > } > > for ((memcg = 0; memcg < $memcgs; memcg++)); do > run $memcg & > done > > wait > > Results > ======= > Before [1] After Change > ---------------------------------------------- > Total samples 6824 7237 +6% > > Notes > ===== > [1] "mm: rmap: Don't flush TLB after checking PTE young for page > reference" was included so that the comparison is apples to > Apples. > https://lore.kernel.org/r/20220706112041.3831-1-21cnbao@xxxxxxxxx/
