TLDR ==== Memcached achieved 10% more operations per second (in ~4 hours) after this patchset [1]. Hardware ======== HOST $ lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 184 On-line CPU(s) list: 0-183 Model name: POWER9 (raw), altivec supported Model: 2.2 (pvr 004e 1202) Thread(s) per core: 4 Core(s) per socket: 23 Socket(s): 2 CPU max MHz: 3000.0000 CPU min MHz: 2300.0000 Caches (sum of all): L1d: 1.4 MiB (46 instances) L1i: 1.4 MiB (46 instances) L2: 12 MiB (24 instances) L3: 240 MiB (24 instances) NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0-91 NUMA node1 CPU(s): 92-183 Vulnerabilities: Itlb multihit: Not affected L1tf: Mitigation; RFI Flush, L1D private per thread Mds: Not affected Meltdown: Mitigation; RFI Flush, L1D private per thread Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Kernel entry/exit barrier (eieio) Spectre v1: Mitigation; __user pointer sanitization, ori31 speculation barrier enabled Spectre v2: Mitigation; Indirect branch serialisation (kernel only), Indirect branch cache disabled, Software link stack flush Srbds: Not affected Tsx async abort: Not affected HOST $ numactl -H available: 2 nodes (0-1) node 0 cpus: 0-91 node 0 size: 261659 MB node 0 free: 259152 MB node 1 cpus: 92-183 node 1 size: 261713 MB node 1 free: 261076 MB node distances: node 0 1 0: 10 40 1: 40 10 HOST $ cat /sys/class/nvme/nvme0/model INTEL SSDPF21Q800GB HOST $ cat /sys/class/nvme/nvme0/numa_node 0 Software ======== HOST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS" HOST $ uname -a Linux ppc 6.3.0 #1 SMP Sun Jun 4 18:26:37 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux HOST $ cat /proc/swaps Filename Type Size Used Priority /dev/nvme0n1p2 partition 466838272 0 -2 HOST $ cat /sys/kernel/mm/lru_gen/enabled 0x0009 HOST $ cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] HOST $ cat /sys/kernel/mm/transparent_hugepage/defrag always defer defer+madvise madvise [never] HOST $ qemu-system-ppc64 --version QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.6) Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers GUEST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS" GUEST $ cat /etc/memcached.conf ... -t 92 -m 262144 -B binary -s /var/run/memcached/memcached.sock -a 0766 GUEST $ memtier_benchmark -v memtier_benchmark 1.4.0 Copyright (C) 2011-2022 Redis Ltd. This is free software. You may redistribute copies of it under the terms of the GNU General Public License <http://www.gnu.org/licenses/gpl.html>. There is NO WARRANTY, to the extent permitted by law. Procedure ========= HOST $ sudo numactl -N 0 -m 0 qemu-system-ppc64 \ -M pseries,accel=kvm,kvm-type=HV -cpu host -smp 92 -m 270g -nographic -nic user \ -drive if=virtio,format=raw,file=/dev/nvme0n1p1 GUEST $ memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -c 1 -t 92 --pipeline 1 --ratio 1:0 \ --key-minimum=1 --key-maximum=120000000 --key-pattern=P:P \ -n allkeys -d 2000 GUEST $ memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -c 1 -t 92 --pipeline 1 --ratio 0:1 \ --key-minimum=1 --key-maximum=120000000 --key-pattern=R:R \ -n allkeys --randomize --distinct-client-seed Results ======= Before [1] After Change ------------------------------------------------- Ops/sec 721586.10 800210.12 +10% Avg. Latency 0.12546 0.11260 -10% p50 Latency 0.08700 0.08700 N/C p99 Latency 0.28700 0.24700 -13% Notes ===== [1] "mm: rmap: Don't flush TLB after checking PTE young for page reference" was included so that the comparison is apples to Apples. https://lore.kernel.org/r/20220706112041.3831-1-21cnbao@xxxxxxxxx/
