OpenWrt / MIPS benchmark with MGLRU

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



TLDR
====
RAM utilization  Throughput (95% CI)  P99 Latency (95% CI)
----------------------------------------------------------
~90%             NS                   NS
~110%            +[12, 16]%           -[20, 22]%

Abbreviations
=============
CI:   confidence interval
NS:   no statistically significant difference
DUT:  device under test
ATE:  automatic test equipment

Rational
========
1. OpenWrt is the most popular distro for WiFi routers; many of its
   targets use big endianness [1].
2. 4 out of the top 5 bestselling WiFi routers in the US use MIPS [2];
   MIPS uses software-managed TLB.
3. Memcached is the best available memory benchmark on OpenWrt;
   admittedly such a use case is very limited in the real world.

Hardware
========
DUT: Ubiquiti EdgeRouter (ER-8) [3]

DUT # cat /proc/cpuinfo
system type             : UBNT_E200 (CN6120p1.1-800-NSP)
machine                 : Unknown
processor               : 0
cpu model               : Cavium Octeon II V0.1
BogoMIPS                : 1600.00
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 128
extra interrupt vector  : yes
hardware watchpoint     : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
isa                     : mips1 mips2 mips3 mips4 mips5 mips32r1 mips32r2 mips64r1 mips64r2
ASEs implemented        :
Options implemented     : tlb rixiex 4kex octeon_cache 32fpr prefetch mcheck ejtag llsc rixi lpa vtag_icache userlocal perf_cntr_intr_bit perf
shadow register sets    : 1
kscratch registers      : 3
package                 : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

processor               : 1
cpu model               : Cavium Octeon II V0.1
BogoMIPS                : 1600.00
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 128
extra interrupt vector  : yes
hardware watchpoint     : yes, count: 2, address/irw mask: [0x0ffc, 0x0ffb]
isa                     : mips1 mips2 mips3 mips4 mips5 mips32r1 mips32r2 mips64r1 mips64r2
ASEs implemented        :
Options implemented     : tlb rixiex 4kex octeon_cache 32fpr prefetch mcheck ejtag llsc rixi lpa vtag_icache userlocal perf_cntr_intr_bit perf
shadow register sets    : 1
kscratch registers      : 3
package                 : 0
core                    : 1
VCED exceptions         : not available
VCEI exceptions         : not available

DUT # cat /proc/meminfo
MemTotal:        1991964 kB
MemFree:         1917304 kB
MemAvailable:    1896856 kB
Buffers:               4 kB
Cached:            33464 kB
SwapCached:            0 kB
Active:             1316 kB
Inactive:          33500 kB
Active(anon):       1316 kB
Inactive(anon):    33496 kB
Active(file):          0 kB
Inactive(file):        4 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        995324 kB
SwapFree:         995324 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          1360 kB
Mapped:             2688 kB
Shmem:             33464 kB
KReclaimable:       8244 kB
Slab:              19772 kB
SReclaimable:       8244 kB
SUnreclaim:        11528 kB
KernelStack:        1056 kB
PageTables:          336 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1991304 kB
Committed_AS:      38916 kB
VmallocTotal: 1069547512 kB
VmallocUsed:        4856 kB
VmallocChunk:          0 kB
Percpu:              272 kB

Software
========
DUT # cat /etc/openwrt_release
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='22.03.0-rc6'
DISTRIB_REVISION='r19590-042d558536'
DISTRIB_TARGET='octeon/generic'
DISTRIB_ARCH='mips64_octeonplus'
DISTRIB_DESCRIPTION='OpenWrt 22.03.0-rc6 r19590-042d558536'
DISTRIB_TAINTS='no-all no-ipv6'

DUT # uname -a
Linux OpenWrt 6.0.0-rc3+ #0 SMP Sun Jul 31 15:12:47 2022 mips64 GNU/Linux

DUT # cat /proc/swaps
Filename    Type       Size    Used  Priority
/dev/zram0  partition  995324  0     100

DUT # memcached -V
memcached 1.6.9

DUT # cat /etc/config/memcached
config memcached
        option user 'memcached'
        option maxconn '1024'
        option listen '0.0.0.0'
        option port '11211'
        option memory '6400'

ATE $ memtier_benchmark -v
memtier_benchmark 1.3.0
Copyright (C) 2011-2022 Redis Ltd.
This is free software.  You may redistribute copies of it under the terms of
the GNU General Public License <http://www.gnu.org/licenses/gpl.html>.
There is NO WARRANTY, to the extent permitted by law.

Procedure
=========
ATE $ cat run_benchmark_matrix.sh
run_memtier_benchmark()
{
    # boot to kernel $3

    # populate dataset
    memtier_benchmark/memtier_benchmark -s $DUT_IP -p 11211 \
        -P memcache_binary -n allkeys -c 1 --ratio 1:0 --pipeline 8 \
        --key-minimum=1 --key-maximum=$2 --key-pattern=P:P \
        -d 1000

    # access dataset using Guassian pattern
    memtier_benchmark/memtier_benchmark -s $DUT_IP -p 11211 \
        -P memcache_binary --test-time $1 -c 1 --ratio 0:1 \
        --pipeline 8 --key-minimum=1 --key-maximum=$2 \
        --key-pattern=G:G --randomize --distinct-client-seed

    # collect results
}

run_duration_secs=1200
mem_utils_90_110=(1600000 2000000)
kernels=("baseline" "patched")

for mem_util in ${mem_utils_90_110[@]}; do
    for kernel in ${kernels[@]}; do
        run_memtier_benchmark $run_duration_secs $mem_util $kernel
    done
done

Results
=======
Baseline                                 90% RAM utilization
------------------------------------------------------------
Ops/sec   Avg. Lat.  p50 Lat.  p99 Lat.  p99.9 Lat.  KB/sec
------------------------------------------------------------
48550.71  0.65687    0.48700   2.84700   5.56700     1812.25
48600.55  0.65629    0.48700   2.86300   5.59900     1814.11
48562.37  0.65674    0.48700   2.84700   5.50300     1812.68
48556.66  0.65688    0.48700   2.84700   5.53500     1812.47
48619.50  0.65600    0.48700   2.87900   5.63100     1814.82
48579.74  0.65654    0.48700   2.84700   5.56700     1813.33
48593.25  0.65764    0.48700   2.86300   5.56700     1814.10
48535.52  0.65716    0.48700   2.86300   5.56700     1811.68
48587.24  0.65645    0.48700   2.83100   5.50300     1813.61
48541.92  0.65704    0.48700   2.81500   5.47100     1811.92

MGLRU                                    90% RAM utilization
------------------------------------------------------------
Ops/sec   Avg. Lat.  p50 Lat.  p99 Lat.  p99.9 Lat.  KB/sec
------------------------------------------------------------
48622.38  0.65594    0.48700   2.81500   5.47100     1814.92
48537.74  0.65715    0.48700   2.84700   5.53500     1811.76
48586.82  0.65646    0.48700   2.84700   5.50300     1813.59
48552.44  0.65695    0.48700   2.83100   5.43900     1812.31
48557.35  0.65680    0.49500   2.83100   5.53500     1812.49
48625.48  0.65593    0.48700   2.81500   5.43900     1815.04
48655.75  0.65557    0.48700   2.84700   5.53500     1816.17
48625.67  0.65595    0.48700   2.84700   5.53500     1815.04
48622.22  0.65600    0.48700   2.84700   5.47100     1814.91
48617.10  0.65610    0.48700   2.84700   5.56700     1814.73

Baseline                                110% RAM utilization
------------------------------------------------------------
Ops/sec   Avg. Lat.  p50 Lat.  p99 Lat.  p99.9 Lat.  KB/sec
------------------------------------------------------------
19813.79  1.61245    0.63100   17.79100  31.74300    744.91
20328.29  1.57158    0.62300   17.27900  31.10300    764.25
20104.12  1.58913    0.62300   17.40700  31.10300    755.82
20342.03  1.57053    0.61500   17.27900  30.84700    764.77
19688.05  1.62268    0.62300   17.91900  31.35900    740.18
19607.31  1.62943    0.63900   17.91900  31.23100    737.15
19250.96  1.65963    0.65500   17.91900  31.10300    723.75
20182.79  1.58290    0.63100   17.40700  30.84700    758.78
20181.88  1.58299    0.63100   17.40700  30.84700    758.75
20615.90  1.54963    0.62300   17.02300  30.84700    775.06

MGLRU                                   110% RAM utilization
------------------------------------------------------------
Ops/sec   Avg. Lat.  p50 Lat.  p99 Lat.  p99.9 Lat.  KB/sec
------------------------------------------------------------
22911.33  1.39405    0.61500   13.69500  28.79900    861.36
22339.08  1.42989    0.61500   14.07900  30.07900    839.85
23394.22  1.36521    0.59900   13.56700  29.05500    879.51
22521.48  1.41830    0.61500   13.88700  29.82300    846.70
22678.10  1.40818    0.61500   13.82300  29.69500    852.59
22344.50  1.42952    0.61500   14.07900  29.95100    840.05
23245.65  1.37406    0.60700   13.50300  28.92700    873.93
23140.17  1.38032    0.59900   13.69500  29.18300    869.96
23003.34  1.38856    0.61500   13.63100  29.05500    864.82
22937.52  1.39253    0.61500   13.69500  29.43900    862.35

Flame graphs
------------
Baseline: https://drive.google.com/file/d/1-Ac4HMPAyZIqxtvKerUTqNNAgBLhpX9R
MGLRU: https://drive.google.com/file/d/1-9x0W2yIYeiRvXWiYRzL6niTqW7zCVPX

References
==========
[1] https://openwrt.org/docs/platforms/start
[2] https://www.amazon.com/bestsellers/pc/300189
[3] https://openwrt.org/toh/ubiquiti/edgerouter




[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux