Hi Yonghong,
> -----Original Message-----
> From: Yonghong Song <yhs@xxxxxx>
> Sent: Friday, August 20, 2021 2:29 PM
> To: Sang, Oliver <oliver.sang@xxxxxxxxx>; Andrii Nakryiko <andrii@xxxxxxxxxx>
> Cc: Daniel Borkmann <daniel@xxxxxxxxxxxxx>; Peter Zijlstra
> <peterz@xxxxxxxxxxxxx>; LKML <linux-kernel@xxxxxxxxxxxxxxx>; Linux Memory
> Management List <linux-mm@xxxxxxxxx>; lkp@xxxxxxxxxxxx; lkp <lkp@xxxxxxxxx>
> Subject: [LKP] Re: [bpf] b89fbfbb85: netperf.Throughput_tps -21.3% regression
>
>
>
> On 8/19/21 10:50 PM, kernel test robot wrote:
> >
> >
> > Greeting,
> >
> > FYI, we noticed a -21.3% regression of netperf.Throughput_tps due to commit:
> >
> >
> > commit: b89fbfbb854c9afc3047e8273cc3a694650b802e ("bpf: Implement
> > minimal BPF perf link")
> > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git
> > master
>
> Could you do a test again to double check?
> The above patch is related to bpf perf_event area.
> It should not have any impact with netperf.
We first tested again and confirmed the regression still exists (-21.4%), the result is stable [1], more details are in [2].
652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc
---------------- ---------------------------
%stddev %change %stddev
\ | \
382134 -21.4% 300300 netperf.Throughput_total_tps
23958 -21.4% 18828 netperf.Throughput_tps
Then we tried another platform, Ice Lake, it has 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G, but we found the regression is still existing (-5.4%), the result is stable [3], more details are in [4].
652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc
---------------- ---------------------------
%stddev %change %stddev
\ | \
522472 -5.3% 494763 netperf.Throughput_total_tps
32654 -5.3% 30922 netperf.Throughput_tps
[1]
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json: "netperf.Throughput_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18860.396249999998,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18808.879375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18905.214375000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18750.746874999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18803.823125000006,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 19201.846,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18819.47,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18979.38,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18837.719375000004,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18809.4725,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18776.799375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18773.092500000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18743.978125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18794.814374999998,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18810.783124999998,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18821.65625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18680.149375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18747.203125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18819.728750000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18830.020625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 18656.501874999998
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- ],
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json: "netperf.Throughput_total_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301766.33999999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300942.07,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 302483.43000000005,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300011.94999999995,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300861.1700000001,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 288027.69,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301111.52,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 303670.08,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301403.51000000007,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300951.56,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300428.79,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300369.48000000004,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 299903.65,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300717.02999999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 300972.52999999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301146.5,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 298882.39,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 299955.25,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301115.66000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 301280.33,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 298504.02999999997
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- ],
[2]
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2ap3/TCP_CRR/netperf/0x5003006
commit:
652c1b17b8 ("bpf: Refactor perf_event_set_bpf_prog() to use struct bpf_prog input")
b89fbfbb85 ("bpf: Implement minimal BPF perf link")
652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc
---------------- ---------------------------
%stddev %change %stddev
\ | \
381418 -21.3% 300214 netperf.Throughput_total_tps
23906 -21.3% 18820 netperf.Throughput_tps
872.95 -5.2% 827.95 netperf.time.percent_of_cpu_this_job_got
2586 -4.3% 2474 netperf.time.system_time
2.25e+08 -22.8% 1.736e+08 netperf.time.voluntary_context_switches
1.144e+08 -21.3% 90064356 netperf.workload
2346775 -8.7% 2143411 vmstat.system.cs
872.95 -5.2% 827.95 time.percent_of_cpu_this_job_got
2586 -4.3% 2474 time.system_time
2.25e+08 -22.8% 1.736e+08 time.voluntary_context_switches
0.01 ± 7% +68.4% 0.01 ± 7% perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
3491773 ± 2% -22.6% 2702148 perf-sched.wait_and_delay.count.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file
4512682 ± 4% +21.9% 5501355 ± 2% perf-sched.wait_and_delay.count.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
0.01 ± 8% +86.5% 0.01 ± 8% perf-sched.wait_time.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
0.03 ± 8% +171.3% 0.08 ±231% perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.apparmor_file_alloc_security.security_file_alloc.__alloc_file
113308 -2.8% 110156 proc-vmstat.nr_slab_unreclaimable
4057280 ± 2% -8.6% 3709088 ± 2% proc-vmstat.numa_hit
3847678 ± 2% -8.9% 3506977 ± 2% proc-vmstat.numa_local
4055392 ± 2% -8.5% 3709694 ± 2% proc-vmstat.pgalloc_normal
1770294 ± 2% -23.5% 1355122 ± 2% proc-vmstat.pgfree
2456040 ± 16% -34.0% 1620559 ± 20% softirqs.CPU121.NET_RX
1737990 ± 24% -36.1% 1109923 ± 27% softirqs.CPU163.NET_RX
3.519e+08 -22.1% 2.742e+08 softirqs.NET_RX
26128869 ± 2% -15.7% 22015818 softirqs.RCU
237092 ± 2% +84.0% 436186 softirqs.TIMER
170292 -12.0% 149884 slabinfo.filp.active_objs
2663 -12.0% 2343 slabinfo.filp.active_slabs
170459 -12.0% 150036 slabinfo.filp.num_objs
2663 -12.0% 2343 slabinfo.filp.num_slabs
49886 ± 5% -11.6% 44079 ± 4% slabinfo.kmalloc-512.active_objs
50080 ± 5% -11.6% 44269 ± 4% slabinfo.kmalloc-512.num_objs
12.06 ± 4% +26.2% 15.22 ± 2% perf-stat.i.MPKI
1.004e+10 -18.0% 8.233e+09 perf-stat.i.branch-instructions
1.562e+08 -19.6% 1.256e+08 perf-stat.i.branch-misses
10.92 +7.2 18.09 perf-stat.i.cache-miss-rate%
64015758 ± 2% +75.1% 1.121e+08 perf-stat.i.cache-misses
5.888e+08 +5.2% 6.196e+08 perf-stat.i.cache-references
2379887 -8.7% 2173629 perf-stat.i.context-switches
1.57 ± 2% +17.2% 1.84 perf-stat.i.cpi
7.739e+10 -3.5% 7.467e+10 perf-stat.i.cpu-cycles
1272 ± 2% -43.8% 715.53 ± 3% perf-stat.i.cycles-between-cache-misses
903463 ± 18% -27.4% 655855 ± 5% perf-stat.i.dTLB-load-misses
1.518e+10 -18.3% 1.24e+10 perf-stat.i.dTLB-loads
228225 ± 31% -52.8% 107778 ± 23% perf-stat.i.dTLB-store-misses
8.879e+09 -18.5% 7.234e+09 perf-stat.i.dTLB-stores
89.92 -4.7 85.18 perf-stat.i.iTLB-load-miss-rate%
1.075e+08 -21.1% 84797404 perf-stat.i.iTLB-load-misses
11549994 ± 4% +23.9% 14312532 ± 2% perf-stat.i.iTLB-loads
5.079e+10 -18.2% 4.155e+10 perf-stat.i.instructions
0.65 -15.2% 0.55 perf-stat.i.ipc
0.40 -3.5% 0.39 perf-stat.i.metric.GHz
180.81 -17.8% 148.59 perf-stat.i.metric.M/sec
13211450 +59.2% 21027780 perf-stat.i.node-load-misses
1191177 ± 5% +68.5% 2006794 ± 9% perf-stat.i.node-loads
10841458 -22.2% 8430280 perf-stat.i.node-store-misses
11.59 +28.7% 14.91 perf-stat.overall.MPKI
1.56 -0.0 1.53 perf-stat.overall.branch-miss-rate%
10.87 +7.2 18.09 perf-stat.overall.cache-miss-rate%
1.52 +18.0% 1.80 perf-stat.overall.cpi
1209 ± 2% -44.9% 666.15 perf-stat.overall.cycles-between-cache-misses
90.30 -4.7 85.56 perf-stat.overall.iTLB-load-miss-rate%
472.39 +3.8% 490.15 perf-stat.overall.instructions-per-iTLB-miss
0.66 -15.2% 0.56 perf-stat.overall.ipc
1e+10 -18.0% 8.205e+09 perf-stat.ps.branch-instructions
1.557e+08 -19.6% 1.251e+08 perf-stat.ps.branch-misses
63792201 ± 2% +75.1% 1.117e+08 perf-stat.ps.cache-misses
5.867e+08 +5.2% 6.174e+08 perf-stat.ps.cache-references
2371522 -8.7% 2165977 perf-stat.ps.context-switches
7.712e+10 -3.5% 7.441e+10 perf-stat.ps.cpu-cycles
900712 ± 18% -27.4% 653896 ± 5% perf-stat.ps.dTLB-load-misses
1.513e+10 -18.3% 1.236e+10 perf-stat.ps.dTLB-loads
227485 ± 31% -52.8% 107457 ± 23% perf-stat.ps.dTLB-store-misses
8.848e+09 -18.5% 7.208e+09 perf-stat.ps.dTLB-stores
1.072e+08 -21.1% 84499344 perf-stat.ps.iTLB-load-misses
11509852 ± 4% +23.9% 14262519 ± 2% perf-stat.ps.iTLB-loads
5.062e+10 -18.2% 4.14e+10 perf-stat.ps.instructions
13165220 +59.2% 20953789 perf-stat.ps.node-load-misses
1187095 ± 5% +68.5% 1999816 ± 9% perf-stat.ps.node-loads
10803432 -22.2% 8400688 perf-stat.ps.node-store-misses
1.543e+13 -18.2% 1.262e+13 perf-stat.total.instructions
5.24 ± 7% -1.2 4.05 ± 7% perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.22 ± 7% -1.2 4.03 ± 7% perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
4.13 ± 21% -1.0 3.17 ± 6% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
2.75 ± 6% -0.6 2.15 ± 8% perf-profile.calltrace.cycles-pp.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.71 ± 9% -0.5 1.25 ± 9% perf-profile.calltrace.cycles-pp.ret_from_fork
1.71 ± 9% -0.5 1.25 ± 9% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
1.67 ± 9% -0.5 1.22 ± 9% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
1.66 ± 9% -0.5 1.21 ± 9% perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
1.67 ± 9% -0.5 1.22 ± 9% perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
1.68 ± 9% -0.5 1.23 ± 9% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
1.65 ± 9% -0.5 1.20 ± 9% perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
1.58 ± 9% -0.4 1.14 ± 9% perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd
1.40 ± 9% -0.4 1.02 ± 9% perf-profile.calltrace.cycles-pp.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start
1.36 ± 9% -0.4 0.99 ± 9% perf-profile.calltrace.cycles-pp.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core
1.39 ± 8% -0.3 1.09 ± 9% perf-profile.calltrace.cycles-pp.inet_accept.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64
1.07 ± 10% -0.3 0.78 ± 9% perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch
1.10 ± 6% -0.3 0.83 ± 7% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
1.03 ± 6% -0.3 0.78 ± 7% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu
1.04 ± 8% -0.2 0.79 ± 8% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
0.96 ± 6% -0.2 0.72 ± 7% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv
0.95 ± 6% -0.2 0.71 ± 7% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process
0.97 ± 8% -0.2 0.74 ± 8% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
0.92 ± 6% -0.2 0.68 ± 7% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
0.90 ± 8% -0.2 0.68 ± 8% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
0.88 ± 8% -0.2 0.67 ± 8% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
0.75 ± 7% -0.2 0.57 ± 23% perf-profile.calltrace.cycles-pp.alloc_file_pseudo.sock_alloc_file.__sys_accept4_file.__sys_accept4.__x64_sys_accept
0.67 ± 7% -0.2 0.49 ± 41% perf-profile.calltrace.cycles-pp.inet_csk_get_port.__inet_bind.inet_bind.__sys_bind.__x64_sys_bind
0.55 ± 22% +0.2 0.72 ± 7% perf-profile.calltrace.cycles-pp.tcp_v4_send_synack.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv
0.49 ± 40% +0.3 0.75 ± 8% perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
1.65 ± 6% +0.3 1.97 ± 7% perf-profile.calltrace.cycles-pp.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
1.87 ± 6% +0.4 2.24 ± 7% perf-profile.calltrace.cycles-pp.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
1.56 ± 6% +0.5 2.02 ± 6% perf-profile.calltrace.cycles-pp.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
0.00 +0.6 0.57 ± 6% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
3.08 ± 6% +0.7 3.74 ± 7% perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect
3.05 ± 6% +0.7 3.73 ± 7% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect
3.42 ± 6% +0.8 4.18 ± 7% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect
3.64 ± 6% +0.9 4.50 ± 6% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
4.42 ± 6% +1.0 5.44 ± 7% perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
5.36 ± 6% +1.3 6.68 ± 7% perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
11.59 ± 5% +2.1 13.65 ± 6% perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
11.54 ± 5% +2.1 13.61 ± 6% perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
5.94 ± 6% -1.3 4.60 ± 6% perf-profile.children.cycles-pp.__x64_sys_recvfrom
5.90 ± 6% -1.3 4.58 ± 6% perf-profile.children.cycles-pp.__sys_recvfrom
5.32 ± 6% -1.2 4.08 ± 6% perf-profile.children.cycles-pp.inet_recvmsg
5.30 ± 6% -1.2 4.07 ± 6% perf-profile.children.cycles-pp.tcp_recvmsg
4.17 ± 22% -1.0 3.19 ± 6% perf-profile.children.cycles-pp.poll_idle
4.48 ± 6% -0.8 3.71 ± 6% perf-profile.children.cycles-pp.tcp_recvmsg_locked
3.06 ± 8% -0.7 2.39 ± 7% perf-profile.children.cycles-pp.kmem_cache_free
1.66 ± 6% -0.6 1.06 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_bh
2.80 ± 6% -0.6 2.21 ± 7% perf-profile.children.cycles-pp.__x64_sys_accept
2.78 ± 6% -0.6 2.20 ± 7% perf-profile.children.cycles-pp.__sys_accept4
2.75 ± 6% -0.6 2.17 ± 7% perf-profile.children.cycles-pp.__sys_accept4_file
2.37 ± 9% -0.5 1.85 ± 8% perf-profile.children.cycles-pp.rcu_core
2.33 ± 9% -0.5 1.81 ± 8% perf-profile.children.cycles-pp.rcu_do_batch
2.35 ± 8% -0.5 1.84 ± 8% perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
2.31 ± 8% -0.5 1.81 ± 8% perf-profile.children.cycles-pp.page_counter_uncharge
1.12 ± 7% -0.5 0.63 ± 7% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
1.37 ± 6% -0.5 0.89 ± 6% perf-profile.children.cycles-pp.lock_sock_nested
1.71 ± 9% -0.5 1.25 ± 9% perf-profile.children.cycles-pp.ret_from_fork
1.71 ± 9% -0.5 1.25 ± 9% perf-profile.children.cycles-pp.kthread
1.67 ± 9% -0.5 1.22 ± 9% perf-profile.children.cycles-pp.run_ksoftirqd
1.68 ± 9% -0.5 1.23 ± 9% perf-profile.children.cycles-pp.smpboot_thread_fn
2.84 ± 6% -0.4 2.43 ± 6% perf-profile.children.cycles-pp.sk_wait_data
1.78 ± 6% -0.4 1.36 ± 6% perf-profile.children.cycles-pp.__dev_queue_xmit
1.83 ± 9% -0.4 1.43 ± 8% perf-profile.children.cycles-pp.page_counter_cancel
2.14 ± 6% -0.4 1.75 ± 6% perf-profile.children.cycles-pp._raw_spin_lock
1.44 ± 7% -0.3 1.13 ± 6% perf-profile.children.cycles-pp.kmem_cache_alloc
1.40 ± 7% -0.3 1.10 ± 8% perf-profile.children.cycles-pp.inet_accept
1.56 ± 6% -0.3 1.26 ± 6% perf-profile.children.cycles-pp.alloc_file_pseudo
1.56 ± 6% -0.3 1.27 ± 6% perf-profile.children.cycles-pp.sock_alloc_file
1.03 ± 6% -0.3 0.73 ± 7% perf-profile.children.cycles-pp.percpu_counter_add_batch
1.20 ± 6% -0.3 0.92 ± 6% perf-profile.children.cycles-pp.__alloc_skb
0.62 ± 6% -0.3 0.37 ± 9% perf-profile.children.cycles-pp.__tcp_get_metrics
0.97 ± 5% -0.2 0.73 ± 6% perf-profile.children.cycles-pp.autoremove_wake_function
1.15 ± 6% -0.2 0.92 ± 8% perf-profile.children.cycles-pp.inet_csk_accept
0.59 ± 5% -0.2 0.36 ± 7% perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb
0.80 ± 6% -0.2 0.58 ± 6% perf-profile.children.cycles-pp.sk_forced_mem_schedule
1.12 ± 6% -0.2 0.90 ± 6% perf-profile.children.cycles-pp.alloc_file
1.10 ± 6% -0.2 0.89 ± 6% perf-profile.children.cycles-pp.alloc_empty_file
1.03 ± 6% -0.2 0.82 ± 7% perf-profile.children.cycles-pp.sock_def_wakeup
1.08 ± 6% -0.2 0.88 ± 6% perf-profile.children.cycles-pp.__alloc_file
1.21 ± 6% -0.2 1.01 ± 6% perf-profile.children.cycles-pp.inet_csk_destroy_sock
0.78 ± 6% -0.2 0.58 ± 6% perf-profile.children.cycles-pp.sk_stream_alloc_skb
0.89 ± 6% -0.2 0.70 ± 7% perf-profile.children.cycles-pp.dev_hard_start_xmit
0.36 ± 6% -0.2 0.18 ± 6% perf-profile.children.cycles-pp.ip_finish_output
0.85 ± 6% -0.2 0.68 ± 7% perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
0.78 ± 6% -0.2 0.61 ± 7% perf-profile.children.cycles-pp.loopback_xmit
1.00 ± 6% -0.2 0.83 ± 6% perf-profile.children.cycles-pp.__kfree_skb
1.15 ± 6% -0.2 0.98 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.68 ± 6% -0.2 0.51 ± 7% perf-profile.children.cycles-pp.tcp_get_metrics
0.79 ± 6% -0.2 0.63 ± 6% perf-profile.children.cycles-pp.tcp_send_fin
0.92 ± 6% -0.2 0.76 ± 6% perf-profile.children.cycles-pp.inet_csk_clone_lock
0.67 ± 6% -0.2 0.51 ± 8% perf-profile.children.cycles-pp.sk_reset_timer
0.68 ± 6% -0.2 0.52 ± 7% perf-profile.children.cycles-pp.sock_alloc
0.67 ± 6% -0.2 0.52 ± 7% perf-profile.children.cycles-pp.new_inode_pseudo
0.89 ± 5% -0.1 0.74 ± 6% perf-profile.children.cycles-pp.sk_clone_lock
0.77 ± 7% -0.1 0.63 ± 7% perf-profile.children.cycles-pp.tcp_done
0.76 ± 6% -0.1 0.62 ± 6% perf-profile.children.cycles-pp.security_file_alloc
0.58 ± 7% -0.1 0.45 ± 7% perf-profile.children.cycles-pp.alloc_inode
0.59 ± 6% -0.1 0.46 ± 7% perf-profile.children.cycles-pp.mod_timer
0.67 ± 6% -0.1 0.54 ± 6% perf-profile.children.cycles-pp.__dentry_kill
0.69 ± 6% -0.1 0.57 ± 6% perf-profile.children.cycles-pp.apparmor_file_alloc_security
0.45 ± 6% -0.1 0.33 ± 7% perf-profile.children.cycles-pp.lock_timer_base
0.58 ± 7% -0.1 0.47 ± 6% perf-profile.children.cycles-pp.apparmor_file_free_security
0.59 ± 7% -0.1 0.47 ± 6% perf-profile.children.cycles-pp.security_file_free
0.68 ± 7% -0.1 0.56 ± 7% perf-profile.children.cycles-pp.inet_csk_get_port
0.44 ± 5% -0.1 0.33 ± 7% perf-profile.children.cycles-pp.sk_filter_trim_cap
0.47 ± 7% -0.1 0.36 ± 7% perf-profile.children.cycles-pp.sock_alloc_inode
0.54 ± 8% -0.1 0.44 ± 10% perf-profile.children.cycles-pp.propagate_protected_usage
0.68 ± 6% -0.1 0.58 ± 6% perf-profile.children.cycles-pp.skb_release_data
0.43 ± 6% -0.1 0.33 ± 7% perf-profile.children.cycles-pp.kmalloc_reserve
0.45 ± 10% -0.1 0.36 ± 10% perf-profile.children.cycles-pp.inet_csk_bind_conflict
0.38 ± 6% -0.1 0.30 ± 7% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
0.40 ± 6% -0.1 0.32 ± 7% perf-profile.children.cycles-pp.raw_local_deliver
0.46 ± 6% -0.1 0.38 ± 7% perf-profile.children.cycles-pp.read_tsc
0.40 ± 6% -0.1 0.32 ± 5% perf-profile.children.cycles-pp.skb_release_all
0.44 ± 6% -0.1 0.36 ± 7% perf-profile.children.cycles-pp.sock_recvmsg
0.37 ± 7% -0.1 0.29 ± 8% perf-profile.children.cycles-pp.__tcp_send_ack
0.46 ± 7% -0.1 0.38 ± 7% perf-profile.children.cycles-pp.__skb_datagram_iter
0.48 ± 6% -0.1 0.40 ± 7% perf-profile.children.cycles-pp.___might_sleep
0.43 ± 7% -0.1 0.36 ± 8% perf-profile.children.cycles-pp.security_socket_recvmsg
0.38 ± 7% -0.1 0.30 ± 5% perf-profile.children.cycles-pp.skb_release_head_state
0.37 ± 7% -0.1 0.29 ± 7% perf-profile.children.cycles-pp.apparmor_sk_clone_security
0.37 ± 7% -0.1 0.30 ± 6% perf-profile.children.cycles-pp.security_sk_clone
0.29 ± 6% -0.1 0.22 ± 8% perf-profile.children.cycles-pp.__ksize
0.31 ± 6% -0.1 0.24 ± 9% perf-profile.children.cycles-pp.netif_rx
0.37 ± 7% -0.1 0.29 ± 7% perf-profile.children.cycles-pp.select_task_rq_fair
0.32 ± 6% -0.1 0.25 ± 8% perf-profile.children.cycles-pp.tcp_write_queue_purge
0.30 ± 7% -0.1 0.23 ± 9% perf-profile.children.cycles-pp.netif_rx_internal
0.24 ± 5% -0.1 0.17 ± 7% perf-profile.children.cycles-pp.ip_local_out
0.30 ± 6% -0.1 0.23 ± 8% perf-profile.children.cycles-pp.irqtime_account_irq
0.40 ± 6% -0.1 0.33 ± 7% perf-profile.children.cycles-pp.apparmor_socket_post_create
0.40 ± 6% -0.1 0.33 ± 6% perf-profile.children.cycles-pp.security_socket_post_create
0.25 ± 8% -0.1 0.18 ± 6% perf-profile.children.cycles-pp.__skb_clone
0.32 ± 6% -0.1 0.25 ± 7% perf-profile.children.cycles-pp.tcp_mstamp_refresh
0.22 ± 4% -0.1 0.16 ± 6% perf-profile.children.cycles-pp.__ip_local_out
0.26 ± 7% -0.1 0.20 ± 9% perf-profile.children.cycles-pp.validate_xmit_skb
0.27 ± 7% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.sk_stop_timer
0.26 ± 6% -0.1 0.20 ± 7% perf-profile.children.cycles-pp.enqueue_to_backlog
0.21 ± 8% -0.1 0.15 ± 8% perf-profile.children.cycles-pp.kmem_cache_alloc_node
0.14 ± 7% -0.1 0.08 ± 10% perf-profile.children.cycles-pp.migrate_enable
0.20 ± 7% -0.1 0.15 ± 8% perf-profile.children.cycles-pp.__check_object_size
0.18 ± 6% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.ip_send_check
0.24 ± 8% -0.1 0.19 ± 7% perf-profile.children.cycles-pp.__slab_free
0.24 ± 6% -0.1 0.19 ± 6% perf-profile.children.cycles-pp.inet_csk_clear_xmit_timers
0.26 ± 8% -0.0 0.22 ± 6% perf-profile.children.cycles-pp.d_alloc_pseudo
0.21 ± 6% -0.0 0.17 ± 9% perf-profile.children.cycles-pp.copy_user_generic_unrolled
0.12 ± 8% -0.0 0.07 ± 8% perf-profile.children.cycles-pp.sock_put
0.28 ± 6% -0.0 0.24 ± 7% perf-profile.children.cycles-pp.__switch_to
0.18 ± 7% -0.0 0.14 ± 7% perf-profile.children.cycles-pp.del_timer
0.21 ± 5% -0.0 0.17 ± 9% perf-profile.children.cycles-pp.kfree
0.15 ± 9% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.get_obj_cgroup_from_current
0.12 ± 10% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.tcp_queue_rcv
0.08 ± 9% -0.0 0.05 ± 41% perf-profile.children.cycles-pp.netdev_core_pick_tx
0.13 ± 7% -0.0 0.10 ± 11% perf-profile.children.cycles-pp.netif_skb_features
0.08 ± 9% -0.0 0.05 ± 34% perf-profile.children.cycles-pp.nf_hook_slow
0.22 ± 6% -0.0 0.18 ± 9% perf-profile.children.cycles-pp.__might_sleep
0.10 ± 9% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.__destroy_inode
0.14 ± 7% -0.0 0.11 ± 8% perf-profile.children.cycles-pp.tcp_rcv_space_adjust
0.10 ± 8% -0.0 0.07 ± 9% perf-profile.children.cycles-pp.destroy_inode
0.09 ± 8% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.inet_csk_complete_hashdance
0.14 ± 7% -0.0 0.11 ± 8% perf-profile.children.cycles-pp.copyout
0.22 ± 7% -0.0 0.19 ± 8% perf-profile.children.cycles-pp.__update_load_avg_se
0.18 ± 6% -0.0 0.15 ± 8% perf-profile.children.cycles-pp.__fget_light
0.14 ± 8% -0.0 0.11 ± 9% perf-profile.children.cycles-pp.simple_copy_to_iter
0.15 ± 8% -0.0 0.13 ± 7% perf-profile.children.cycles-pp.__cond_resched
0.09 ± 9% -0.0 0.06 ± 10% perf-profile.children.cycles-pp.inet_reqsk_alloc
0.13 ± 6% -0.0 0.11 ± 8% perf-profile.children.cycles-pp.d_instantiate
0.09 ± 12% -0.0 0.06 ± 10% perf-profile.children.cycles-pp.security_sock_rcv_skb
0.08 ± 11% -0.0 0.05 ± 24% perf-profile.children.cycles-pp.migrate_disable
0.11 ± 10% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.tcp_validate_incoming
0.14 ± 8% -0.0 0.11 ± 8% perf-profile.children.cycles-pp.tcp_wfree
0.12 ± 7% -0.0 0.09 ± 7% perf-profile.children.cycles-pp.tcp_add_backlog
0.07 ± 10% -0.0 0.05 ± 33% perf-profile.children.cycles-pp.fsnotify_destroy_marks
0.09 ± 8% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.sock_setsockopt
0.10 ± 7% -0.0 0.08 ± 8% perf-profile.children.cycles-pp.tcp_init_xmit_timers
0.09 ± 9% +0.0 0.13 ± 9% perf-profile.children.cycles-pp.inet_twsk_alloc
0.06 ± 8% +0.0 0.09 ± 7% perf-profile.children.cycles-pp.tcp_mtup_init
0.23 ± 6% +0.0 0.27 ± 5% perf-profile.children.cycles-pp.tcp_make_synack
0.06 ± 9% +0.1 0.12 ± 8% perf-profile.children.cycles-pp.tcp_synack_rtt_meas
0.06 ± 10% +0.1 0.11 ± 8% perf-profile.children.cycles-pp.tcp_ca_openreq_child
0.00 +0.1 0.06 ± 10% perf-profile.children.cycles-pp.tcp_rack_update_reo_wnd
0.07 ± 9% +0.1 0.14 ± 9% perf-profile.children.cycles-pp.__inet_lookup_listener
0.12 ± 7% +0.1 0.20 ± 7% perf-profile.children.cycles-pp.__ip_finish_output
0.06 ± 13% +0.1 0.14 ± 8% perf-profile.children.cycles-pp.__ip_dev_find
0.01 ±212% +0.1 0.09 ± 8% perf-profile.children.cycles-pp.inet_lhash2_lookup
0.00 +0.1 0.08 ± 8% perf-profile.children.cycles-pp.sock_prot_inuse_add
0.00 +0.1 0.09 ± 10% perf-profile.children.cycles-pp.tcp_push
0.07 ± 10% +0.1 0.16 ± 8% perf-profile.children.cycles-pp.tcp_select_initial_window
0.07 ± 10% +0.1 0.16 ± 7% perf-profile.children.cycles-pp.tcp_sync_mss
0.07 ± 9% +0.1 0.16 ± 8% perf-profile.children.cycles-pp.secure_tcp_ts_off
0.36 ± 6% +0.1 0.47 ± 6% perf-profile.children.cycles-pp.tcp_finish_connect
0.02 ±120% +0.1 0.13 ± 10% perf-profile.children.cycles-pp.inet_lookup_ifaddr_rcu
0.08 ± 11% +0.1 0.19 ± 8% perf-profile.children.cycles-pp.skb_entail
0.17 ± 5% +0.1 0.29 ± 7% perf-profile.children.cycles-pp.tcp_connect_init
0.10 ± 9% +0.1 0.21 ± 7% perf-profile.children.cycles-pp.tcp_openreq_init_rwin
0.00 +0.1 0.13 ± 9% perf-profile.children.cycles-pp.tcp_stream_memory_free
0.14 ± 6% +0.1 0.26 ± 7% perf-profile.children.cycles-pp.tcp_parse_options
0.13 ± 8% +0.1 0.26 ± 8% perf-profile.children.cycles-pp.tcp_update_pacing_rate
0.58 ± 5% +0.2 0.73 ± 5% perf-profile.children.cycles-pp.tcp_v4_send_synack
0.08 ± 7% +0.2 0.24 ± 6% perf-profile.children.cycles-pp.tcp_assign_congestion_control
0.34 ± 7% +0.2 0.50 ± 6% perf-profile.children.cycles-pp.tcp_v4_init_sock
0.33 ± 6% +0.2 0.49 ± 6% perf-profile.children.cycles-pp.tcp_init_sock
0.23 ± 6% +0.2 0.40 ± 6% perf-profile.children.cycles-pp.fib_table_lookup
0.00 +0.2 0.17 ± 8% perf-profile.children.cycles-pp.__sk_free
0.22 ± 7% +0.2 0.41 ± 6% perf-profile.children.cycles-pp.tcp_schedule_loss_probe
0.14 ± 7% +0.2 0.33 ± 7% perf-profile.children.cycles-pp.ip_rcv_finish
0.63 ± 5% +0.2 0.83 ± 7% perf-profile.children.cycles-pp.tcp_init_transfer
0.10 ± 8% +0.2 0.31 ± 7% perf-profile.children.cycles-pp.ip_rcv_finish_core
0.04 ± 40% +0.2 0.25 ± 6% perf-profile.children.cycles-pp.tcp_newly_delivered
0.07 ± 10% +0.2 0.28 ± 7% perf-profile.children.cycles-pp.fib_lookup_good_nhc
0.19 ± 7% +0.2 0.41 ± 6% perf-profile.children.cycles-pp.inet_csk_route_child_sock
0.18 ± 6% +0.2 0.40 ± 6% perf-profile.children.cycles-pp.inet_csk_route_req
0.11 ± 7% +0.2 0.34 ± 8% perf-profile.children.cycles-pp.inet_sk_rebuild_header
0.11 ± 8% +0.2 0.35 ± 7% perf-profile.children.cycles-pp.inet_sock_destruct
0.10 ± 9% +0.3 0.37 ± 8% perf-profile.children.cycles-pp.__tcp_select_window
1.69 ± 5% +0.3 2.00 ± 6% perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
0.20 ± 6% +0.3 0.52 ± 8% perf-profile.children.cycles-pp.tcp_ack_update_rtt
0.15 ± 8% +0.3 0.49 ± 7% perf-profile.children.cycles-pp.inet_ehashfn
1.91 ± 5% +0.4 2.27 ± 6% perf-profile.children.cycles-pp.tcp_check_req
0.17 ± 6% +0.4 0.54 ± 7% perf-profile.children.cycles-pp.ip_rcv_core
1.59 ± 5% +0.5 2.05 ± 6% perf-profile.children.cycles-pp.tcp_conn_request
0.43 ± 5% +0.6 0.99 ± 5% perf-profile.children.cycles-pp.ip_route_output_flow
2.02 ± 6% +0.6 2.67 ± 6% perf-profile.children.cycles-pp.tcp_ack
0.26 ± 7% +0.8 1.02 ± 7% perf-profile.children.cycles-pp.__sk_dst_check
0.60 ± 5% +0.8 1.37 ± 6% perf-profile.children.cycles-pp.ip_route_output_key_hash
0.55 ± 5% +0.8 1.33 ± 6% perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
5.73 ± 5% +0.9 6.60 ± 6% perf-profile.children.cycles-pp.tcp_rcv_synsent_state_process
0.24 ± 6% +0.9 1.19 ± 7% perf-profile.children.cycles-pp.ipv4_dst_check
4.49 ± 5% +1.0 5.47 ± 6% perf-profile.children.cycles-pp.tcp_connect
5.45 ± 5% +1.3 6.72 ± 6% perf-profile.children.cycles-pp.tcp_v4_connect
11.81 ± 5% +2.0 13.85 ± 6% perf-profile.children.cycles-pp.__x64_sys_connect
11.79 ± 5% +2.1 13.84 ± 6% perf-profile.children.cycles-pp.__sys_connect
11.59 ± 5% +2.1 13.65 ± 6% perf-profile.children.cycles-pp.inet_stream_connect
11.54 ± 5% +2.1 13.61 ± 6% perf-profile.children.cycles-pp.__inet_stream_connect
4.00 ± 20% -0.9 3.08 ± 6% perf-profile.self.cycles-pp.poll_idle
1.12 ± 7% -0.5 0.63 ± 7% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
1.82 ± 9% -0.4 1.42 ± 8% perf-profile.self.cycles-pp.page_counter_cancel
1.01 ± 6% -0.3 0.72 ± 6% perf-profile.self.cycles-pp.percpu_counter_add_batch
1.65 ± 6% -0.3 1.36 ± 5% perf-profile.self.cycles-pp._raw_spin_lock
0.62 ± 6% -0.3 0.36 ± 8% perf-profile.self.cycles-pp.__tcp_get_metrics
1.09 ± 6% -0.3 0.84 ± 6% perf-profile.self.cycles-pp.__inet_lookup_established
0.47 ± 9% -0.2 0.25 ± 7% perf-profile.self.cycles-pp.__sk_destruct
0.79 ± 6% -0.2 0.57 ± 6% perf-profile.self.cycles-pp.sk_forced_mem_schedule
0.99 ± 6% -0.2 0.78 ± 7% perf-profile.self.cycles-pp._raw_spin_lock_bh
0.84 ± 6% -0.2 0.67 ± 7% perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
1.14 ± 6% -0.2 0.97 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.36 ± 6% -0.1 0.22 ± 8% perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb
0.67 ± 6% -0.1 0.55 ± 6% perf-profile.self.cycles-pp.apparmor_file_alloc_security
0.40 ± 7% -0.1 0.28 ± 7% perf-profile.self.cycles-pp.tcp_recvmsg_locked
0.58 ± 7% -0.1 0.46 ± 5% perf-profile.self.cycles-pp.apparmor_file_free_security
0.50 ± 6% -0.1 0.38 ± 7% perf-profile.self.cycles-pp.__dev_queue_xmit
0.54 ± 9% -0.1 0.43 ± 10% perf-profile.self.cycles-pp.propagate_protected_usage
0.40 ± 6% -0.1 0.30 ± 7% perf-profile.self.cycles-pp.kmem_cache_free
0.33 ± 6% -0.1 0.24 ± 7% perf-profile.self.cycles-pp.tcp_clean_rtx_queue
0.45 ± 10% -0.1 0.35 ± 10% perf-profile.self.cycles-pp.inet_csk_bind_conflict
0.28 ± 8% -0.1 0.20 ± 9% perf-profile.self.cycles-pp.ip_finish_output2
0.39 ± 6% -0.1 0.31 ± 7% perf-profile.self.cycles-pp.raw_local_deliver
0.45 ± 6% -0.1 0.37 ± 6% perf-profile.self.cycles-pp.read_tsc
0.47 ± 6% -0.1 0.39 ± 7% perf-profile.self.cycles-pp.___might_sleep
0.37 ± 7% -0.1 0.29 ± 7% perf-profile.self.cycles-pp.apparmor_sk_clone_security
0.26 ± 7% -0.1 0.19 ± 8% perf-profile.self.cycles-pp.tcp_sendmsg_locked
0.28 ± 6% -0.1 0.21 ± 7% perf-profile.self.cycles-pp.__ksize
0.39 ± 6% -0.1 0.32 ± 7% perf-profile.self.cycles-pp.do_idle
0.39 ± 6% -0.1 0.33 ± 7% perf-profile.self.cycles-pp.apparmor_socket_post_create
0.20 ± 8% -0.1 0.14 ± 7% perf-profile.self.cycles-pp.__skb_clone
0.23 ± 8% -0.1 0.17 ± 9% perf-profile.self.cycles-pp.exit_to_user_mode_prepare
0.13 ± 7% -0.1 0.07 ± 11% perf-profile.self.cycles-pp.migrate_enable
0.17 ± 6% -0.1 0.12 ± 6% perf-profile.self.cycles-pp.ip_send_check
0.24 ± 8% -0.1 0.19 ± 8% perf-profile.self.cycles-pp.__slab_free
0.24 ± 7% -0.1 0.19 ± 7% perf-profile.self.cycles-pp.__kmalloc_node_track_caller
0.24 ± 7% -0.1 0.19 ± 8% perf-profile.self.cycles-pp.__alloc_skb
0.17 ± 7% -0.0 0.13 ± 7% perf-profile.self.cycles-pp.net_rx_action
0.18 ± 7% -0.0 0.13 ± 10% perf-profile.self.cycles-pp.process_backlog
0.16 ± 7% -0.0 0.12 ± 6% perf-profile.self.cycles-pp.fib_table_lookup
0.12 ± 9% -0.0 0.07 ± 8% perf-profile.self.cycles-pp.sock_put
0.27 ± 5% -0.0 0.23 ± 8% perf-profile.self.cycles-pp.__switch_to
0.14 ± 11% -0.0 0.10 ± 11% perf-profile.self.cycles-pp.do_syscall_64
0.08 ± 13% -0.0 0.04 ± 49% perf-profile.self.cycles-pp.select_task_rq_fair
0.15 ± 9% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.kmem_cache_alloc_node
0.21 ± 5% -0.0 0.17 ± 9% perf-profile.self.cycles-pp.copy_user_generic_unrolled
0.21 ± 7% -0.0 0.17 ± 9% perf-profile.self.cycles-pp.__local_bh_enable_ip
0.20 ± 5% -0.0 0.17 ± 9% perf-profile.self.cycles-pp.kfree
0.08 ± 7% -0.0 0.04 ± 64% perf-profile.self.cycles-pp.netdev_core_pick_tx
0.15 ± 6% -0.0 0.12 ± 9% perf-profile.self.cycles-pp.__tcp_close
0.11 ± 9% -0.0 0.08 ± 10% perf-profile.self.cycles-pp.tcp_queue_rcv
0.07 ± 9% -0.0 0.04 ± 56% perf-profile.self.cycles-pp.migrate_disable
0.07 ± 8% -0.0 0.04 ± 64% perf-profile.self.cycles-pp.inet_reqsk_alloc
0.09 ± 9% -0.0 0.05 ± 10% perf-profile.self.cycles-pp.tcp_data_queue
0.18 ± 8% -0.0 0.15 ± 6% perf-profile.self.cycles-pp.__entry_text_start
0.18 ± 6% -0.0 0.14 ± 8% perf-profile.self.cycles-pp.__fget_light
0.11 ± 9% -0.0 0.08 ± 7% perf-profile.self.cycles-pp.get_obj_cgroup_from_current
0.14 ± 8% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.__check_object_size
0.14 ± 8% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.enqueue_to_backlog
0.14 ± 9% -0.0 0.11 ± 10% perf-profile.self.cycles-pp.sk_reset_timer
0.12 ± 9% -0.0 0.09 ± 8% perf-profile.self.cycles-pp.validate_xmit_skb
0.14 ± 9% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.sk_setup_caps
0.09 ± 7% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.irqtime_account_irq
0.11 ± 10% -0.0 0.08 ± 10% perf-profile.self.cycles-pp.tcp_validate_incoming
0.14 ± 8% -0.0 0.11 ± 8% perf-profile.self.cycles-pp.tcp_wfree
0.07 ± 10% -0.0 0.05 ± 23% perf-profile.self.cycles-pp.tcp_rcv_synsent_state_process
0.08 ± 9% -0.0 0.06 ± 10% perf-profile.self.cycles-pp.sk_filter_trim_cap
0.08 ± 6% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.__netif_receive_skb_one_core
0.07 ± 8% -0.0 0.05 ± 23% perf-profile.self.cycles-pp.tcp_options_write
0.08 ± 9% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.netif_skb_features
0.08 ± 9% -0.0 0.06 ± 9% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.06 ± 10% +0.0 0.08 ± 8% perf-profile.self.cycles-pp.tcp_init_transfer
0.09 ± 9% +0.0 0.12 ± 9% perf-profile.self.cycles-pp.inet_twsk_alloc
0.07 ± 8% +0.0 0.10 ± 7% perf-profile.self.cycles-pp.inet_csk_get_port
0.06 ± 8% +0.0 0.09 ± 8% perf-profile.self.cycles-pp.tcp_mtup_init
0.11 ± 7% +0.0 0.16 ± 7% perf-profile.self.cycles-pp.sk_clone_lock
0.00 +0.1 0.06 ± 10% perf-profile.self.cycles-pp.tcp_rack_update_reo_wnd
0.00 +0.1 0.06 ± 12% perf-profile.self.cycles-pp.inet_twsk_kill
0.07 ± 7% +0.1 0.13 ± 8% perf-profile.self.cycles-pp.tcp_connect
0.07 ± 7% +0.1 0.14 ± 7% perf-profile.self.cycles-pp.__inet_bind
0.12 ± 7% +0.1 0.19 ± 7% perf-profile.self.cycles-pp.__ip_finish_output
0.06 ± 11% +0.1 0.14 ± 5% perf-profile.self.cycles-pp.tcp_make_synack
0.06 ± 6% +0.1 0.14 ± 8% perf-profile.self.cycles-pp.tcp_v4_syn_recv_sock
0.01 ±251% +0.1 0.09 ± 8% perf-profile.self.cycles-pp.inet_lhash2_lookup
0.00 +0.1 0.08 ± 8% perf-profile.self.cycles-pp.sock_prot_inuse_add
0.00 +0.1 0.09 ± 9% perf-profile.self.cycles-pp.tcp_push
0.07 ± 10% +0.1 0.16 ± 8% perf-profile.self.cycles-pp.tcp_select_initial_window
0.06 ± 10% +0.1 0.15 ± 8% perf-profile.self.cycles-pp.tcp_sync_mss
0.00 +0.1 0.09 ± 11% perf-profile.self.cycles-pp.tcp_v4_send_synack
0.05 ± 22% +0.1 0.14 ± 8% perf-profile.self.cycles-pp.tcp_create_openreq_child
0.04 ± 55% +0.1 0.14 ± 8% perf-profile.self.cycles-pp.tcp_get_metrics
0.03 ± 91% +0.1 0.13 ± 7% perf-profile.self.cycles-pp.tcp_openreq_init_rwin
0.06 ± 11% +0.1 0.16 ± 8% perf-profile.self.cycles-pp.tcp_connect_init
0.02 ±120% +0.1 0.13 ± 10% perf-profile.self.cycles-pp.inet_lookup_ifaddr_rcu
0.08 ± 15% +0.1 0.19 ± 9% perf-profile.self.cycles-pp.tcp_update_metrics
0.06 ± 13% +0.1 0.19 ± 7% perf-profile.self.cycles-pp.skb_entail
0.14 ± 6% +0.1 0.26 ± 7% perf-profile.self.cycles-pp.tcp_parse_options
0.00 +0.1 0.12 ± 8% perf-profile.self.cycles-pp.tcp_stream_memory_free
0.04 ± 55% +0.1 0.17 ± 8% perf-profile.self.cycles-pp.tcp_init_metrics
0.46 ± 6% +0.1 0.59 ± 6% perf-profile.self.cycles-pp.tcp_ack
0.11 ± 7% +0.1 0.24 ± 8% perf-profile.self.cycles-pp.tcp_event_new_data_sent
0.12 ± 8% +0.1 0.25 ± 8% perf-profile.self.cycles-pp.tcp_update_pacing_rate
0.00 ±458% +0.1 0.13 ± 8% perf-profile.self.cycles-pp.secure_tcp_ts_off
0.44 ± 4% +0.1 0.58 ± 8% perf-profile.self.cycles-pp.tcp_v4_rcv
0.00 +0.1 0.15 ± 9% perf-profile.self.cycles-pp.__sk_free
0.06 ± 8% +0.2 0.21 ± 8% perf-profile.self.cycles-pp.tcp_set_state
0.08 ± 6% +0.2 0.23 ± 6% perf-profile.self.cycles-pp.tcp_assign_congestion_control
0.02 ±146% +0.2 0.21 ± 7% perf-profile.self.cycles-pp.tcp_init_sock
0.10 ± 7% +0.2 0.30 ± 7% perf-profile.self.cycles-pp.ip_rcv_finish_core
0.16 ± 7% +0.2 0.37 ± 7% perf-profile.self.cycles-pp.tcp_schedule_loss_probe
0.07 ± 9% +0.2 0.28 ± 7% perf-profile.self.cycles-pp.fib_lookup_good_nhc
0.03 ± 83% +0.2 0.25 ± 6% perf-profile.self.cycles-pp.tcp_newly_delivered
0.10 ± 10% +0.3 0.37 ± 8% perf-profile.self.cycles-pp.__tcp_select_window
0.28 ± 6% +0.3 0.58 ± 8% perf-profile.self.cycles-pp.tcp_write_xmit
0.11 ± 6% +0.3 0.46 ± 8% perf-profile.self.cycles-pp.tcp_ack_update_rtt
0.14 ± 9% +0.3 0.49 ± 7% perf-profile.self.cycles-pp.inet_ehashfn
0.16 ± 8% +0.4 0.53 ± 7% perf-profile.self.cycles-pp.ip_rcv_core
0.75 ± 5% +0.4 1.17 ± 7% perf-profile.self.cycles-pp.__tcp_transmit_skb
0.08 ± 8% +0.5 0.56 ± 5% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
0.24 ± 5% +0.5 0.77 ± 6% perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
0.23 ± 5% +0.6 0.84 ± 7% perf-profile.self.cycles-pp.ip_output
0.24 ± 6% +0.9 1.17 ± 7% perf-profile.self.cycles-pp.ipv4_dst_check
[3]
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json: "netperf.Throughput_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30212.148125000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30112.125625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30178.713125000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30206.963125000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30208.091875000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30139.37625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30163.055625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30238.72875,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30188.946874999994,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30239.25625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30241.643125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30208.288125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30232.425,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30157.838125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30220.53375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30256.23625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30214.021250000005,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30203.740624999995,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30209.22375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 30245.735624999998
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- ],
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json: "netperf.Throughput_total_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483394.37000000005,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 481794.01,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 482859.41000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483311.41000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483329.47000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 482230.02,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 482608.89,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483819.66,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483023.1499999999,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483828.1,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483866.29,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483332.61,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483718.8,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 482525.41,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483528.54,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 484099.78,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483424.3400000001,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483259.8499999999,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483347.58,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- 483931.76999999996
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json- ],
[4]
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-icl-2sp2/TCP_CRR/netperf/0xd000280
commit:
652c1b17b8 ("bpf: Refactor perf_event_set_bpf_prog() to use struct bpf_prog input")
b89fbfbb85 ("bpf: Implement minimal BPF perf link")
652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc
---------------- ---------------------------
%stddev %change %stddev
\ | \
522472 -5.3% 494763 netperf.Throughput_total_tps
32654 -5.3% 30922 netperf.Throughput_tps
305.01 -0.0% 305.01 netperf.time.elapsed_time
305.01 -0.0% 305.01 netperf.time.elapsed_time.max
514102 +0.3% 515832 netperf.time.involuntary_context_switches
0.25 ±214% -20.0% 0.20 ±200% netperf.time.major_page_faults
2497 -0.0% 2497 netperf.time.maximum_resident_set_size
7473 ± 5% -0.3% 7451 ± 5% netperf.time.minor_page_faults
4096 +0.0% 4096 netperf.time.page_size
940.65 -1.0% 931.30 netperf.time.percent_of_cpu_this_job_got
2762 -0.7% 2743 netperf.time.system_time
107.63 -8.0% 99.06 netperf.time.user_time
1.601e+08 -1.5% 1.577e+08 netperf.time.voluntary_context_switches
1.567e+08 -5.3% 1.484e+08 netperf.workload
305.01 -0.0% 305.01 time.elapsed_time
305.01 -0.0% 305.01 time.elapsed_time.max
514102 +0.3% 515832 time.involuntary_context_switches
0.25 ±214% -20.0% 0.20 ±200% time.major_page_faults
2497 -0.0% 2497 time.maximum_resident_set_size
7473 ± 5% -0.3% 7451 ± 5% time.minor_page_faults
4096 +0.0% 4096 time.page_size
940.65 -1.0% 931.30 time.percent_of_cpu_this_job_got
2762 -0.7% 2743 time.system_time
107.63 -8.0% 99.06 time.user_time
1.601e+08 -1.5% 1.577e+08 time.voluntary_context_switches
>
> >
> >
> > in testcase: netperf
> > on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242
> > CPU @ 2.30GHz with 192G memory with following parameters:
> >
> > ip: ipv4
> > runtime: 300s
> > nr_threads: 16
> > cluster: cs-localhost
> > test: TCP_CRR
> > cpufreq_governor: performance
> > ucode: 0x5003006
> >
> > test-description: Netperf is a benchmark that can be use to measure various
> aspect of networking performance.
> > test-url: http://www.netperf.org/netperf/
> >
> >
> >
> [...]
> _______________________________________________
> LKP mailing list -- lkp@xxxxxxxxxxxx
> To unsubscribe send an email to lkp-leave@xxxxxxxxxxxx
