RE: [LKP] Re: [bpf] b89fbfbb85: netperf.Throughput_tps -21.3% regression

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Yonghong,


> -----Original Message-----
> From: Yonghong Song <yhs@xxxxxx>
> Sent: Friday, August 20, 2021 2:29 PM
> To: Sang, Oliver <oliver.sang@xxxxxxxxx>; Andrii Nakryiko <andrii@xxxxxxxxxx>
> Cc: Daniel Borkmann <daniel@xxxxxxxxxxxxx>; Peter Zijlstra
> <peterz@xxxxxxxxxxxxx>; LKML <linux-kernel@xxxxxxxxxxxxxxx>; Linux Memory
> Management List <linux-mm@xxxxxxxxx>; lkp@xxxxxxxxxxxx; lkp <lkp@xxxxxxxxx>
> Subject: [LKP] Re: [bpf] b89fbfbb85: netperf.Throughput_tps -21.3% regression
> 
> 
> 
> On 8/19/21 10:50 PM, kernel test robot wrote:
> >
> >
> > Greeting,
> >
> > FYI, we noticed a -21.3% regression of netperf.Throughput_tps due to commit:
> >
> >
> > commit: b89fbfbb854c9afc3047e8273cc3a694650b802e ("bpf: Implement
> > minimal BPF perf link")
> > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git
> > master
> 
> Could you do a test again to double check?
> The above patch is related to bpf perf_event area.
> It should not have any impact with netperf.
 
We first tested again and confirmed the regression still exists (-21.4%), the result is stable [1], more details are in [2].

652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    382134           -21.4%     300300        netperf.Throughput_total_tps
     23958           -21.4%      18828        netperf.Throughput_tps

Then we tried another platform, Ice Lake, it has 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G, but we found the regression is still existing (-5.4%), the result is stable [3], more details are in [4].

652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    522472            -5.3%     494763        netperf.Throughput_total_tps
     32654            -5.3%      30922        netperf.Throughput_tps

[1]

b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json:  "netperf.Throughput_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18860.396249999998,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18808.879375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18905.214375000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18750.746874999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18803.823125000006,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    19201.846,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18819.47,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18979.38,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18837.719375000004,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18809.4725,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18776.799375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18773.092500000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18743.978125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18794.814374999998,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18810.783124999998,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18821.65625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18680.149375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18747.203125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18819.728750000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18830.020625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    18656.501874999998
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-  ],


b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json:  "netperf.Throughput_total_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    301766.33999999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    300942.07,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    302483.43000000005,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    300011.94999999995,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    300861.1700000001,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    288027.69,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    301111.52,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    303670.08,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    301403.51000000007,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    300951.56,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    300428.79,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    300369.48000000004,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    299903.65,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    300717.02999999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    300972.52999999997,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    301146.5,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    298882.39,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    299955.25,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    301115.66000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    301280.33,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    298504.02999999997
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-  ], 

[2]

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-csl-2ap3/TCP_CRR/netperf/0x5003006

commit: 
  652c1b17b8 ("bpf: Refactor perf_event_set_bpf_prog() to use struct bpf_prog input")
  b89fbfbb85 ("bpf: Implement minimal BPF perf link")

652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    381418           -21.3%     300214        netperf.Throughput_total_tps
     23906           -21.3%      18820        netperf.Throughput_tps
    872.95            -5.2%     827.95        netperf.time.percent_of_cpu_this_job_got
      2586            -4.3%       2474        netperf.time.system_time
  2.25e+08           -22.8%  1.736e+08        netperf.time.voluntary_context_switches
 1.144e+08           -21.3%   90064356        netperf.workload
   2346775            -8.7%    2143411        vmstat.system.cs
    872.95            -5.2%     827.95        time.percent_of_cpu_this_job_got
      2586            -4.3%       2474        time.system_time
  2.25e+08           -22.8%  1.736e+08        time.voluntary_context_switches
      0.01 ±  7%     +68.4%       0.01 ±  7%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
   3491773 ±  2%     -22.6%    2702148        perf-sched.wait_and_delay.count.schedule_timeout.inet_csk_accept.inet_accept.__sys_accept4_file
   4512682 ±  4%     +21.9%    5501355 ±  2%  perf-sched.wait_and_delay.count.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      0.01 ±  8%     +86.5%       0.01 ±  8%  perf-sched.wait_time.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
      0.03 ±  8%    +171.3%       0.08 ±231%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.apparmor_file_alloc_security.security_file_alloc.__alloc_file
    113308            -2.8%     110156        proc-vmstat.nr_slab_unreclaimable
   4057280 ±  2%      -8.6%    3709088 ±  2%  proc-vmstat.numa_hit
   3847678 ±  2%      -8.9%    3506977 ±  2%  proc-vmstat.numa_local
   4055392 ±  2%      -8.5%    3709694 ±  2%  proc-vmstat.pgalloc_normal
   1770294 ±  2%     -23.5%    1355122 ±  2%  proc-vmstat.pgfree
   2456040 ± 16%     -34.0%    1620559 ± 20%  softirqs.CPU121.NET_RX
   1737990 ± 24%     -36.1%    1109923 ± 27%  softirqs.CPU163.NET_RX
 3.519e+08           -22.1%  2.742e+08        softirqs.NET_RX
  26128869 ±  2%     -15.7%   22015818        softirqs.RCU
    237092 ±  2%     +84.0%     436186        softirqs.TIMER
    170292           -12.0%     149884        slabinfo.filp.active_objs
      2663           -12.0%       2343        slabinfo.filp.active_slabs
    170459           -12.0%     150036        slabinfo.filp.num_objs
      2663           -12.0%       2343        slabinfo.filp.num_slabs
     49886 ±  5%     -11.6%      44079 ±  4%  slabinfo.kmalloc-512.active_objs
     50080 ±  5%     -11.6%      44269 ±  4%  slabinfo.kmalloc-512.num_objs
     12.06 ±  4%     +26.2%      15.22 ±  2%  perf-stat.i.MPKI
 1.004e+10           -18.0%  8.233e+09        perf-stat.i.branch-instructions
 1.562e+08           -19.6%  1.256e+08        perf-stat.i.branch-misses
     10.92            +7.2       18.09        perf-stat.i.cache-miss-rate%
  64015758 ±  2%     +75.1%  1.121e+08        perf-stat.i.cache-misses
 5.888e+08            +5.2%  6.196e+08        perf-stat.i.cache-references
   2379887            -8.7%    2173629        perf-stat.i.context-switches
      1.57 ±  2%     +17.2%       1.84        perf-stat.i.cpi
 7.739e+10            -3.5%  7.467e+10        perf-stat.i.cpu-cycles
      1272 ±  2%     -43.8%     715.53 ±  3%  perf-stat.i.cycles-between-cache-misses
    903463 ± 18%     -27.4%     655855 ±  5%  perf-stat.i.dTLB-load-misses
 1.518e+10           -18.3%   1.24e+10        perf-stat.i.dTLB-loads
    228225 ± 31%     -52.8%     107778 ± 23%  perf-stat.i.dTLB-store-misses
 8.879e+09           -18.5%  7.234e+09        perf-stat.i.dTLB-stores
     89.92            -4.7       85.18        perf-stat.i.iTLB-load-miss-rate%
 1.075e+08           -21.1%   84797404        perf-stat.i.iTLB-load-misses
  11549994 ±  4%     +23.9%   14312532 ±  2%  perf-stat.i.iTLB-loads
 5.079e+10           -18.2%  4.155e+10        perf-stat.i.instructions
      0.65           -15.2%       0.55        perf-stat.i.ipc
      0.40            -3.5%       0.39        perf-stat.i.metric.GHz
    180.81           -17.8%     148.59        perf-stat.i.metric.M/sec
  13211450           +59.2%   21027780        perf-stat.i.node-load-misses
   1191177 ±  5%     +68.5%    2006794 ±  9%  perf-stat.i.node-loads
  10841458           -22.2%    8430280        perf-stat.i.node-store-misses
     11.59           +28.7%      14.91        perf-stat.overall.MPKI
      1.56            -0.0        1.53        perf-stat.overall.branch-miss-rate%
     10.87            +7.2       18.09        perf-stat.overall.cache-miss-rate%
      1.52           +18.0%       1.80        perf-stat.overall.cpi
      1209 ±  2%     -44.9%     666.15        perf-stat.overall.cycles-between-cache-misses
     90.30            -4.7       85.56        perf-stat.overall.iTLB-load-miss-rate%
    472.39            +3.8%     490.15        perf-stat.overall.instructions-per-iTLB-miss
      0.66           -15.2%       0.56        perf-stat.overall.ipc
     1e+10           -18.0%  8.205e+09        perf-stat.ps.branch-instructions
 1.557e+08           -19.6%  1.251e+08        perf-stat.ps.branch-misses
  63792201 ±  2%     +75.1%  1.117e+08        perf-stat.ps.cache-misses
 5.867e+08            +5.2%  6.174e+08        perf-stat.ps.cache-references
   2371522            -8.7%    2165977        perf-stat.ps.context-switches
 7.712e+10            -3.5%  7.441e+10        perf-stat.ps.cpu-cycles
    900712 ± 18%     -27.4%     653896 ±  5%  perf-stat.ps.dTLB-load-misses
 1.513e+10           -18.3%  1.236e+10        perf-stat.ps.dTLB-loads
    227485 ± 31%     -52.8%     107457 ± 23%  perf-stat.ps.dTLB-store-misses
 8.848e+09           -18.5%  7.208e+09        perf-stat.ps.dTLB-stores
 1.072e+08           -21.1%   84499344        perf-stat.ps.iTLB-load-misses
  11509852 ±  4%     +23.9%   14262519 ±  2%  perf-stat.ps.iTLB-loads
 5.062e+10           -18.2%   4.14e+10        perf-stat.ps.instructions
  13165220           +59.2%   20953789        perf-stat.ps.node-load-misses
   1187095 ±  5%     +68.5%    1999816 ±  9%  perf-stat.ps.node-loads
  10803432           -22.2%    8400688        perf-stat.ps.node-store-misses
 1.543e+13           -18.2%  1.262e+13        perf-stat.total.instructions
      5.24 ±  7%      -1.2        4.05 ±  7%  perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.22 ±  7%      -1.2        4.03 ±  7%  perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      4.13 ± 21%      -1.0        3.17 ±  6%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      2.75 ±  6%      -0.6        2.15 ±  8%  perf-profile.calltrace.cycles-pp.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.71 ±  9%      -0.5        1.25 ±  9%  perf-profile.calltrace.cycles-pp.ret_from_fork
      1.71 ±  9%      -0.5        1.25 ±  9%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      1.67 ±  9%      -0.5        1.22 ±  9%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      1.66 ±  9%      -0.5        1.21 ±  9%  perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
      1.67 ±  9%      -0.5        1.22 ±  9%  perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
      1.68 ±  9%      -0.5        1.23 ±  9%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork
      1.65 ±  9%      -0.5        1.20 ±  9%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
      1.58 ±  9%      -0.4        1.14 ±  9%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd
      1.40 ±  9%      -0.4        1.02 ±  9%  perf-profile.calltrace.cycles-pp.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core.__softirqentry_text_start
      1.36 ±  9%      -0.4        0.99 ±  9%  perf-profile.calltrace.cycles-pp.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch.rcu_core
      1.39 ±  8%      -0.3        1.09 ±  9%  perf-profile.calltrace.cycles-pp.inet_accept.__sys_accept4_file.__sys_accept4.__x64_sys_accept.do_syscall_64
      1.07 ± 10%      -0.3        0.78 ±  9%  perf-profile.calltrace.cycles-pp.page_counter_cancel.page_counter_uncharge.obj_cgroup_uncharge_pages.kmem_cache_free.rcu_do_batch
      1.10 ±  6%      -0.3        0.83 ±  7%  perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      1.03 ±  6%      -0.3        0.78 ±  7%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu
      1.04 ±  8%      -0.2        0.79 ±  8%  perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      0.96 ±  6%      -0.2        0.72 ±  7%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process.tcp_v4_rcv
      0.95 ±  6%      -0.2        0.71 ±  7%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_child_process
      0.97 ±  8%      -0.2        0.74 ±  8%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.92 ±  6%      -0.2        0.68 ±  7%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable
      0.90 ±  8%      -0.2        0.68 ±  8%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
      0.88 ±  8%      -0.2        0.67 ±  8%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
      0.75 ±  7%      -0.2        0.57 ± 23%  perf-profile.calltrace.cycles-pp.alloc_file_pseudo.sock_alloc_file.__sys_accept4_file.__sys_accept4.__x64_sys_accept
      0.67 ±  7%      -0.2        0.49 ± 41%  perf-profile.calltrace.cycles-pp.inet_csk_get_port.__inet_bind.inet_bind.__sys_bind.__x64_sys_bind
      0.55 ± 22%      +0.2        0.72 ±  7%  perf-profile.calltrace.cycles-pp.tcp_v4_send_synack.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv
      0.49 ± 40%      +0.3        0.75 ±  8%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      1.65 ±  6%      +0.3        1.97 ±  7%  perf-profile.calltrace.cycles-pp.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      1.87 ±  6%      +0.4        2.24 ±  7%  perf-profile.calltrace.cycles-pp.tcp_check_req.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
      1.56 ±  6%      +0.5        2.02 ±  6%  perf-profile.calltrace.cycles-pp.tcp_conn_request.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      0.00            +0.6        0.57 ±  6%  perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      3.08 ±  6%      +0.7        3.74 ±  7%  perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect
      3.05 ±  6%      +0.7        3.73 ±  7%  perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect
      3.42 ±  6%      +0.8        4.18 ±  7%  perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect
      3.64 ±  6%      +0.9        4.50 ±  6%  perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect
      4.42 ±  6%      +1.0        5.44 ±  7%  perf-profile.calltrace.cycles-pp.tcp_connect.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect
      5.36 ±  6%      +1.3        6.68 ±  7%  perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect
     11.59 ±  5%      +2.1       13.65 ±  6%  perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe
     11.54 ±  5%      +2.1       13.61 ±  6%  perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64
      5.94 ±  6%      -1.3        4.60 ±  6%  perf-profile.children.cycles-pp.__x64_sys_recvfrom
      5.90 ±  6%      -1.3        4.58 ±  6%  perf-profile.children.cycles-pp.__sys_recvfrom
      5.32 ±  6%      -1.2        4.08 ±  6%  perf-profile.children.cycles-pp.inet_recvmsg
      5.30 ±  6%      -1.2        4.07 ±  6%  perf-profile.children.cycles-pp.tcp_recvmsg
      4.17 ± 22%      -1.0        3.19 ±  6%  perf-profile.children.cycles-pp.poll_idle
      4.48 ±  6%      -0.8        3.71 ±  6%  perf-profile.children.cycles-pp.tcp_recvmsg_locked
      3.06 ±  8%      -0.7        2.39 ±  7%  perf-profile.children.cycles-pp.kmem_cache_free
      1.66 ±  6%      -0.6        1.06 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      2.80 ±  6%      -0.6        2.21 ±  7%  perf-profile.children.cycles-pp.__x64_sys_accept
      2.78 ±  6%      -0.6        2.20 ±  7%  perf-profile.children.cycles-pp.__sys_accept4
      2.75 ±  6%      -0.6        2.17 ±  7%  perf-profile.children.cycles-pp.__sys_accept4_file
      2.37 ±  9%      -0.5        1.85 ±  8%  perf-profile.children.cycles-pp.rcu_core
      2.33 ±  9%      -0.5        1.81 ±  8%  perf-profile.children.cycles-pp.rcu_do_batch
      2.35 ±  8%      -0.5        1.84 ±  8%  perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages
      2.31 ±  8%      -0.5        1.81 ±  8%  perf-profile.children.cycles-pp.page_counter_uncharge
      1.12 ±  7%      -0.5        0.63 ±  7%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.37 ±  6%      -0.5        0.89 ±  6%  perf-profile.children.cycles-pp.lock_sock_nested
      1.71 ±  9%      -0.5        1.25 ±  9%  perf-profile.children.cycles-pp.ret_from_fork
      1.71 ±  9%      -0.5        1.25 ±  9%  perf-profile.children.cycles-pp.kthread
      1.67 ±  9%      -0.5        1.22 ±  9%  perf-profile.children.cycles-pp.run_ksoftirqd
      1.68 ±  9%      -0.5        1.23 ±  9%  perf-profile.children.cycles-pp.smpboot_thread_fn
      2.84 ±  6%      -0.4        2.43 ±  6%  perf-profile.children.cycles-pp.sk_wait_data
      1.78 ±  6%      -0.4        1.36 ±  6%  perf-profile.children.cycles-pp.__dev_queue_xmit
      1.83 ±  9%      -0.4        1.43 ±  8%  perf-profile.children.cycles-pp.page_counter_cancel
      2.14 ±  6%      -0.4        1.75 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock
      1.44 ±  7%      -0.3        1.13 ±  6%  perf-profile.children.cycles-pp.kmem_cache_alloc
      1.40 ±  7%      -0.3        1.10 ±  8%  perf-profile.children.cycles-pp.inet_accept
      1.56 ±  6%      -0.3        1.26 ±  6%  perf-profile.children.cycles-pp.alloc_file_pseudo
      1.56 ±  6%      -0.3        1.27 ±  6%  perf-profile.children.cycles-pp.sock_alloc_file
      1.03 ±  6%      -0.3        0.73 ±  7%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      1.20 ±  6%      -0.3        0.92 ±  6%  perf-profile.children.cycles-pp.__alloc_skb
      0.62 ±  6%      -0.3        0.37 ±  9%  perf-profile.children.cycles-pp.__tcp_get_metrics
      0.97 ±  5%      -0.2        0.73 ±  6%  perf-profile.children.cycles-pp.autoremove_wake_function
      1.15 ±  6%      -0.2        0.92 ±  8%  perf-profile.children.cycles-pp.inet_csk_accept
      0.59 ±  5%      -0.2        0.36 ±  7%  perf-profile.children.cycles-pp.__cgroup_bpf_run_filter_skb
      0.80 ±  6%      -0.2        0.58 ±  6%  perf-profile.children.cycles-pp.sk_forced_mem_schedule
      1.12 ±  6%      -0.2        0.90 ±  6%  perf-profile.children.cycles-pp.alloc_file
      1.10 ±  6%      -0.2        0.89 ±  6%  perf-profile.children.cycles-pp.alloc_empty_file
      1.03 ±  6%      -0.2        0.82 ±  7%  perf-profile.children.cycles-pp.sock_def_wakeup
      1.08 ±  6%      -0.2        0.88 ±  6%  perf-profile.children.cycles-pp.__alloc_file
      1.21 ±  6%      -0.2        1.01 ±  6%  perf-profile.children.cycles-pp.inet_csk_destroy_sock
      0.78 ±  6%      -0.2        0.58 ±  6%  perf-profile.children.cycles-pp.sk_stream_alloc_skb
      0.89 ±  6%      -0.2        0.70 ±  7%  perf-profile.children.cycles-pp.dev_hard_start_xmit
      0.36 ±  6%      -0.2        0.18 ±  6%  perf-profile.children.cycles-pp.ip_finish_output
      0.85 ±  6%      -0.2        0.68 ±  7%  perf-profile.children.cycles-pp.__sk_mem_reduce_allocated
      0.78 ±  6%      -0.2        0.61 ±  7%  perf-profile.children.cycles-pp.loopback_xmit
      1.00 ±  6%      -0.2        0.83 ±  6%  perf-profile.children.cycles-pp.__kfree_skb
      1.15 ±  6%      -0.2        0.98 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.68 ±  6%      -0.2        0.51 ±  7%  perf-profile.children.cycles-pp.tcp_get_metrics
      0.79 ±  6%      -0.2        0.63 ±  6%  perf-profile.children.cycles-pp.tcp_send_fin
      0.92 ±  6%      -0.2        0.76 ±  6%  perf-profile.children.cycles-pp.inet_csk_clone_lock
      0.67 ±  6%      -0.2        0.51 ±  8%  perf-profile.children.cycles-pp.sk_reset_timer
      0.68 ±  6%      -0.2        0.52 ±  7%  perf-profile.children.cycles-pp.sock_alloc
      0.67 ±  6%      -0.2        0.52 ±  7%  perf-profile.children.cycles-pp.new_inode_pseudo
      0.89 ±  5%      -0.1        0.74 ±  6%  perf-profile.children.cycles-pp.sk_clone_lock
      0.77 ±  7%      -0.1        0.63 ±  7%  perf-profile.children.cycles-pp.tcp_done
      0.76 ±  6%      -0.1        0.62 ±  6%  perf-profile.children.cycles-pp.security_file_alloc
      0.58 ±  7%      -0.1        0.45 ±  7%  perf-profile.children.cycles-pp.alloc_inode
      0.59 ±  6%      -0.1        0.46 ±  7%  perf-profile.children.cycles-pp.mod_timer
      0.67 ±  6%      -0.1        0.54 ±  6%  perf-profile.children.cycles-pp.__dentry_kill
      0.69 ±  6%      -0.1        0.57 ±  6%  perf-profile.children.cycles-pp.apparmor_file_alloc_security
      0.45 ±  6%      -0.1        0.33 ±  7%  perf-profile.children.cycles-pp.lock_timer_base
      0.58 ±  7%      -0.1        0.47 ±  6%  perf-profile.children.cycles-pp.apparmor_file_free_security
      0.59 ±  7%      -0.1        0.47 ±  6%  perf-profile.children.cycles-pp.security_file_free
      0.68 ±  7%      -0.1        0.56 ±  7%  perf-profile.children.cycles-pp.inet_csk_get_port
      0.44 ±  5%      -0.1        0.33 ±  7%  perf-profile.children.cycles-pp.sk_filter_trim_cap
      0.47 ±  7%      -0.1        0.36 ±  7%  perf-profile.children.cycles-pp.sock_alloc_inode
      0.54 ±  8%      -0.1        0.44 ± 10%  perf-profile.children.cycles-pp.propagate_protected_usage
      0.68 ±  6%      -0.1        0.58 ±  6%  perf-profile.children.cycles-pp.skb_release_data
      0.43 ±  6%      -0.1        0.33 ±  7%  perf-profile.children.cycles-pp.kmalloc_reserve
      0.45 ± 10%      -0.1        0.36 ± 10%  perf-profile.children.cycles-pp.inet_csk_bind_conflict
      0.38 ±  6%      -0.1        0.30 ±  7%  perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      0.40 ±  6%      -0.1        0.32 ±  7%  perf-profile.children.cycles-pp.raw_local_deliver
      0.46 ±  6%      -0.1        0.38 ±  7%  perf-profile.children.cycles-pp.read_tsc
      0.40 ±  6%      -0.1        0.32 ±  5%  perf-profile.children.cycles-pp.skb_release_all
      0.44 ±  6%      -0.1        0.36 ±  7%  perf-profile.children.cycles-pp.sock_recvmsg
      0.37 ±  7%      -0.1        0.29 ±  8%  perf-profile.children.cycles-pp.__tcp_send_ack
      0.46 ±  7%      -0.1        0.38 ±  7%  perf-profile.children.cycles-pp.__skb_datagram_iter
      0.48 ±  6%      -0.1        0.40 ±  7%  perf-profile.children.cycles-pp.___might_sleep
      0.43 ±  7%      -0.1        0.36 ±  8%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.38 ±  7%      -0.1        0.30 ±  5%  perf-profile.children.cycles-pp.skb_release_head_state
      0.37 ±  7%      -0.1        0.29 ±  7%  perf-profile.children.cycles-pp.apparmor_sk_clone_security
      0.37 ±  7%      -0.1        0.30 ±  6%  perf-profile.children.cycles-pp.security_sk_clone
      0.29 ±  6%      -0.1        0.22 ±  8%  perf-profile.children.cycles-pp.__ksize
      0.31 ±  6%      -0.1        0.24 ±  9%  perf-profile.children.cycles-pp.netif_rx
      0.37 ±  7%      -0.1        0.29 ±  7%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.32 ±  6%      -0.1        0.25 ±  8%  perf-profile.children.cycles-pp.tcp_write_queue_purge
      0.30 ±  7%      -0.1        0.23 ±  9%  perf-profile.children.cycles-pp.netif_rx_internal
      0.24 ±  5%      -0.1        0.17 ±  7%  perf-profile.children.cycles-pp.ip_local_out
      0.30 ±  6%      -0.1        0.23 ±  8%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.40 ±  6%      -0.1        0.33 ±  7%  perf-profile.children.cycles-pp.apparmor_socket_post_create
      0.40 ±  6%      -0.1        0.33 ±  6%  perf-profile.children.cycles-pp.security_socket_post_create
      0.25 ±  8%      -0.1        0.18 ±  6%  perf-profile.children.cycles-pp.__skb_clone
      0.32 ±  6%      -0.1        0.25 ±  7%  perf-profile.children.cycles-pp.tcp_mstamp_refresh
      0.22 ±  4%      -0.1        0.16 ±  6%  perf-profile.children.cycles-pp.__ip_local_out
      0.26 ±  7%      -0.1        0.20 ±  9%  perf-profile.children.cycles-pp.validate_xmit_skb
      0.27 ±  7%      -0.1        0.21 ±  6%  perf-profile.children.cycles-pp.sk_stop_timer
      0.26 ±  6%      -0.1        0.20 ±  7%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.21 ±  8%      -0.1        0.15 ±  8%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.14 ±  7%      -0.1        0.08 ± 10%  perf-profile.children.cycles-pp.migrate_enable
      0.20 ±  7%      -0.1        0.15 ±  8%  perf-profile.children.cycles-pp.__check_object_size
      0.18 ±  6%      -0.1        0.12 ±  6%  perf-profile.children.cycles-pp.ip_send_check
      0.24 ±  8%      -0.1        0.19 ±  7%  perf-profile.children.cycles-pp.__slab_free
      0.24 ±  6%      -0.1        0.19 ±  6%  perf-profile.children.cycles-pp.inet_csk_clear_xmit_timers
      0.26 ±  8%      -0.0        0.22 ±  6%  perf-profile.children.cycles-pp.d_alloc_pseudo
      0.21 ±  6%      -0.0        0.17 ±  9%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.12 ±  8%      -0.0        0.07 ±  8%  perf-profile.children.cycles-pp.sock_put
      0.28 ±  6%      -0.0        0.24 ±  7%  perf-profile.children.cycles-pp.__switch_to
      0.18 ±  7%      -0.0        0.14 ±  7%  perf-profile.children.cycles-pp.del_timer
      0.21 ±  5%      -0.0        0.17 ±  9%  perf-profile.children.cycles-pp.kfree
      0.15 ±  9%      -0.0        0.11 ±  7%  perf-profile.children.cycles-pp.get_obj_cgroup_from_current
      0.12 ± 10%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.tcp_queue_rcv
      0.08 ±  9%      -0.0        0.05 ± 41%  perf-profile.children.cycles-pp.netdev_core_pick_tx
      0.13 ±  7%      -0.0        0.10 ± 11%  perf-profile.children.cycles-pp.netif_skb_features
      0.08 ±  9%      -0.0        0.05 ± 34%  perf-profile.children.cycles-pp.nf_hook_slow
      0.22 ±  6%      -0.0        0.18 ±  9%  perf-profile.children.cycles-pp.__might_sleep
      0.10 ±  9%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.__destroy_inode
      0.14 ±  7%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.tcp_rcv_space_adjust
      0.10 ±  8%      -0.0        0.07 ±  9%  perf-profile.children.cycles-pp.destroy_inode
      0.09 ±  8%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.inet_csk_complete_hashdance
      0.14 ±  7%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.copyout
      0.22 ±  7%      -0.0        0.19 ±  8%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.18 ±  6%      -0.0        0.15 ±  8%  perf-profile.children.cycles-pp.__fget_light
      0.14 ±  8%      -0.0        0.11 ±  9%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.15 ±  8%      -0.0        0.13 ±  7%  perf-profile.children.cycles-pp.__cond_resched
      0.09 ±  9%      -0.0        0.06 ± 10%  perf-profile.children.cycles-pp.inet_reqsk_alloc
      0.13 ±  6%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.d_instantiate
      0.09 ± 12%      -0.0        0.06 ± 10%  perf-profile.children.cycles-pp.security_sock_rcv_skb
      0.08 ± 11%      -0.0        0.05 ± 24%  perf-profile.children.cycles-pp.migrate_disable
      0.11 ± 10%      -0.0        0.08 ± 10%  perf-profile.children.cycles-pp.tcp_validate_incoming
      0.14 ±  8%      -0.0        0.11 ±  8%  perf-profile.children.cycles-pp.tcp_wfree
      0.12 ±  7%      -0.0        0.09 ±  7%  perf-profile.children.cycles-pp.tcp_add_backlog
      0.07 ± 10%      -0.0        0.05 ± 33%  perf-profile.children.cycles-pp.fsnotify_destroy_marks
      0.09 ±  8%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.sock_setsockopt
      0.10 ±  7%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.tcp_init_xmit_timers
      0.09 ±  9%      +0.0        0.13 ±  9%  perf-profile.children.cycles-pp.inet_twsk_alloc
      0.06 ±  8%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.tcp_mtup_init
      0.23 ±  6%      +0.0        0.27 ±  5%  perf-profile.children.cycles-pp.tcp_make_synack
      0.06 ±  9%      +0.1        0.12 ±  8%  perf-profile.children.cycles-pp.tcp_synack_rtt_meas
      0.06 ± 10%      +0.1        0.11 ±  8%  perf-profile.children.cycles-pp.tcp_ca_openreq_child
      0.00            +0.1        0.06 ± 10%  perf-profile.children.cycles-pp.tcp_rack_update_reo_wnd
      0.07 ±  9%      +0.1        0.14 ±  9%  perf-profile.children.cycles-pp.__inet_lookup_listener
      0.12 ±  7%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.__ip_finish_output
      0.06 ± 13%      +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.__ip_dev_find
      0.01 ±212%      +0.1        0.09 ±  8%  perf-profile.children.cycles-pp.inet_lhash2_lookup
      0.00            +0.1        0.08 ±  8%  perf-profile.children.cycles-pp.sock_prot_inuse_add
      0.00            +0.1        0.09 ± 10%  perf-profile.children.cycles-pp.tcp_push
      0.07 ± 10%      +0.1        0.16 ±  8%  perf-profile.children.cycles-pp.tcp_select_initial_window
      0.07 ± 10%      +0.1        0.16 ±  7%  perf-profile.children.cycles-pp.tcp_sync_mss
      0.07 ±  9%      +0.1        0.16 ±  8%  perf-profile.children.cycles-pp.secure_tcp_ts_off
      0.36 ±  6%      +0.1        0.47 ±  6%  perf-profile.children.cycles-pp.tcp_finish_connect
      0.02 ±120%      +0.1        0.13 ± 10%  perf-profile.children.cycles-pp.inet_lookup_ifaddr_rcu
      0.08 ± 11%      +0.1        0.19 ±  8%  perf-profile.children.cycles-pp.skb_entail
      0.17 ±  5%      +0.1        0.29 ±  7%  perf-profile.children.cycles-pp.tcp_connect_init
      0.10 ±  9%      +0.1        0.21 ±  7%  perf-profile.children.cycles-pp.tcp_openreq_init_rwin
      0.00            +0.1        0.13 ±  9%  perf-profile.children.cycles-pp.tcp_stream_memory_free
      0.14 ±  6%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.tcp_parse_options
      0.13 ±  8%      +0.1        0.26 ±  8%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.58 ±  5%      +0.2        0.73 ±  5%  perf-profile.children.cycles-pp.tcp_v4_send_synack
      0.08 ±  7%      +0.2        0.24 ±  6%  perf-profile.children.cycles-pp.tcp_assign_congestion_control
      0.34 ±  7%      +0.2        0.50 ±  6%  perf-profile.children.cycles-pp.tcp_v4_init_sock
      0.33 ±  6%      +0.2        0.49 ±  6%  perf-profile.children.cycles-pp.tcp_init_sock
      0.23 ±  6%      +0.2        0.40 ±  6%  perf-profile.children.cycles-pp.fib_table_lookup
      0.00            +0.2        0.17 ±  8%  perf-profile.children.cycles-pp.__sk_free
      0.22 ±  7%      +0.2        0.41 ±  6%  perf-profile.children.cycles-pp.tcp_schedule_loss_probe
      0.14 ±  7%      +0.2        0.33 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish
      0.63 ±  5%      +0.2        0.83 ±  7%  perf-profile.children.cycles-pp.tcp_init_transfer
      0.10 ±  8%      +0.2        0.31 ±  7%  perf-profile.children.cycles-pp.ip_rcv_finish_core
      0.04 ± 40%      +0.2        0.25 ±  6%  perf-profile.children.cycles-pp.tcp_newly_delivered
      0.07 ± 10%      +0.2        0.28 ±  7%  perf-profile.children.cycles-pp.fib_lookup_good_nhc
      0.19 ±  7%      +0.2        0.41 ±  6%  perf-profile.children.cycles-pp.inet_csk_route_child_sock
      0.18 ±  6%      +0.2        0.40 ±  6%  perf-profile.children.cycles-pp.inet_csk_route_req
      0.11 ±  7%      +0.2        0.34 ±  8%  perf-profile.children.cycles-pp.inet_sk_rebuild_header
      0.11 ±  8%      +0.2        0.35 ±  7%  perf-profile.children.cycles-pp.inet_sock_destruct
      0.10 ±  9%      +0.3        0.37 ±  8%  perf-profile.children.cycles-pp.__tcp_select_window
      1.69 ±  5%      +0.3        2.00 ±  6%  perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
      0.20 ±  6%      +0.3        0.52 ±  8%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.15 ±  8%      +0.3        0.49 ±  7%  perf-profile.children.cycles-pp.inet_ehashfn
      1.91 ±  5%      +0.4        2.27 ±  6%  perf-profile.children.cycles-pp.tcp_check_req
      0.17 ±  6%      +0.4        0.54 ±  7%  perf-profile.children.cycles-pp.ip_rcv_core
      1.59 ±  5%      +0.5        2.05 ±  6%  perf-profile.children.cycles-pp.tcp_conn_request
      0.43 ±  5%      +0.6        0.99 ±  5%  perf-profile.children.cycles-pp.ip_route_output_flow
      2.02 ±  6%      +0.6        2.67 ±  6%  perf-profile.children.cycles-pp.tcp_ack
      0.26 ±  7%      +0.8        1.02 ±  7%  perf-profile.children.cycles-pp.__sk_dst_check
      0.60 ±  5%      +0.8        1.37 ±  6%  perf-profile.children.cycles-pp.ip_route_output_key_hash
      0.55 ±  5%      +0.8        1.33 ±  6%  perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu
      5.73 ±  5%      +0.9        6.60 ±  6%  perf-profile.children.cycles-pp.tcp_rcv_synsent_state_process
      0.24 ±  6%      +0.9        1.19 ±  7%  perf-profile.children.cycles-pp.ipv4_dst_check
      4.49 ±  5%      +1.0        5.47 ±  6%  perf-profile.children.cycles-pp.tcp_connect
      5.45 ±  5%      +1.3        6.72 ±  6%  perf-profile.children.cycles-pp.tcp_v4_connect
     11.81 ±  5%      +2.0       13.85 ±  6%  perf-profile.children.cycles-pp.__x64_sys_connect
     11.79 ±  5%      +2.1       13.84 ±  6%  perf-profile.children.cycles-pp.__sys_connect
     11.59 ±  5%      +2.1       13.65 ±  6%  perf-profile.children.cycles-pp.inet_stream_connect
     11.54 ±  5%      +2.1       13.61 ±  6%  perf-profile.children.cycles-pp.__inet_stream_connect
      4.00 ± 20%      -0.9        3.08 ±  6%  perf-profile.self.cycles-pp.poll_idle
      1.12 ±  7%      -0.5        0.63 ±  7%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.82 ±  9%      -0.4        1.42 ±  8%  perf-profile.self.cycles-pp.page_counter_cancel
      1.01 ±  6%      -0.3        0.72 ±  6%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      1.65 ±  6%      -0.3        1.36 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock
      0.62 ±  6%      -0.3        0.36 ±  8%  perf-profile.self.cycles-pp.__tcp_get_metrics
      1.09 ±  6%      -0.3        0.84 ±  6%  perf-profile.self.cycles-pp.__inet_lookup_established
      0.47 ±  9%      -0.2        0.25 ±  7%  perf-profile.self.cycles-pp.__sk_destruct
      0.79 ±  6%      -0.2        0.57 ±  6%  perf-profile.self.cycles-pp.sk_forced_mem_schedule
      0.99 ±  6%      -0.2        0.78 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.84 ±  6%      -0.2        0.67 ±  7%  perf-profile.self.cycles-pp.__sk_mem_reduce_allocated
      1.14 ±  6%      -0.2        0.97 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.36 ±  6%      -0.1        0.22 ±  8%  perf-profile.self.cycles-pp.__cgroup_bpf_run_filter_skb
      0.67 ±  6%      -0.1        0.55 ±  6%  perf-profile.self.cycles-pp.apparmor_file_alloc_security
      0.40 ±  7%      -0.1        0.28 ±  7%  perf-profile.self.cycles-pp.tcp_recvmsg_locked
      0.58 ±  7%      -0.1        0.46 ±  5%  perf-profile.self.cycles-pp.apparmor_file_free_security
      0.50 ±  6%      -0.1        0.38 ±  7%  perf-profile.self.cycles-pp.__dev_queue_xmit
      0.54 ±  9%      -0.1        0.43 ± 10%  perf-profile.self.cycles-pp.propagate_protected_usage
      0.40 ±  6%      -0.1        0.30 ±  7%  perf-profile.self.cycles-pp.kmem_cache_free
      0.33 ±  6%      -0.1        0.24 ±  7%  perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.45 ± 10%      -0.1        0.35 ± 10%  perf-profile.self.cycles-pp.inet_csk_bind_conflict
      0.28 ±  8%      -0.1        0.20 ±  9%  perf-profile.self.cycles-pp.ip_finish_output2
      0.39 ±  6%      -0.1        0.31 ±  7%  perf-profile.self.cycles-pp.raw_local_deliver
      0.45 ±  6%      -0.1        0.37 ±  6%  perf-profile.self.cycles-pp.read_tsc
      0.47 ±  6%      -0.1        0.39 ±  7%  perf-profile.self.cycles-pp.___might_sleep
      0.37 ±  7%      -0.1        0.29 ±  7%  perf-profile.self.cycles-pp.apparmor_sk_clone_security
      0.26 ±  7%      -0.1        0.19 ±  8%  perf-profile.self.cycles-pp.tcp_sendmsg_locked
      0.28 ±  6%      -0.1        0.21 ±  7%  perf-profile.self.cycles-pp.__ksize
      0.39 ±  6%      -0.1        0.32 ±  7%  perf-profile.self.cycles-pp.do_idle
      0.39 ±  6%      -0.1        0.33 ±  7%  perf-profile.self.cycles-pp.apparmor_socket_post_create
      0.20 ±  8%      -0.1        0.14 ±  7%  perf-profile.self.cycles-pp.__skb_clone
      0.23 ±  8%      -0.1        0.17 ±  9%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.13 ±  7%      -0.1        0.07 ± 11%  perf-profile.self.cycles-pp.migrate_enable
      0.17 ±  6%      -0.1        0.12 ±  6%  perf-profile.self.cycles-pp.ip_send_check
      0.24 ±  8%      -0.1        0.19 ±  8%  perf-profile.self.cycles-pp.__slab_free
      0.24 ±  7%      -0.1        0.19 ±  7%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.24 ±  7%      -0.1        0.19 ±  8%  perf-profile.self.cycles-pp.__alloc_skb
      0.17 ±  7%      -0.0        0.13 ±  7%  perf-profile.self.cycles-pp.net_rx_action
      0.18 ±  7%      -0.0        0.13 ± 10%  perf-profile.self.cycles-pp.process_backlog
      0.16 ±  7%      -0.0        0.12 ±  6%  perf-profile.self.cycles-pp.fib_table_lookup
      0.12 ±  9%      -0.0        0.07 ±  8%  perf-profile.self.cycles-pp.sock_put
      0.27 ±  5%      -0.0        0.23 ±  8%  perf-profile.self.cycles-pp.__switch_to
      0.14 ± 11%      -0.0        0.10 ± 11%  perf-profile.self.cycles-pp.do_syscall_64
      0.08 ± 13%      -0.0        0.04 ± 49%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.15 ±  9%      -0.0        0.11 ±  9%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.21 ±  5%      -0.0        0.17 ±  9%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.21 ±  7%      -0.0        0.17 ±  9%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.20 ±  5%      -0.0        0.17 ±  9%  perf-profile.self.cycles-pp.kfree
      0.08 ±  7%      -0.0        0.04 ± 64%  perf-profile.self.cycles-pp.netdev_core_pick_tx
      0.15 ±  6%      -0.0        0.12 ±  9%  perf-profile.self.cycles-pp.__tcp_close
      0.11 ±  9%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.tcp_queue_rcv
      0.07 ±  9%      -0.0        0.04 ± 56%  perf-profile.self.cycles-pp.migrate_disable
      0.07 ±  8%      -0.0        0.04 ± 64%  perf-profile.self.cycles-pp.inet_reqsk_alloc
      0.09 ±  9%      -0.0        0.05 ± 10%  perf-profile.self.cycles-pp.tcp_data_queue
      0.18 ±  8%      -0.0        0.15 ±  6%  perf-profile.self.cycles-pp.__entry_text_start
      0.18 ±  6%      -0.0        0.14 ±  8%  perf-profile.self.cycles-pp.__fget_light
      0.11 ±  9%      -0.0        0.08 ±  7%  perf-profile.self.cycles-pp.get_obj_cgroup_from_current
      0.14 ±  8%      -0.0        0.11 ±  9%  perf-profile.self.cycles-pp.__check_object_size
      0.14 ±  8%      -0.0        0.11 ±  9%  perf-profile.self.cycles-pp.enqueue_to_backlog
      0.14 ±  9%      -0.0        0.11 ± 10%  perf-profile.self.cycles-pp.sk_reset_timer
      0.12 ±  9%      -0.0        0.09 ±  8%  perf-profile.self.cycles-pp.validate_xmit_skb
      0.14 ±  9%      -0.0        0.11 ±  9%  perf-profile.self.cycles-pp.sk_setup_caps
      0.09 ±  7%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.irqtime_account_irq
      0.11 ± 10%      -0.0        0.08 ± 10%  perf-profile.self.cycles-pp.tcp_validate_incoming
      0.14 ±  8%      -0.0        0.11 ±  8%  perf-profile.self.cycles-pp.tcp_wfree
      0.07 ± 10%      -0.0        0.05 ± 23%  perf-profile.self.cycles-pp.tcp_rcv_synsent_state_process
      0.08 ±  9%      -0.0        0.06 ± 10%  perf-profile.self.cycles-pp.sk_filter_trim_cap
      0.08 ±  6%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.__netif_receive_skb_one_core
      0.07 ±  8%      -0.0        0.05 ± 23%  perf-profile.self.cycles-pp.tcp_options_write
      0.08 ±  9%      -0.0        0.06 ± 13%  perf-profile.self.cycles-pp.netif_skb_features
      0.08 ±  9%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.06 ± 10%      +0.0        0.08 ±  8%  perf-profile.self.cycles-pp.tcp_init_transfer
      0.09 ±  9%      +0.0        0.12 ±  9%  perf-profile.self.cycles-pp.inet_twsk_alloc
      0.07 ±  8%      +0.0        0.10 ±  7%  perf-profile.self.cycles-pp.inet_csk_get_port
      0.06 ±  8%      +0.0        0.09 ±  8%  perf-profile.self.cycles-pp.tcp_mtup_init
      0.11 ±  7%      +0.0        0.16 ±  7%  perf-profile.self.cycles-pp.sk_clone_lock
      0.00            +0.1        0.06 ± 10%  perf-profile.self.cycles-pp.tcp_rack_update_reo_wnd
      0.00            +0.1        0.06 ± 12%  perf-profile.self.cycles-pp.inet_twsk_kill
      0.07 ±  7%      +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.tcp_connect
      0.07 ±  7%      +0.1        0.14 ±  7%  perf-profile.self.cycles-pp.__inet_bind
      0.12 ±  7%      +0.1        0.19 ±  7%  perf-profile.self.cycles-pp.__ip_finish_output
      0.06 ± 11%      +0.1        0.14 ±  5%  perf-profile.self.cycles-pp.tcp_make_synack
      0.06 ±  6%      +0.1        0.14 ±  8%  perf-profile.self.cycles-pp.tcp_v4_syn_recv_sock
      0.01 ±251%      +0.1        0.09 ±  8%  perf-profile.self.cycles-pp.inet_lhash2_lookup
      0.00            +0.1        0.08 ±  8%  perf-profile.self.cycles-pp.sock_prot_inuse_add
      0.00            +0.1        0.09 ±  9%  perf-profile.self.cycles-pp.tcp_push
      0.07 ± 10%      +0.1        0.16 ±  8%  perf-profile.self.cycles-pp.tcp_select_initial_window
      0.06 ± 10%      +0.1        0.15 ±  8%  perf-profile.self.cycles-pp.tcp_sync_mss
      0.00            +0.1        0.09 ± 11%  perf-profile.self.cycles-pp.tcp_v4_send_synack
      0.05 ± 22%      +0.1        0.14 ±  8%  perf-profile.self.cycles-pp.tcp_create_openreq_child
      0.04 ± 55%      +0.1        0.14 ±  8%  perf-profile.self.cycles-pp.tcp_get_metrics
      0.03 ± 91%      +0.1        0.13 ±  7%  perf-profile.self.cycles-pp.tcp_openreq_init_rwin
      0.06 ± 11%      +0.1        0.16 ±  8%  perf-profile.self.cycles-pp.tcp_connect_init
      0.02 ±120%      +0.1        0.13 ± 10%  perf-profile.self.cycles-pp.inet_lookup_ifaddr_rcu
      0.08 ± 15%      +0.1        0.19 ±  9%  perf-profile.self.cycles-pp.tcp_update_metrics
      0.06 ± 13%      +0.1        0.19 ±  7%  perf-profile.self.cycles-pp.skb_entail
      0.14 ±  6%      +0.1        0.26 ±  7%  perf-profile.self.cycles-pp.tcp_parse_options
      0.00            +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.tcp_stream_memory_free
      0.04 ± 55%      +0.1        0.17 ±  8%  perf-profile.self.cycles-pp.tcp_init_metrics
      0.46 ±  6%      +0.1        0.59 ±  6%  perf-profile.self.cycles-pp.tcp_ack
      0.11 ±  7%      +0.1        0.24 ±  8%  perf-profile.self.cycles-pp.tcp_event_new_data_sent
      0.12 ±  8%      +0.1        0.25 ±  8%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.00 ±458%      +0.1        0.13 ±  8%  perf-profile.self.cycles-pp.secure_tcp_ts_off
      0.44 ±  4%      +0.1        0.58 ±  8%  perf-profile.self.cycles-pp.tcp_v4_rcv
      0.00            +0.1        0.15 ±  9%  perf-profile.self.cycles-pp.__sk_free
      0.06 ±  8%      +0.2        0.21 ±  8%  perf-profile.self.cycles-pp.tcp_set_state
      0.08 ±  6%      +0.2        0.23 ±  6%  perf-profile.self.cycles-pp.tcp_assign_congestion_control
      0.02 ±146%      +0.2        0.21 ±  7%  perf-profile.self.cycles-pp.tcp_init_sock
      0.10 ±  7%      +0.2        0.30 ±  7%  perf-profile.self.cycles-pp.ip_rcv_finish_core
      0.16 ±  7%      +0.2        0.37 ±  7%  perf-profile.self.cycles-pp.tcp_schedule_loss_probe
      0.07 ±  9%      +0.2        0.28 ±  7%  perf-profile.self.cycles-pp.fib_lookup_good_nhc
      0.03 ± 83%      +0.2        0.25 ±  6%  perf-profile.self.cycles-pp.tcp_newly_delivered
      0.10 ± 10%      +0.3        0.37 ±  8%  perf-profile.self.cycles-pp.__tcp_select_window
      0.28 ±  6%      +0.3        0.58 ±  8%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.11 ±  6%      +0.3        0.46 ±  8%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.14 ±  9%      +0.3        0.49 ±  7%  perf-profile.self.cycles-pp.inet_ehashfn
      0.16 ±  8%      +0.4        0.53 ±  7%  perf-profile.self.cycles-pp.ip_rcv_core
      0.75 ±  5%      +0.4        1.17 ±  7%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.08 ±  8%      +0.5        0.56 ±  5%  perf-profile.self.cycles-pp.ip_protocol_deliver_rcu
      0.24 ±  5%      +0.5        0.77 ±  6%  perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu
      0.23 ±  5%      +0.6        0.84 ±  7%  perf-profile.self.cycles-pp.ip_output
      0.24 ±  6%      +0.9        1.17 ±  7%  perf-profile.self.cycles-pp.ipv4_dst_check

[3]

b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json:  "netperf.Throughput_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30212.148125000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30112.125625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30178.713125000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30206.963125000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30208.091875000002,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30139.37625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30163.055625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30238.72875,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30188.946874999994,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30239.25625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30241.643125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30208.288125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30232.425,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30157.838125,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30220.53375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30256.23625,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30214.021250000005,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30203.740624999995,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30209.22375,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    30245.735624999998
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-  ],

b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json:  "netperf.Throughput_total_tps": [
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483394.37000000005,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    481794.01,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    482859.41000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483311.41000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483329.47000000003,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    482230.02,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    482608.89,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483819.66,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483023.1499999999,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483828.1,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483866.29,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483332.61,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483718.8,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    482525.41,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483528.54,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    484099.78,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483424.3400000001,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483259.8499999999,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483347.58,
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-    483931.76999999996
b89fbfbb854c9afc3047e8273cc3a694650b802e/matrix.json-  ],

[4]

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
  cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/16/debian-10.4-x86_64-20200603.cgz/300s/lkp-icl-2sp2/TCP_CRR/netperf/0xd000280

commit: 
  652c1b17b8 ("bpf: Refactor perf_event_set_bpf_prog() to use struct bpf_prog input")
  b89fbfbb85 ("bpf: Implement minimal BPF perf link")

652c1b17b85b9c19 b89fbfbb854c9afc3047e8273cc 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    522472            -5.3%     494763        netperf.Throughput_total_tps
     32654            -5.3%      30922        netperf.Throughput_tps
    305.01            -0.0%     305.01        netperf.time.elapsed_time
    305.01            -0.0%     305.01        netperf.time.elapsed_time.max
    514102            +0.3%     515832        netperf.time.involuntary_context_switches
      0.25 ±214%     -20.0%       0.20 ±200%  netperf.time.major_page_faults
      2497            -0.0%       2497        netperf.time.maximum_resident_set_size
      7473 ±  5%      -0.3%       7451 ±  5%  netperf.time.minor_page_faults
      4096            +0.0%       4096        netperf.time.page_size
    940.65            -1.0%     931.30        netperf.time.percent_of_cpu_this_job_got
      2762            -0.7%       2743        netperf.time.system_time
    107.63            -8.0%      99.06        netperf.time.user_time
 1.601e+08            -1.5%  1.577e+08        netperf.time.voluntary_context_switches
 1.567e+08            -5.3%  1.484e+08        netperf.workload
    305.01            -0.0%     305.01        time.elapsed_time
    305.01            -0.0%     305.01        time.elapsed_time.max
    514102            +0.3%     515832        time.involuntary_context_switches
      0.25 ±214%     -20.0%       0.20 ±200%  time.major_page_faults
      2497            -0.0%       2497        time.maximum_resident_set_size
      7473 ±  5%      -0.3%       7451 ±  5%  time.minor_page_faults
      4096            +0.0%       4096        time.page_size
    940.65            -1.0%     931.30        time.percent_of_cpu_this_job_got
      2762            -0.7%       2743        time.system_time
    107.63            -8.0%      99.06        time.user_time
 1.601e+08            -1.5%  1.577e+08        time.voluntary_context_switches
> 
> >
> >
> > in testcase: netperf
> > on test machine: 192 threads 4 sockets Intel(R) Xeon(R) Platinum 9242
> > CPU @ 2.30GHz with 192G memory with following parameters:
> >
> > 	ip: ipv4
> > 	runtime: 300s
> > 	nr_threads: 16
> > 	cluster: cs-localhost
> > 	test: TCP_CRR
> > 	cpufreq_governor: performance
> > 	ucode: 0x5003006
> >
> > test-description: Netperf is a benchmark that can be use to measure various
> aspect of networking performance.
> > test-url: http://www.netperf.org/netperf/
> >
> >
> >
> [...]
> _______________________________________________
> LKP mailing list -- lkp@xxxxxxxxxxxx
> To unsubscribe send an email to lkp-leave@xxxxxxxxxxxx





[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux