On Wed, 11 Sep 2024 20:27:27 +0200 Jiri Olsa <olsajiri@xxxxxxxxx> wrote: > On Sun, Aug 18, 2024 at 09:47:53PM +0900, Masami Hiramatsu (Google) wrote: > > Hi, > > > > Here is the 13th version of the series to re-implement the fprobe on > > function-graph tracer. The previous version is; > > > > https://lore.kernel.org/all/172000134410.63468.13742222887213469474.stgit@devnote2/ > > > > This version is based on v6.11-rc3. > > In this version, I added a bugfix as [1/20], which should go to urgent > > branch, and dropped the performance improvement patch which was introduced > > in v12 because I found that does not work with new kernel. > > > > Overview > > -------- > > This series rewrites the fprobe on this function-graph. > > The purposes of this change are; > > > > 1) Remove dependency of the rethook from fprobe so that we can reduce > > the return hook code and shadow stack. > > > > 2) Make 'ftrace_regs' the common trace interface for the function > > boundary. > > > > 1) Currently we have 2(or 3) different function return hook codes, > > the function-graph tracer and rethook (and legacy kretprobe). > > But since this is redundant and needs double maintenance cost, > > I would like to unify those. From the user's viewpoint, function- > > graph tracer is very useful to grasp the execution path. For this > > purpose, it is hard to use the rethook in the function-graph > > tracer, but the opposite is possible. (Strictly speaking, kretprobe > > can not use it because it requires 'pt_regs' for historical reasons.) > > > > 2) Now the fprobe provides the 'pt_regs' for its handler, but that is > > wrong for the function entry and exit. Moreover, depending on the > > architecture, there is no way to accurately reproduce 'pt_regs' > > outside of interrupt or exception handlers. This means fprobe should > > not use 'pt_regs' because it does not use such exceptions. > > (Conversely, kprobe should use 'pt_regs' because it is an abstract > > interface of the software breakpoint exception.) > > > > This series changes fprobe to use function-graph tracer for tracing > > function entry and exit, instead of mixture of ftrace and rethook. > > Unlike the rethook which is a per-task list of system-wide allocated > > nodes, the function graph's ret_stack is a per-task shadow stack. > > Thus it does not need to set 'nr_maxactive' (which is the number of > > pre-allocated nodes). > > Also the handlers will get the 'ftrace_regs' instead of 'pt_regs'. > > Since eBPF mulit_kprobe/multi_kretprobe events still use 'pt_regs' as > > their register interface, this changes it to convert 'ftrace_regs' to > > 'pt_regs'. Of course this conversion makes an incomplete 'pt_regs', > > so users must access only registers for function parameters or > > return value. > > > > Design > > ------ > > Instead of using ftrace's function entry hook directly, the new fprobe > > is built on top of the function-graph's entry and return callbacks > > with 'ftrace_regs'. > > > > Since the fprobe requires access to 'ftrace_regs', the architecture > > must support CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS and > > CONFIG_HAVE_FTRACE_GRAPH_FUNC, which enables to call function-graph > > entry callback with 'ftrace_regs', and also > > CONFIG_HAVE_FUNCTION_GRAPH_FREGS, which passes the ftrace_regs to > > return_to_handler. > > > > All fprobes share a single function-graph ops (means shares a common > > ftrace filter) similar to the kprobe-on-ftrace. This needs another > > layer to find corresponding fprobe in the common function-graph > > callbacks, but has much better scalability, since the number of > > registered function-graph ops is limited. > > > > In the entry callback, the fprobe runs its entry_handler and saves the > > address of 'fprobe' on the function-graph's shadow stack as data. The > > return callback decodes the data to get the 'fprobe' address, and runs > > the exit_handler. > > > > The fprobe introduces two hash-tables, one is for entry callback which > > searches fprobes related to the given function address passed by entry > > callback. The other is for a return callback which checks if the given > > 'fprobe' data structure pointer is still valid. Note that it is > > possible to unregister fprobe before the return callback runs. Thus > > the address validation must be done before using it in the return > > callback. > > > > Download > > -------- > > This series can be applied against the ftrace/for-next branch in > > linux-trace tree. > > > > This series can also be found below branch. > > > > https://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git/log/?h=topic/fprobe-on-fgraph > > > > Thank you, > > > > --- > > hi, > I ran the kprobe_multi bench and I'm seeing lower throughput numbers > with new fprobe implementation > > base (bpf-next/master) > > root@amd:/home/jolsa/bpf-next/tools/testing/selftests/bpf# ./bench -w2 -d5 -a trig-kprobe-multi > Setting up benchmark 'trig-kprobe-multi'... > Benchmark 'trig-kprobe-multi' started. > Iter 0 ( 70.969us): hits 8.530M/s ( 8.530M/prod), drops 0.000M/s, total operations 8.530M/s > Iter 1 (-24.985us): hits 8.541M/s ( 8.541M/prod), drops 0.000M/s, total operations 8.541M/s > Iter 2 ( -2.976us): hits 8.537M/s ( 8.537M/prod), drops 0.000M/s, total operations 8.537M/s > Iter 3 ( -1.502us): hits 8.540M/s ( 8.540M/prod), drops 0.000M/s, total operations 8.540M/s > Iter 4 ( 0.392us): hits 8.540M/s ( 8.540M/prod), drops 0.000M/s, total operations 8.540M/s > Iter 5 ( 1.308us): hits 8.545M/s ( 8.545M/prod), drops 0.000M/s, total operations 8.545M/s > Iter 6 ( 2.783us): hits 8.530M/s ( 8.530M/prod), drops 0.000M/s, total operations 8.530M/s > Iter 7 ( -3.775us): hits 8.535M/s ( 8.535M/prod), drops 0.000M/s, total operations 8.535M/s > Summary: hits 8.538 ± 0.005M/s ( 8.538M/prod), drops 0.000 ± 0.000M/s, total operations 8.538 ± 0.005M/s > > your patchset on top of bpf-next/master > > root@amd:/home/jolsa/bpf-next/tools/testing/selftests/bpf# ./bench -w2 -d5 -a trig-kprobe-multi > Setting up benchmark 'trig-kprobe-multi'... > Benchmark 'trig-kprobe-multi' started. > Iter 0 ( 55.712us): hits 7.224M/s ( 7.224M/prod), drops 0.000M/s, total operations 7.224M/s > Iter 1 ( 15.226us): hits 7.222M/s ( 7.222M/prod), drops 0.000M/s, total operations 7.222M/s > Iter 2 ( -1.353us): hits 7.221M/s ( 7.221M/prod), drops 0.000M/s, total operations 7.221M/s > Iter 3 (-13.029us): hits 7.223M/s ( 7.223M/prod), drops 0.000M/s, total operations 7.223M/s > Iter 4 ( 9.111us): hits 7.222M/s ( 7.222M/prod), drops 0.000M/s, total operations 7.222M/s > Iter 5 ( -0.106us): hits 7.222M/s ( 7.222M/prod), drops 0.000M/s, total operations 7.222M/s > Iter 6 ( 0.734us): hits 7.221M/s ( 7.221M/prod), drops 0.000M/s, total operations 7.221M/s > Iter 7 (-10.233us): hits 7.220M/s ( 7.220M/prod), drops 0.000M/s, total operations 7.220M/s > Summary: hits 7.221 ± 0.001M/s ( 7.221M/prod), drops 0.000 ± 0.000M/s, total operations 7.221 ± 0.001M/s > > > I did perf profile (attached), but nothing stands out on the first look, Hm, we may need to compare both before and after, and need annotate in the code to find the bottleneck. Anyway, performance optimization can be done afterwards, so I'm not so worried it :) > I'll try to check on that later this week Thank you for your help! Thanks, > > jirka > > > --- > # To display the perf.data header info, please use --header/--header-only options. > # > # > # Total Lost Samples: 0 > # > # Samples: 42K of event 'cycles:k' > # Event count (approx.): 44580427937 > # > # Overhead Command Shared Object Symbol > # ........ ............... .................................................... ........................................................ > # > 17.47% bench [kernel.vmlinux] [k] find_kallsyms_symbol > | > ---find_kallsyms_symbol > module_address_lookup > kallsyms_lookup_buildid > kallsyms_lookup > print_rec > t_show > seq_read_iter > seq_read > vfs_read > ksys_read > __x64_sys_read > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > read > 0 > > 11.71% bench [kernel.vmlinux] [k] kprobe_multi_link_prog_run > | > ---kprobe_multi_link_prog_run > | > --11.56%--kprobe_multi_link_handler > fprobe_entry > function_graph_enter_regs > ftrace_graph_func > 0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 7.65% bench [kernel.vmlinux] [k] function_graph_enter_regs > | > ---function_graph_enter_regs > | > --7.65%--ftrace_graph_func > 0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 7.55% bench [kernel.vmlinux] [k] kprobe_multi_link_handler > | > ---kprobe_multi_link_handler > | > |--4.35%--function_graph_enter_regs > | ftrace_graph_func > | 0xffffffffc29850ab > | bpf_get_numa_node_id > | bpf_prog_d9703036495d54b0_trigger_driver > | __bpf_prog_test_run_raw_tp > | bpf_prog_test_run_raw_tp > | __sys_bpf > | __x64_sys_bpf > | x64_sys_call > | do_syscall_64 > | entry_SYSCALL_64 > | syscall > | bpf_prog_test_run_opts > | trigger_producer_batch > | 0x7a2a73a94ac3 > | > --3.20%--fprobe_entry > function_graph_enter_regs > ftrace_graph_func > 0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 7.50% bench [kernel.vmlinux] [k] fprobe_entry > | > ---fprobe_entry > | > --7.50%--function_graph_enter_regs > ftrace_graph_func > 0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 7.08% bench bpf_prog_7790468e40a289ea_bench_trigger_kprobe_multi [k] bpf_prog_7790468e40a289ea_bench_trigger_kprobe_multi > | > ---bpf_prog_7790468e40a289ea_bench_trigger_kprobe_multi > | > |--4.10%--kprobe_multi_link_handler > | fprobe_entry > | function_graph_enter_regs > | ftrace_graph_func > | 0xffffffffc29850ab > | bpf_get_numa_node_id > | bpf_prog_d9703036495d54b0_trigger_driver > | __bpf_prog_test_run_raw_tp > | bpf_prog_test_run_raw_tp > | __sys_bpf > | __x64_sys_bpf > | x64_sys_call > | do_syscall_64 > | entry_SYSCALL_64 > | syscall > | bpf_prog_test_run_opts > | trigger_producer_batch > | 0x7a2a73a94ac3 > | > --2.97%--kprobe_multi_link_prog_run > kprobe_multi_link_handler > fprobe_entry > function_graph_enter_regs > ftrace_graph_func > 0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 6.11% bench [kernel.vmlinux] [k] migrate_enable > | > ---migrate_enable > | > |--3.37%--kprobe_multi_link_handler > | fprobe_entry > | function_graph_enter_regs > | ftrace_graph_func > | 0xffffffffc29850ab > | bpf_get_numa_node_id > | bpf_prog_d9703036495d54b0_trigger_driver > | __bpf_prog_test_run_raw_tp > | bpf_prog_test_run_raw_tp > | __sys_bpf > | __x64_sys_bpf > | x64_sys_call > | do_syscall_64 > | entry_SYSCALL_64 > | syscall > | bpf_prog_test_run_opts > | trigger_producer_batch > | 0x7a2a73a94ac3 > | > --2.73%--kprobe_multi_link_prog_run > kprobe_multi_link_handler > fprobe_entry > function_graph_enter_regs > ftrace_graph_func > 0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 5.90% bench [kernel.vmlinux] [k] ftrace_graph_func > | > ---ftrace_graph_func > | > --5.75%--0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 4.03% bench [kernel.vmlinux] [k] __rcu_read_lock > | > ---__rcu_read_lock > | > --4.00%--kprobe_multi_link_handler > fprobe_entry > function_graph_enter_regs > ftrace_graph_func > 0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 3.61% bench [kernel.vmlinux] [k] bpf_get_numa_node_id > | > ---bpf_get_numa_node_id > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 2.99% bench [nf_conntrack] [k] 0x000000000002c0b3 > | > ---0xffffffffc29850b3 > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 2.69% bench bpf_prog_d9703036495d54b0_trigger_driver [k] bpf_prog_d9703036495d54b0_trigger_driver > | > ---bpf_prog_d9703036495d54b0_trigger_driver > | > --2.68%--__bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 2.04% bench [kernel.vmlinux] [k] t_start > | > ---t_start > seq_read_iter > seq_read > vfs_read > ksys_read > __x64_sys_read > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > read > 0 > > 1.90% bench [kernel.vmlinux] [k] srso_untrain_ret > | > ---srso_untrain_ret > | > |--0.68%--fprobe_entry > | function_graph_enter_regs > | ftrace_graph_func > | 0xffffffffc29850ab > | bpf_get_numa_node_id > | bpf_prog_d9703036495d54b0_trigger_driver > | __bpf_prog_test_run_raw_tp > | bpf_prog_test_run_raw_tp > | __sys_bpf > | __x64_sys_bpf > | x64_sys_call > | do_syscall_64 > | entry_SYSCALL_64 > | syscall > | bpf_prog_test_run_opts > | trigger_producer_batch > | 0x7a2a73a94ac3 > | > --0.54%--seq_read_iter > seq_read > vfs_read > ksys_read > __x64_sys_read > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > read > 0 > > 1.14% bench [kernel.vmlinux] [k] __rcu_read_unlock > | > ---__rcu_read_unlock > | > --1.12%--kprobe_multi_link_handler > fprobe_entry > function_graph_enter_regs > ftrace_graph_func > 0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 1.02% bench [kernel.vmlinux] [k] migrate_disable > | > ---migrate_disable > kprobe_multi_link_handler > fprobe_entry > function_graph_enter_regs > ftrace_graph_func > 0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 0.99% bench [kernel.vmlinux] [k] t_func_next.isra.0 > | > ---t_func_next.isra.0 > | > --0.83%--t_start > seq_read_iter > seq_read > vfs_read > ksys_read > __x64_sys_read > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > read > 0 > > 0.78% bench [nf_conntrack] [k] 0x000000000002c0d4 > | > ---0xffffffffc29850d4 > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 0.75% bench [nf_conntrack] [k] 0x000000000002c0ab > | > ---0xffffffffc29850ab > bpf_get_numa_node_id > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 0.68% bench [nf_conntrack] [k] 0x000000000002c00d > | > ---0xffffffffc298500d > bpf_prog_d9703036495d54b0_trigger_driver > __bpf_prog_test_run_raw_tp > bpf_prog_test_run_raw_tp > __sys_bpf > __x64_sys_bpf > x64_sys_call > do_syscall_64 > entry_SYSCALL_64 > syscall > bpf_prog_test_run_opts > trigger_producer_batch > 0x7a2a73a94ac3 > > 0.52% bench [kernel.vmlinux] [k] __x86_indirect_thunk_array > | > ---__x86_indirect_thunk_array > > 0.42% bench [nf_conntrack] [k] 0x000000000002c005 > 0.38% bench [kernel.vmlinux] [k] __sys_bpf > 0.37% bench [kernel.vmlinux] [k] srso_return_thunk > 0.24% bench [nf_conntrack] [k] 0x000000000002c001 > 0.23% bench [kernel.vmlinux] [k] read_hpet > 0.21% bench [kernel.vmlinux] [k] entry_SYSCALL_64 > 0.19% bench [kernel.vmlinux] [k] bpf_prog_test_run_raw_tp > 0.17% bench [nf_conntrack] [k] 0x000000000002c000 > 0.15% bench [kernel.vmlinux] [k] _copy_from_user > 0.15% bench [kernel.vmlinux] [k] syscall_exit_to_user_mode > 0.14% bench [nf_conntrack] [k] 0x000000000002c027 > 0.14% bench [nf_conntrack] [k] 0x000000000002c0ca > 0.13% swapper [kernel.vmlinux] [k] read_hpet > 0.12% bench [kernel.vmlinux] [k] _copy_to_user > 0.12% bench [nf_conntrack] [k] 0x000000000002c086 > 0.11% bench [nf_conntrack] [k] 0x000000000002c0c0 > 0.11% bench [kernel.vmlinux] [k] __fdget > 0.11% bench bench [.] bpf_prog_test_run_opts > 0.11% bench [kernel.vmlinux] [k] __bpf_prog_test_run_raw_tp > 0.10% bench [nf_conntrack] [k] 0x000000000002c059 > 0.09% bench [kernel.vmlinux] [k] get_symbol_offset > 0.09% bench [kernel.vmlinux] [k] do_syscall_64 > 0.08% bench [kernel.vmlinux] [k] __bpf_prog_get > 0.08% bench [kernel.vmlinux] [k] memchr_inv > 0.07% bench [kernel.vmlinux] [k] syscall_return_via_sysret > 0.06% bench [kernel.vmlinux] [k] x64_sys_call > 0.06% bench [kernel.vmlinux] [k] rep_movs_alternative > 0.06% bench [kernel.vmlinux] [k] ftrace_replace_code > 0.04% bench [kernel.vmlinux] [k] __x64_sys_bpf > 0.04% bench [kernel.vmlinux] [k] get_symbol_pos > 0.04% bench bench [.] sys_bpf > 0.04% bench [kernel.vmlinux] [k] vsnprintf > 0.04% bench [kernel.vmlinux] [k] number > 0.04% bench bench [.] ptr_to_u64 > 0.04% bench [kernel.vmlinux] [k] kfree > 0.03% bench [kernel.vmlinux] [k] seq_read_iter > 0.03% bench [kernel.vmlinux] [k] security_bpf > 0.03% bench libc.so.6 [.] syscall > 0.03% bench [kernel.vmlinux] [k] kallsyms_expand_symbol.constprop.0 > 0.03% kworker/2:2-eve [kernel.vmlinux] [k] memcpy_orig > 0.03% swapper [amdgpu] [k] 0x0000000000002f75 > 0.03% bench bench [.] trigger_producer_batch > 0.03% bench [kernel.vmlinux] [k] __bpf_prog_put > 0.02% bench [kernel.vmlinux] [k] seq_printf > 0.02% bench [kernel.vmlinux] [k] smp_call_function_many_cond > 0.02% kworker/2:2-eve [amdgpu] [k] 0x0000000000002f75 > 0.02% bench [kernel.vmlinux] [k] __check_object_size > 0.02% bench [kernel.vmlinux] [k] ftrace_test_record > 0.02% bench [kernel.vmlinux] [k] seq_write > 0.02% bench [kernel.vmlinux] [k] ftrace_lookup_ip > 0.02% bench [kernel.vmlinux] [k] t_show > 0.02% bench [kernel.vmlinux] [k] print_rec > 0.02% swapper [kernel.vmlinux] [k] io_idle > 0.01% bench [amdgpu] [k] 0x0000000000002f75 > 0.01% kworker/2:2-eve [kernel.vmlinux] [k] memcmp > 0.01% bench [kernel.vmlinux] [k] format_decode > 0.01% bench [kernel.vmlinux] [k] memcpy_orig > 0.01% bench [kernel.vmlinux] [k] __ftrace_hash_rec_update.part.0 > 0.01% bench [kernel.vmlinux] [k] mod_find > 0.01% swapper [kernel.vmlinux] [k] srso_untrain_ret > 0.01% bench [kernel.vmlinux] [k] kallsyms_lookup_buildid > 0.01% bench [kernel.vmlinux] [k] __seq_puts > 0.01% bench [kernel.vmlinux] [k] native_read_msr > 0.01% bench [kernel.vmlinux] [k] check_stack_object > 0.01% bench [kernel.vmlinux] [k] fput > 0.01% bench bench [.] libbpf_err_errno > 0.01% bench [kernel.vmlinux] [k] fpregs_assert_state_consistent > 0.01% bench [kernel.vmlinux] [k] ftrace_check_record > 0.01% bench [kernel.vmlinux] [k] t_next > 0.01% bench [kernel.vmlinux] [k] strlen > 0.01% swapper [kernel.vmlinux] [k] native_sched_clock > 0.01% bench [kernel.vmlinux] [k] __sysvec_apic_timer_interrupt > 0.01% bench libc.so.6 [.] 0x00000000001a0f10 > 0.01% bench [kernel.vmlinux] [k] native_write_msr > 0.01% bench [kernel.vmlinux] [k] __update_load_avg_se > 0.01% bench [kernel.vmlinux] [k] amd_pmu_addr_offset > 0.01% bench [kernel.vmlinux] [k] module_address_lookup > 0.01% bench [kernel.vmlinux] [k] ftrace_rec_iter_next > 0.01% bench [kernel.vmlinux] [k] string > 0.01% bench [kernel.vmlinux] [k] asm_exc_page_fault > 0.01% swapper [kernel.vmlinux] [k] __get_next_timer_interrupt > 0.01% bench [kernel.vmlinux] [k] clear_page_rep > 0.01% kworker/2:2-eve [kernel.vmlinux] [k] delay_halt_mwaitx > 0.01% bench [kernel.vmlinux] [k] __handle_mm_fault > 0.01% swapper [kernel.vmlinux] [k] ktime_get > 0.01% kworker/2:2-eve [kernel.vmlinux] [k] clear_page_rep > 0.01% swapper [kernel.vmlinux] [k] menu_select > 0.00% perf [kernel.vmlinux] [k] clear_page_rep > 0.00% bench [kernel.vmlinux] [k] ftrace_rec_iter_record > 0.00% bench [kernel.vmlinux] [k] account_process_tick > 0.00% bench [kernel.vmlinux] [k] __update_load_avg_cfs_rq > 0.00% bench [kernel.vmlinux] [k] update_curr > 0.00% bench [kernel.vmlinux] [k] do_sync_core > 0.00% bench [kernel.vmlinux] [k] kallsyms_lookup > 0.00% bench [kernel.vmlinux] [k] memset_orig > 0.00% kworker/6:1-eve [kernel.vmlinux] [k] memcpy_orig > 0.00% perf [kernel.vmlinux] [k] rep_movs_alternative > 0.00% sshd [kernel.vmlinux] [k] p4d_offset > 0.00% kworker/6:1-eve [amdgpu] [k] 0x0000000000002f75 > 0.00% bench [kernel.vmlinux] [k] __mod_memcg_lruvec_state > 0.00% swapper [kernel.vmlinux] [k] _raw_spin_unlock > 0.00% swapper [kernel.vmlinux] [k] cpuidle_enter_state > 0.00% swapper [kernel.vmlinux] [k] rb_erase > 0.00% containerd [kernel.vmlinux] [k] read_hpet > 0.00% kworker/6:1-eve [kernel.vmlinux] [k] memcmp > 0.00% swapper [kernel.vmlinux] [k] irq_entries_start > 0.00% swapper [kernel.vmlinux] [k] do_idle > 0.00% kworker/2:2-eve [kernel.vmlinux] [k] read_hpet > 0.00% swapper [kernel.vmlinux] [k] update_rq_clock > 0.00% swapper [kernel.vmlinux] [k] psi_group_change > 0.00% swapper [kernel.vmlinux] [k] sched_clock_cpu > 0.00% swapper [kernel.vmlinux] [k] do_sync_core > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064ec11 > 0.00% swapper [kernel.vmlinux] [k] _raw_spin_lock_irqsave > 0.00% swapper [kernel.vmlinux] [k] asm_sysvec_call_function_single > 0.00% swapper [kernel.vmlinux] [k] update_sd_lb_stats.constprop.0 > 0.00% perf [kernel.vmlinux] [k] __filemap_get_folio > 0.00% swapper [kernel.vmlinux] [k] irqentry_enter > 0.00% bench [kernel.vmlinux] [k] amd_pmu_test_overflow_topbit > 0.00% bench [amdgpu] [k] 0x0000000000124f4c > 0.00% bench [kernel.vmlinux] [k] update_rq_clock > 0.00% bench [kernel.vmlinux] [k] perf_event_task_tick > 0.00% bench [amdgpu] [k] 0x0000000000126f3a > 0.00% bench [kernel.vmlinux] [k] sched_balance_update_blocked_averages > 0.00% bench [kernel.vmlinux] [k] hrtimer_active > 0.00% bench [kernel.vmlinux] [k] _copy_to_iter > 0.00% bench [kernel.vmlinux] [k] _raw_spin_lock_irqsave > 0.00% bench [kernel.vmlinux] [k] calc_global_load_tick > 0.00% bench [kernel.vmlinux] [k] ftrace_shutdown.part.0 > 0.00% bench libc.so.6 [.] 0x00000000001a0fb5 > 0.00% bench [kernel.vmlinux] [k] task_tick_mm_cid > 0.00% bench libc.so.6 [.] 0x00000000001a0fba > 0.00% bench [kernel.vmlinux] [k] sched_clock_cpu > 0.00% bench [kernel.vmlinux] [k] update_cfs_group > 0.00% bench [kernel.vmlinux] [k] irq_exit_rcu > 0.00% bench [kernel.vmlinux] [k] native_sched_clock > 0.00% bench [kernel.vmlinux] [k] sched_tick > 0.00% bench [kernel.vmlinux] [k] sized_strscpy > 0.00% bench [kernel.vmlinux] [k] task_tick_fair > 0.00% bench [nf_conntrack] [k] 0x000000000002c036 > 0.00% bench [kernel.vmlinux] [k] perf_adjust_freq_unthr_context > 0.00% bench [kernel.vmlinux] [k] idle_cpu > 0.00% bench [kernel.vmlinux] [k] bpf_check_uarg_tail_zero > 0.00% swapper [kernel.vmlinux] [k] __hrtimer_next_event_base > 0.00% swapper [kernel.vmlinux] [k] native_write_msr > 0.00% bench [kernel.vmlinux] [k] __module_address.part.0 > 0.00% bench [kernel.vmlinux] [k] lru_gen_add_folio > 0.00% bench [kernel.vmlinux] [k] set_pte_range > 0.00% bench [kernel.vmlinux] [k] pfn_pte > 0.00% swapper [kernel.vmlinux] [k] error_entry > 0.00% swapper [kernel.vmlinux] [k] idle_cpu > 0.00% bench [kernel.vmlinux] [k] unmap_page_range > 0.00% bench [kernel.vmlinux] [k] __count_memcg_events > 0.00% bench [kernel.vmlinux] [k] __irq_exit_rcu > 0.00% perf [kernel.vmlinux] [k] __perf_event_enable > 0.00% bench [kernel.vmlinux] [k] seq_putc > 0.00% perf [kernel.vmlinux] [k] srso_untrain_ret > 0.00% swapper [kernel.vmlinux] [k] __irq_exit_rcu > 0.00% kworker/2:2-eve [kernel.vmlinux] [k] __kmalloc_cache_noprof > 0.00% bench libc.so.6 [.] 0x00000000001a0bd7 > 0.00% perf [kernel.vmlinux] [k] _raw_spin_lock > 0.00% kworker/u65:2-e [kernel.vmlinux] [k] __bio_split_to_limits > 0.00% bench [kernel.vmlinux] [k] do_anonymous_page > 0.00% perf [kernel.vmlinux] [k] __es_insert_extent > 0.00% bench [kernel.vmlinux] [k] __mem_cgroup_charge > 0.00% kworker/2:2-eve [kernel.vmlinux] [k] __wait_for_common > 0.00% bench [kernel.vmlinux] [k] __folio_throttle_swaprate > 0.00% swapper [kernel.vmlinux] [k] __switch_to_asm > 0.00% swapper [kernel.vmlinux] [k] add_interrupt_randomness > 0.00% kworker/2:2-eve [kernel.vmlinux] [k] srso_untrain_ret > 0.00% kworker/2:2-eve [kernel.vmlinux] [k] memset_orig > 0.00% kworker/u65:2-e [kernel.vmlinux] [k] percpu_counter_add_batch > 0.00% swapper [kernel.vmlinux] [k] tick_irq_enter > 0.00% swapper [kernel.vmlinux] [k] tick_check_broadcast_expired > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000045d28e > 0.00% perf [kernel.vmlinux] [k] __lruvec_stat_mod_folio > 0.00% bench [kernel.vmlinux] [k] kernfs_fop_read_iter > 0.00% bench [kernel.vmlinux] [k] __mod_zone_page_state > 0.00% kworker/2:2-eve [kernel.vmlinux] [k] delay_halt > 0.00% perf [kernel.vmlinux] [k] __mod_memcg_lruvec_state > 0.00% perf [kernel.vmlinux] [k] xas_load > 0.00% perf [kernel.vmlinux] [k] workingset_update_node > 0.00% perf [kernel.vmlinux] [k] _raw_spin_unlock > 0.00% swapper [kernel.vmlinux] [k] __handle_irq_event_percpu > 0.00% kworker/2:2-eve [kernel.vmlinux] [k] post_alloc_hook > 0.00% bench [kernel.vmlinux] [k] refill_obj_stock > 0.00% perf [kernel.vmlinux] [k] __alloc_pages_noprof > 0.00% swapper [kernel.vmlinux] [k] cpuidle_enter > 0.00% swapper [kernel.vmlinux] [k] sched_balance_domains > 0.00% perf [kernel.vmlinux] [k] __handle_mm_fault > 0.00% swapper [kernel.vmlinux] [k] ct_kernel_enter.constprop.0 > 0.00% bench [kernel.vmlinux] [k] mab_mas_cp > 0.00% swapper [amdgpu] [k] 0x00000000004619a7 > 0.00% swapper [kernel.vmlinux] [k] tick_nohz_irq_exit > 0.00% perf [kernel.vmlinux] [k] __memcg_slab_post_alloc_hook > 0.00% bench [kernel.vmlinux] [k] __alloc_pages_noprof > 0.00% swapper [kernel.vmlinux] [k] irq_exit_rcu > 0.00% bench [kernel.vmlinux] [k] memcmp > 0.00% swapper [kernel.vmlinux] [k] asm_sysvec_call_function > 0.00% perf [kernel.vmlinux] [k] perf_poll > 0.00% swapper [kernel.vmlinux] [k] enqueue_task_fair > 0.00% bench [kernel.vmlinux] [k] __rmqueue_pcplist > 0.00% kworker/u65:4-e [kernel.vmlinux] [k] __iommu_map > 0.00% kworker/6:1-eve [amdgpu] [k] 0x0000000000992cd2 > 0.00% kworker/6:1-eve [amdgpu] [k] 0x00000000004b2ade > 0.00% swapper [kernel.vmlinux] [k] wakeup_preempt > 0.00% migration/11 [kernel.vmlinux] [k] enqueue_task > 0.00% swapper [kernel.vmlinux] [k] hrtimer_update_next_event > 0.00% swapper [kernel.vmlinux] [k] native_apic_mem_eoi > 0.00% bench [kernel.vmlinux] [k] is_sync_callback_calling_insn > 0.00% kworker/6:1-eve [amdgpu] [k] 0x0000000000992cb3 > 0.00% systemd-network libc.so.6 [.] clock_gettime > 0.00% kworker/u65:4-e [kernel.vmlinux] [k] filemap_get_folios_tag > 0.00% kworker/u65:4-e [kernel.vmlinux] [k] fscrypt_mergeable_bio_bh > 0.00% swapper [kernel.vmlinux] [k] hrtimer_start_range_ns > 0.00% multipathd [kernel.vmlinux] [k] cpuacct_charge > 0.00% swapper [kernel.vmlinux] [k] hrtimer_get_next_event > 0.00% kworker/12:1-ev [amdgpu] [k] 0x00000000004b09fc > 0.00% kworker/12:1-ev [kernel.vmlinux] [k] memcpy_orig > 0.00% kworker/u65:4-e [kernel.vmlinux] [k] wbt_track > 0.00% kworker/u65:4-e [kernel.vmlinux] [k] __lruvec_stat_mod_folio > 0.00% kworker/12:1-ev [amdgpu] [k] 0x0000000000002f75 > 0.00% bench [kernel.vmlinux] [k] do_jit > 0.00% kworker/u65:4-e [kernel.vmlinux] [k] __folio_start_writeback > 0.00% bench [kernel.vmlinux] [k] uncharge_folio > 0.00% perf [kernel.vmlinux] [k] do_fault > 0.00% perf [kernel.vmlinux] [k] __kmalloc_noprof > 0.00% containerd [kernel.vmlinux] [k] blkcg_maybe_throttle_current > 0.00% kworker/12:1-ev [amdgpu] [k] 0x0000000000523901 > 0.00% kworker/12:1-ev [amdgpu] [k] 0x0000000000538414 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000465f98 > 0.00% irqbalance [kernel.vmlinux] [k] format_decode > 0.00% swapper [kernel.vmlinux] [k] handle_edge_irq > 0.00% kworker/12:1-ev [kernel.vmlinux] [k] srso_untrain_ret > 0.00% kworker/u65:4-e [kernel.vmlinux] [k] __find_get_block > 0.00% systemd-journal [kernel.vmlinux] [k] __put_user_nocheck_4 > 0.00% systemd-journal libsystemd-shared-249.so [.] 0x00000000001c9c3e > 0.00% swapper [kernel.vmlinux] [k] local_touch_nmi > 0.00% bench [kernel.vmlinux] [k] __mem_cgroup_uncharge_folios > 0.00% swapper [kernel.vmlinux] [k] _raw_spin_lock > 0.00% swapper [kernel.vmlinux] [k] select_task_rq_fair > 0.00% swapper [kernel.vmlinux] [k] need_update > 0.00% bench [kernel.vmlinux] [k] rcu_segcblist_enqueue > 0.00% swapper [kernel.vmlinux] [k] tick_nohz_get_sleep_length > 0.00% containerd [kernel.vmlinux] [k] _raw_spin_unlock > 0.00% containerd [kernel.vmlinux] [k] ep_autoremove_wake_function > 0.00% kworker/u66:1-e [kernel.vmlinux] [k] __percpu_counter_sum > 0.00% swapper [kernel.vmlinux] [k] ktime_get_mono_fast_ns > 0.00% swapper [kernel.vmlinux] [k] sysvec_apic_timer_interrupt > 0.00% swapper [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore > 0.00% migration/10 [kernel.vmlinux] [k] dequeue_entity > 0.00% swapper [kernel.vmlinux] [k] handle_irq_event > 0.00% perf [kernel.vmlinux] [k] srso_return_thunk > 0.00% perf [kernel.vmlinux] [k] xas_find_conflict > 0.00% perf-exec [kernel.vmlinux] [k] mas_next_slot > 0.00% swapper [kernel.vmlinux] [k] fetch_next_timer_interrupt.constprop.0 > 0.00% bench [kernel.vmlinux] [k] get_page_from_freelist > 0.00% swapper [amdgpu] [k] 0x0000000000125079 > 0.00% kworker/u65:4-e [kernel.vmlinux] [k] sched_balance_newidle > 0.00% migration/6 [kernel.vmlinux] [k] raw_spin_rq_unlock > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000444a05 > 0.00% swapper [kernel.vmlinux] [k] fetch_pte > 0.00% perf [kernel.vmlinux] [k] __pte_offset_map > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000534985 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000003ac5 > 0.00% bench [kernel.vmlinux] [k] perf_iterate_sb > 0.00% swapper [kernel.vmlinux] [k] get_cpu_device > 0.00% perf [kernel.vmlinux] [k] xas_start > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000098fea5 > 0.00% perf [kernel.vmlinux] [k] mark_buffer_dirty > 0.00% containerd containerd [.] runtime.cleantimers > 0.00% bench [kernel.vmlinux] [k] __cond_resched > 0.00% kworker/2:2-eve [drm_kms_helper] [k] 0x00000000000008eb > 0.00% kworker/2:2-eve [raid6_pq] [k] 0x000000000000c24b > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064ea79 > 0.00% perf [kernel.vmlinux] [k] cpu_util > 0.00% swapper [kernel.vmlinux] [k] asm_common_interrupt > 0.00% perf [kernel.vmlinux] [k] __filemap_add_folio > 0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000007d1eab > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051a45f > 0.00% perf [kernel.vmlinux] [k] ext4_es_lookup_extent > 0.00% swapper [kernel.vmlinux] [k] __flush_smp_call_function_queue > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000519af6 > 0.00% swapper [kernel.vmlinux] [k] refresh_cpu_vm_stats > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000477759 > 0.00% perf-exec [kernel.vmlinux] [k] kmem_cache_free > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000539303 > 0.00% kworker/2:2-eve [drm] [k] 0x0000000000020c1c > 0.00% swapper [kernel.vmlinux] [k] asm_sysvec_apic_timer_interrupt > 0.00% bench [kernel.vmlinux] [k] alloc_pages_mpol_noprof > 0.00% bench [kernel.vmlinux] [k] filemap_map_pages > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000477767 > 0.00% swapper [kernel.vmlinux] [k] update_load_avg > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000045d3d6 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051d905 > 0.00% kworker/2:2-eve [drm_kms_helper] [k] 0x0000000000004ae8 > 0.00% kworker/u65:2-e [kernel.vmlinux] [k] __mod_memcg_lruvec_state > 0.00% kworker/u65:2-e [kernel.vmlinux] [k] bio_associate_blkg > 0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004916dd > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000465ecb > 0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004b780a > 0.00% swapper [kernel.vmlinux] [k] sysvec_call_function > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051e764 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064ed72 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051c7bf > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064ebe0 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000523fba > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000832660 > 0.00% swapper [kernel.vmlinux] [k] acpi_idle_do_entry > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000097ddfc > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000045d334 > 0.00% kworker/u65:2-e [kernel.vmlinux] [k] soft_cursor > 0.00% perf [kernel.vmlinux] [k] sched_balance_rq > 0.00% kworker/u65:2-e [kernel.vmlinux] [k] __fprop_add_percpu > 0.00% swapper [kernel.vmlinux] [k] rb_next > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000977ece > 0.00% perf [kernel.vmlinux] [k] __folio_mark_dirty > 0.00% perf [kernel.vmlinux] [k] filemap_alloc_folio_noprof > 0.00% perf [kernel.vmlinux] [k] get_mem_cgroup_from_mm > 0.00% perf [kernel.vmlinux] [k] ext4_da_reserve_space > 0.00% perf [kernel.vmlinux] [k] read_hpet > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051cd3e > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000532f0f > 0.00% kworker/2:2-eve [kernel.vmlinux] [k] __mod_node_page_state > 0.00% systemd-journal [kernel.vmlinux] [k] mod_objcg_state > 0.00% jbd2/dm-0-8 [kernel.vmlinux] [k] dm_submit_bio > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000517b6d > 0.00% kcompactd0 [kernel.vmlinux] [k] idle_cpu > 0.00% kworker/2:2-eve [kernel.vmlinux] [k] free_tail_page_prepare > 0.00% swapper [amdgpu] [k] 0x00000000002f7f65 > 0.00% swapper [kernel.vmlinux] [k] tick_program_event > 0.00% containerd [kernel.vmlinux] [k] psi_group_change > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000053776f > 0.00% swapper [kernel.vmlinux] [k] srso_return_thunk > 0.00% perf [kernel.vmlinux] [k] ext4_claim_free_clusters > 0.00% perf [kernel.vmlinux] [k] generic_perform_write > 0.00% perf [kernel.vmlinux] [k] mem_cgroup_commit_charge > 0.00% perf [kernel.vmlinux] [k] file_modified > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051e779 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000053974c > 0.00% perf [kernel.vmlinux] [k] ext4_da_write_begin > 0.00% perf [kernel.vmlinux] [k] filemap_add_folio > 0.00% perf libc.so.6 [.] write > 0.00% perf [kernel.vmlinux] [k] __es_remove_extent > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000052244a > 0.00% swapper [kernel.vmlinux] [k] ct_nmi_exit > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000034bee5 > 0.00% kworker/2:2-eve [kernel.vmlinux] [k] __rcu_read_lock > 0.00% perf [kernel.vmlinux] [k] __dquot_alloc_space > 0.00% swapper [kernel.vmlinux] [k] native_read_msr > 0.00% perf [kernel.vmlinux] [k] folio_unlock > 0.00% perf [kernel.vmlinux] [k] radix_tree_node_ctor > 0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000006c5b82 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000527259 > 0.00% perf [kernel.vmlinux] [k] __mod_node_page_state > 0.00% perf [kernel.vmlinux] [k] __radix_tree_lookup > 0.00% kworker/u65:2-e [kernel.vmlinux] [k] __rcu_read_unlock > 0.00% perf [kernel.vmlinux] [k] ext4_get_reserved_space > 0.00% swapper [kernel.vmlinux] [k] hrtimer_cancel > 0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004ba8ce > 0.00% swapper [amdgpu] [k] 0x0000000000126f3a > 0.00% perf perf [.] perf_cpu_map__idx > 0.00% swapper [kernel.vmlinux] [k] sched_balance_update_blocked_averages > 0.00% systemd-journal [kernel.vmlinux] [k] do_timerfd_settime > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000097ddc5 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000097e5a4 > 0.00% perf [kernel.vmlinux] [k] folio_batch_move_lru > 0.00% swapper [amdgpu] [k] 0x000000000046f8a5 > 0.00% perf [kernel.vmlinux] [k] xas_nomem > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051d24c > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000097df7c > 0.00% tuned [kernel.vmlinux] [k] psi_task_switch > 0.00% swapper [kernel.vmlinux] [k] __dequeue_entity > 0.00% swapper [amdgpu] [k] 0x00000000006e7c5d > 0.00% swapper [kernel.vmlinux] [k] rcu_preempt_deferred_qs > 0.00% kworker/7:1-eve [kernel.vmlinux] [k] __free_one_page > 0.00% swapper [kernel.vmlinux] [k] memchr_inv > 0.00% perf [kernel.vmlinux] [k] psi_task_switch > 0.00% swapper [kernel.vmlinux] [k] _nohz_idle_balance.isra.0 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000009a6fbd > 0.00% swapper [kernel.vmlinux] [k] acpi_idle_enter > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000456a9f > 0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004ac642 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000477888 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000529a52 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004b36d2 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000525b33 > 0.00% swapper [kernel.vmlinux] [k] tmigr_active_up > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000452444 > 0.00% swapper [amdgpu] [k] 0x000000000046f865 > 0.00% swapper [kernel.vmlinux] [k] next_timer_interrupt > 0.00% kworker/6:1-eve [kernel.vmlinux] [k] __free_pages_ok > 0.00% perf [kernel.vmlinux] [k] exc_page_fault > 0.00% swapper [kernel.vmlinux] [k] __schedule > 0.00% irqbalance [kernel.vmlinux] [k] __kmalloc_cache_noprof > 0.00% swapper [kernel.vmlinux] [k] cpuidle_not_available > 0.00% containerd [kernel.vmlinux] [k] pick_next_task_fair > 0.00% kworker/6:1-eve [amdgpu] [k] 0x00000000009a70b7 > 0.00% swapper [kernel.vmlinux] [k] tick_nohz_next_event > 0.00% swapper [kernel.vmlinux] [k] tick_check_oneshot_broadcast_this_cpu > 0.00% swapper [kernel.vmlinux] [k] irq_chip_ack_parent > 0.00% irqbalance [kernel.vmlinux] [k] show_stat > 0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000004b8295 > 0.00% swapper [amdgpu] [k] 0x000000000045d385 > 0.00% swapper [kernel.vmlinux] [k] tick_nohz_idle_stop_tick > 0.00% containerd [kernel.vmlinux] [k] __schedule > 0.00% swapper [kernel.vmlinux] [k] switch_mm_irqs_off > 0.00% swapper [amdgpu] [k] 0x0000000000461a21 > 0.00% kworker/6:1-eve [amdgpu] [k] 0x000000000051d7c2 > 0.00% swapper [kernel.vmlinux] [k] try_to_wake_up > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000051e7cf > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064ed4e > 0.00% perf [kernel.vmlinux] [k] perf_mmap_to_page > 0.00% migration/7 [kernel.vmlinux] [k] update_sd_lb_stats.constprop.0 > 0.00% irqbalance [kernel.vmlinux] [k] __rcu_read_lock > 0.00% migration/3 [kernel.vmlinux] [k] __update_load_avg_cfs_rq > 0.00% swapper [br_netfilter] [k] 0x0000000000000775 > 0.00% bench [kernel.vmlinux] [k] remove_vma > 0.00% swapper [kernel.vmlinux] [k] sched_clock_noinstr > 0.00% kworker/6:1-eve [amdgpu] [k] 0x0000000000529169 > 0.00% swapper [amdgpu] [k] 0x000000000000df07 > 0.00% swapper [kernel.vmlinux] [k] __switch_to > 0.00% swapper [kernel.vmlinux] [k] place_entity > 0.00% kworker/6:1-eve [amdgpu] [k] 0x000000000097dd9a > 0.00% swapper [amdgpu] [k] 0x00000000006e7c5a > 0.00% irqbalance [kernel.vmlinux] [k] __rcu_read_unlock > 0.00% migration/6 [kernel.vmlinux] [k] enqueue_entity > 0.00% swapper [kernel.vmlinux] [k] __update_load_avg_cfs_rq > 0.00% perf [kernel.vmlinux] [k] event_function > 0.00% perf [kernel.vmlinux] [k] do_user_addr_fault > 0.00% kworker/2:2-eve [amdgpu] [k] 0x000000000064eaee > 0.00% migration/4 [kernel.vmlinux] [k] __update_load_avg_cfs_rq > 0.00% swapper [amdgpu] [k] 0x00000000006e79ff > 0.00% containerd [kernel.vmlinux] [k] schedule_hrtimeout_range_clock > 0.00% irqbalance [kernel.vmlinux] [k] security_file_open > 0.00% perf [kernel.vmlinux] [k] _find_next_and_bit > 0.00% multipathd [kernel.vmlinux] [k] __rseq_handle_notify_resume > 0.00% swapper [kernel.vmlinux] [k] __x86_indirect_thunk_r12 > 0.00% swapper [kernel.vmlinux] [k] rcu_note_context_switch > 0.00% bench [kernel.vmlinux] [k] free_unref_folios > 0.00% migration/0 [kernel.vmlinux] [k] migration_cpu_stop > 0.00% migration/1 [kernel.vmlinux] [k] rcu_note_context_switch > 0.00% kworker/u66:1-e [kernel.vmlinux] [k] n_tty_receive_buf2 > 0.00% swapper [kernel.vmlinux] [k] __common_interrupt > 0.00% swapper [drm] [k] 0x000000000003d658 > 0.00% migration/5 [kernel.vmlinux] [k] psi_group_change > 0.00% swapper [kernel.vmlinux] [k] clockevents_program_event > 0.00% swapper [kernel.vmlinux] [k] __x86_indirect_thunk_array > 0.00% perf [kernel.vmlinux] [k] set_pte_range > 0.00% perf [kernel.vmlinux] [k] __schedule > 0.00% swapper [kernel.vmlinux] [k] __update_load_avg_se > 0.00% swapper [kernel.vmlinux] [k] asm_sysvec_reschedule_ipi > 0.00% swapper [kernel.vmlinux] [k] local_clock_noinstr > 0.00% swapper [kernel.vmlinux] [k] tick_nohz_stop_idle > 0.00% swapper [amdgpu] [k] 0x000000000044a039 > 0.00% kworker/4:1-mm_ [kernel.vmlinux] [k] native_queued_spin_lock_slowpath > 0.00% rcu_preempt [kernel.vmlinux] [k] __update_idle_core > 0.00% rcu_preempt [kernel.vmlinux] [k] pick_next_task_fair > 0.00% rcu_preempt [kernel.vmlinux] [k] prepare_to_swait_event > 0.00% rcu_preempt [kernel.vmlinux] [k] sched_balance_rq > 0.00% rcu_preempt [kernel.vmlinux] [k] update_rq_clock > 0.00% rs:main Q:Reg [kernel.vmlinux] [k] ext4_inode_csum > 0.00% swapper [kernel.vmlinux] [k] tmigr_inactive_up > 0.00% swapper [kernel.vmlinux] [k] hrtimer_forward > 0.00% swapper [kernel.vmlinux] [k] cpuidle_reflect > 0.00% perf [kernel.vmlinux] [k] percpu_counter_add_batch > 0.00% swapper [amdgpu] [k] 0x000000000031e745 > 0.00% swapper [kernel.vmlinux] [k] sched_idle_set_state > 0.00% kworker/6:1-eve [amdgpu] [k] 0x000000000045675c > 0.00% kworker/6:1-eve [amdgpu] [k] 0x000000000051cbb1 > 0.00% swapper [amdgpu] [k] 0x000000000002dce5 > 0.00% systemd-journal [kernel.vmlinux] [k] __ext4_ext_check > 0.00% swapper [kernel.vmlinux] [k] sched_core_idle_cpu > 0.00% swapper [drm] [k] 0x000000000003de52 > 0.00% swapper [kernel.vmlinux] [k] __msecs_to_jiffies > 0.00% swapper [amdgpu] [k] 0x000000000046f9c4 > 0.00% swapper [kernel.vmlinux] [k] call_cpuidle > 0.00% swapper [kernel.vmlinux] [k] cpuidle_governor_latency_req > 0.00% kworker/7:1-eve [kernel.vmlinux] [k] __switch_to_asm > 0.00% dockerd [kernel.vmlinux] [k] apparmor_file_permission > 0.00% swapper [amdgpu] [k] 0x0000000000126f37 > 0.00% swapper [amdgpu] [k] 0x00000000003218b6 > 0.00% swapper [amdgpu] [k] 0x00000000006e8d65 > 0.00% swapper [amdgpu] [k] 0x000000000096f62d > 0.00% systemd-journal [kernel.vmlinux] [k] __es_tree_search.isra.0 > 0.00% swapper [kernel.vmlinux] [k] update_group_capacity > 0.00% kworker/2:2-eve [amdgpu] [k] 0x00000000005225c3 > 0.00% kworker/2:2-eve [amdgpu] [k] 0x0000000000990c4f > 0.00% swapper [kernel.vmlinux] [k] timer_base_try_to_set_idle > 0.00% swapper [kernel.vmlinux] [k] ct_idle_exit > 0.00% swapper [kernel.vmlinux] [k] __x86_indirect_thunk_rbx > 0.00% swapper [kernel.vmlinux] [k] fast_mix > 0.00% swapper [kernel.vmlinux] [k] sched_clock_idle_sleep_event > 0.00% swapper [kernel.vmlinux] [k] touch_softlockup_watchdog_sched > 0.00% containerd [kernel.vmlinux] [k] psi_task_switch > 0.00% swapper [amdgpu] [k] 0x0000000000123820 > 0.00% swapper [kernel.vmlinux] [k] sched_clock > 0.00% swapper [amdgpu] [k] 0x00000000002f46a5 > 0.00% swapper [amdgpu] [k] 0x0000000000449ff5 > 0.00% containerd [kernel.vmlinux] [k] _copy_to_user > 0.00% swapper [kernel.vmlinux] [k] enqueue_task > 0.00% swapper [amdgpu] [k] 0x000000000096f625 > 0.00% swapper [amdgpu] [k] 0x00000000006e7c66 > 0.00% swapper [kernel.vmlinux] [k] hrtimer_interrupt > 0.00% swapper [kernel.vmlinux] [k] irq_enter_rcu > 0.00% perf [kernel.vmlinux] [k] _raw_spin_lock_irqsave > 0.00% perf [kernel.vmlinux] [k] poll_freewait > 0.00% rcu_preempt [kernel.vmlinux] [k] __rcu_read_lock > 0.00% swapper [kernel.vmlinux] [k] wake_q_add > 0.00% kworker/7:1H-kb [kernel.vmlinux] [k] __switch_to_asm > 0.00% swapper [amdgpu] [k] 0x00000000002f8058 > 0.00% swapper [kernel.vmlinux] [k] sched_clock_tick > 0.00% swapper [kernel.vmlinux] [k] slab_update_freelist.constprop.0.isra.0 > 0.00% swapper [kernel.vmlinux] [k] tick_nohz_get_next_hrtimer > 0.00% swapper [kernel.vmlinux] [k] ct_kernel_enter_state > 0.00% swapper [kernel.vmlinux] [k] hrtimer_next_event_without > 0.00% swapper [kernel.vmlinux] [k] resched_curr > 0.00% swapper [amdgpu] [k] 0x0000000000003a55 > 0.00% swapper [amdgpu] [k] 0x000000000045d3df > 0.00% swapper [amdgpu] [k] 0x000000000098aeb9 > 0.00% containerd [kernel.vmlinux] [k] __switch_to > 0.00% swapper [kernel.vmlinux] [k] rcu_sched_clock_irq > 0.00% kworker/9:1-eve [kernel.vmlinux] [k] sched_clock_cpu > 0.00% swapper [amdgpu] [k] 0x0000000000124ef5 > 0.00% containerd [kernel.vmlinux] [k] update_curr > 0.00% swapper [drm] [k] 0x0000000000035f4e > 0.00% swapper [kernel.vmlinux] [k] tmigr_update_events > 0.00% swapper [kernel.vmlinux] [k] profile_pc > 0.00% swapper [nvme] [k] 0x0000000000004315 > 0.00% swapper [kernel.vmlinux] [k] raw_spin_rq_unlock > 0.00% containerd [kernel.vmlinux] [k] posix_get_monotonic_timespec > 0.00% swapper [amdgpu] [k] 0x0000000000002f4b > 0.00% swapper [amdgpu] [k] 0x000000000034bf25 > 0.00% perf [kernel.vmlinux] [k] get_page_from_freelist > 0.00% rcu_preempt [kernel.vmlinux] [k] __mod_timer > 0.00% perf [kernel.vmlinux] [k] mutex_unlock > 0.00% swapper [amdgpu] [k] 0x0000000000460f85 > 0.00% swapper [amdgpu] [k] 0x0000000000977ec5 > 0.00% swapper [amdgpu] [k] 0x0000000000977ed6 > 0.00% swapper [kernel.vmlinux] [k] sched_clock_idle_wakeup_event > 0.00% swapper [amdgpu] [k] 0x0000000000124016 > 0.00% swapper [kernel.vmlinux] [k] pick_next_task_fair > 0.00% swapper [raid6_pq] [k] 0x000000000001acf5 > 0.00% swapper [amdgpu] [k] 0x0000000000977ec6 > 0.00% systemd-journal [kernel.vmlinux] [k] security_inode_setattr > 0.00% swapper [kernel.vmlinux] [k] __sysvec_apic_timer_interrupt > 0.00% swapper [amdgpu] [k] 0x0000000000461985 > 0.00% swapper [amdgpu] [k] 0x0000000000123825 > 0.00% swapper [amdgpu] [k] 0x0000000000126ee9 > 0.00% swapper [amdgpu] [k] 0x000000000096f6cd > 0.00% perf [kernel.vmlinux] [k] do_syscall_64 > 0.00% systemd-journal [kernel.vmlinux] [k] unmap_mapping_range > 0.00% systemd-journal libc.so.6 [.] 0x0000000000090a64 > 0.00% containerd [kernel.vmlinux] [k] dequeue_entity > 0.00% systemd-journal [kernel.vmlinux] [k] dax_layout_busy_page > 0.00% swapper [kernel.vmlinux] [k] apic_ack_irq > 0.00% swapper [drm] [k] 0x000000000003d83b > 0.00% containerd [kernel.vmlinux] [k] __futex_queue > 0.00% swapper [kernel.vmlinux] [k] complete_all > 0.00% swapper [amdgpu] [k] 0x000000000098b0d6 > 0.00% swapper [kernel.vmlinux] [k] can_stop_idle_tick > 0.00% swapper [amdgpu] [k] 0x000000000000dee5 > 0.00% swapper [amdgpu] [k] 0x000000000044ecef > 0.00% swapper [amdgpu] [k] 0x000000000000dee6 > 0.00% migration/14 [kernel.vmlinux] [k] update_curr_common > 0.00% rs:main Q:Reg [kernel.vmlinux] [k] ext4_get_group_desc > 0.00% swapper [kernel.vmlinux] [k] tick_do_update_jiffies64 > 0.00% swapper [kernel.vmlinux] [k] tmigr_cpu_new_timer > 0.00% kworker/7:1-eve [kernel.vmlinux] [k] free_pcppages_bulk > 0.00% dockerd [kernel.vmlinux] [k] srso_untrain_ret > 0.00% dockerd [kernel.vmlinux] [k] update_curr > 0.00% swapper [amdgpu] [k] 0x0000000000124f0c > 0.00% swapper [drm] [k] 0x000000000003f455 > 0.00% swapper [amdgpu] [k] 0x00000000003218b5 > 0.00% swapper [kernel.vmlinux] [k] ct_irq_enter > 0.00% swapper [kernel.vmlinux] [k] ct_irq_exit > 0.00% swapper [kernel.vmlinux] [k] blk_stat_add > 0.00% swapper [amdgpu] [k] 0x00000000006e8dd0 > 0.00% swapper [amdgpu] [k] 0x0000000000124f4c > 0.00% swapper [amdgpu] [k] 0x000000000098aef6 > 0.00% swapper [drm] [k] 0x000000000003f468 > 0.00% swapper [kernel.vmlinux] [k] nohz_balance_enter_idle > 0.00% swapper [kernel.vmlinux] [k] __wake_up_common > 0.00% swapper [amdgpu] [k] 0x00000000001240ea > 0.00% swapper [kernel.vmlinux] [k] __wake_up > 0.00% swapper [amdgpu] [k] 0x00000000002f7f81 > 0.00% swapper [amdgpu] [k] 0x000000000046f97f > 0.00% swapper [drm] [k] 0x000000000003f45c > 0.00% swapper [kernel.vmlinux] [k] __mod_memcg_lruvec_state > 0.00% swapper [kernel.vmlinux] [k] mempool_free_slab > 0.00% swapper [amdgpu] [k] 0x0000000000977ef0 > 0.00% swapper [kernel.vmlinux] [k] x86_pmu_disable_all > 0.00% swapper [amdgpu] [k] 0x00000000002f7f71 > 0.00% kworker/3:1H-kb [kernel.vmlinux] [k] read_hpet > 0.00% swapper [kernel.vmlinux] [k] psi_task_change > 0.00% swapper [kernel.vmlinux] [k] perf_adjust_freq_unthr_context > 0.00% swapper [kernel.vmlinux] [k] account_idle_ticks > 0.00% swapper [amdgpu] [k] 0x000000000045d365 > 0.00% swapper [kernel.vmlinux] [k] ttwu_do_activate > 0.00% swapper [raid6_pq] [k] 0x000000000001acc5 > 0.00% swapper [kernel.vmlinux] [k] enqueue_entity > 0.00% wpa_supplicant [kernel.vmlinux] [k] read_hpet > 0.00% swapper [amdgpu] [k] 0x0000000000012205 > 0.00% swapper [drm] [k] 0x000000000003d649 > 0.00% containerd [kernel.vmlinux] [k] __update_load_avg_se > 0.00% rs:main Q:Reg [kernel.vmlinux] [k] futex_wake > 0.00% swapper [kernel.vmlinux] [k] __x86_indirect_thunk_r13 > 0.00% swapper [kernel.vmlinux] [k] sched_balance_softirq > 0.00% swapper [kernel.vmlinux] [k] timerqueue_del > 0.00% swapper [kernel.vmlinux] [k] cpu_util > > > # > # (Tip: Limit to show entries above 5% only: perf report --percent-limit 5) > # -- Masami Hiramatsu (Google) <mhiramat@xxxxxxxxxx>
