hi, we got report of kernel crash [1][3] within bpf_event_output helper. The reason seems to be the nesting protection code in bpf_event_output that expects disabled preemption, which is not guaranteed for programs executed by bpf_prog_run_array_cg. I managed to reproduce on tracing side where we have the same problem in bpf_perf_event_output. The reproducer [2] just creates busy uprobe and kprobe which call bpf_perf_event_output helper a lot. Eventually the code will be preempted within nesting protection and cause crash like: kernel tried to execute NX-protected page - exploit attempt? (uid: 0) BUG: unable to handle page fault for address: ffffffff82be3eea ... Call Trace: <TASK> ? __die+0x1f/0x70 ? page_fault_oops+0x176/0x4d0 ? exc_page_fault+0x132/0x230 ? asm_exc_page_fault+0x22/0x30 ? perf_output_sample+0x12b/0x910 ? perf_event_output+0xd0/0x1d0 ? bpf_perf_event_output+0x162/0x1d0 ? bpf_prog_c6271286d9a4c938_krava1+0x76/0x87 ? __uprobe_perf_func+0x12b/0x540 ? uprobe_dispatcher+0x2c4/0x430 ? uprobe_notify_resume+0x2da/0xce0 ? atomic_notifier_call_chain+0x7b/0x110 ? exit_to_user_mode_prepare+0x13e/0x290 ? irqentry_exit_to_user_mode+0x5/0x30 ? asm_exc_int3+0x35/0x40 I did not come up with reproducer for bpf_event_output case and have no way to reproduce or test it so far, hence I'm sending patch 2 as RFC for discussion. However it seems to me it suffers the same issue as bpf_perf_event_output on tracing side. thanks, jirka [1] https://github.com/cilium/cilium/issues/26756 [2] https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git/commit/?h=bpf_output_fix_reproducer&id=8054dcc634121b884c7c331329d61d93351d03b5 [3] slack: [66194.378161] BUG: kernel NULL pointer dereference, address: 0000000000000001 [66194.378324] #PF: supervisor instruction fetch in kernel mode [66194.378447] #PF: error_code(0x0010) - not-present page ... [66194.378692] Oops: 0010 [#1] PREEMPT SMP NOPTI ... [66194.380666] <TASK> [66194.380775] ? perf_output_sample+0x12a/0x9a0 [66194.380902] ? finish_task_switch.isra.0+0x81/0x280 [66194.381024] ? perf_event_output+0x66/0xa0 [66194.381148] ? bpf_event_output+0x13a/0x190 [66194.381270] ? bpf_event_output_data+0x22/0x40 [66194.381391] ? bpf_prog_dfc84bbde731b257_cil_sock4_connect+0x40a/0xacb [66194.381519] ? xa_load+0x87/0xe0 [66194.381635] ? __cgroup_bpf_run_filter_sock_addr+0xc1/0x1a0 [66194.381759] ? release_sock+0x3e/0x90 [66194.381876] ? sk_setsockopt+0x1a1/0x12f0 [66194.381996] ? udp_pre_connect+0x36/0x50 [66194.382114] ? inet_dgram_connect+0x93/0xa0 [66194.382233] ? __sys_connect+0xb4/0xe0 [66194.382353] ? udp_setsockopt+0x27/0x40 [66194.382470] ? __pfx_udp_push_pending_frames+0x10/0x10 [66194.382593] ? __sys_setsockopt+0xdf/0x1a0 [66194.382713] ? __x64_sys_connect+0xf/0x20 [66194.382832] ? do_syscall_64+0x3a/0x90 [66194.382949] ? entry_SYSCALL_64_after_hwframe+0x72/0xdc [66194.383077] </TASK> --- Jiri Olsa (2): bpf: Disable preemption in bpf_perf_event_output bpf: Disable preemption in bpf_event_output kernel/trace/bpf_trace.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
