Hi,
A recent patch [1] to 'fs' broke the TX TLS device-offloaded flow
starting from v6.11-rc1.
The kernel crashes. Different runs result in different kernel traces.
See below [2].
All of them disappear once patch [1] is reverted.
The issues appears only with "sendfile on and zerocopy on".
We couldn't repro with "sendfile off", or with "sendfile on and zerocopy
off".
The repro test is as simple as a repeated client/server communication
(wrk/nginx), with sendfile on and zc on, and with "tls-hw-tx-offload: on".
$ for i in `seq 10`; do wrk -b::2:2:2:3 -t10 -c100 -d15 --timeout 5s
https://[::2:2:2:2]:20448/16000b.img; done
We can provide more details if needed, to help with the analysis and debug.
Regards,
Tariq
[1]
commit 49b29a573da83b65d5f4ecf2db6619bab7aa910c
Author: Christoph Hellwig <hch@xxxxxx>
Date: Mon May 27 18:36:09 2024 +0200
nfs: add support for large folios
NFS already is void of folio size assumption, so just pass the
chunk size
to __filemap_get_folio and set the large folio address_space flag
for all
regular files.
Signed-off-by: Christoph Hellwig <hch@xxxxxx>
Tested-by: Sagi Grimberg <sagi@xxxxxxxxxxx>
Signed-off-by: Anna Schumaker <Anna.Schumaker@xxxxxxxxxx>
fs/nfs/file.c | 4 +++-
fs/nfs/inode.c | 1 +
2 files changed, 4 insertions(+), 1 deletion(-)
[2]
Example #1:
rcu: INFO: rcu_sched self-detected stall on CPU
rcu: 0-....: (5249 ticks this GP) idle=cfb4/1/0x4000000000000000
softirq=1809/1813 fqs=2527
rcu: (t=5250 jiffies g=2281 q=2004 ncpus=24)
CPU: 0 PID: 1047 Comm: nginx_openssl_3 Not tainted 6.10.0-bisect+ #21
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:xas_start+0x3f/0xc0
Code: 05 c0 ff ff 77 2d 48 8b 07 48 8b 57 08 48 8b 40 08 48 89 c1 83 e1
03 48 83 f9 02 75 08 48 3d 00 10 00 00 77 19 48 85 d2 75 21 <48> c7 47
18 00 00 00 00 c3 48 c1 fa 02 85 d2 74 cb 31 c0 c3 0f b6
RSP: 0018:ffff888108a4bad8 EFLAGS: 00000293
RAX: ffff88810c236912 RBX: ffff888108a4bc58 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff888108a4bae8
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103d30318
R13: ffff8881002a3700 R14: 0000000000000000 R15: ffff888105ba2e40
FS: 00007fa598930740(0000) GS:ffff88885f800000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055dd8d91fca0 CR3: 0000000108b7e005 CR4: 0000000000370eb0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<IRQ>
? rcu_dump_cpu_stacks+0xc7/0x100
? rcu_sched_clock_irq+0x516/0xb20
? update_process_times+0x69/0xa0
? tick_nohz_handler+0x87/0x110
? tick_do_update_jiffies64+0xd0/0xd0
? __hrtimer_run_queues+0x121/0x270
? hrtimer_interrupt+0x10f/0x260
? __sysvec_apic_timer_interrupt+0x4f/0x110
? sysvec_apic_timer_interrupt+0x6c/0x90
</IRQ>
<TASK>
? asm_sysvec_apic_timer_interrupt+0x16/0x20
? xas_start+0x3f/0xc0
xas_load+0x5/0xa0
filemap_get_read_batch+0x19e/0x2a0
filemap_get_pages+0x97/0x600
? nfs_update_inode+0x4b9/0xb70
filemap_splice_read+0x12b/0x300
? tls_push_sg+0x13e/0x220
? tls_push_data+0x6bd/0xa40
nfs_file_splice_read+0x78/0xa0
splice_direct_to_actor+0xb0/0x230
? splice_file_range_actor+0x40/0x40
do_splice_direct+0x73/0xb0
? propagate_umount+0x560/0x560
do_sendfile+0x33b/0x3e0
__x64_sys_sendfile64+0x5d/0xd0
do_syscall_64+0x4c/0x100
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7fa598705dae
Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 fd fd ff ff 0f 1f 44 00 00 31 c0
c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 4a 40 0f 00 f7 d8 64 89 01 48
RSP: 002b:00007ffc17804728 EFLAGS: 00000206 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 0000000039960ce0 RCX: 00007fa598705dae
RDX: 00007ffc17804738 RSI: 0000000000000030 RDI: 0000000000000020
RBP: 0000000000000030 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000003e80 R11: 0000000000000206 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000003e80 R15: 00000000399b8a68
</TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 1048 Comm: nginx_openssl_3 Not tainted 6.10.0-bisect+ #21
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:xas_load+0x5/0xa0
Code: 48 c1 e8 02 0f b6 c0 48 83 c0 04 48 8b 44 c2 08 c3 48 8b 07 48 8b
40 08 c3 66 66 2e 0f 1f 84 00 00 00 00 00 90 e8 3b ff ff ff <48> 89 c2
83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 01 c3 0f
RSP: 0018:ffff888108a3bae0 EFLAGS: 00000293
RAX: ffff88810c236912 RBX: ffff888108a3bc58 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff888108a3bae8
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103d30318
R13: ffff888103b14700 R14: 0000000000000000 R15: ffff888105fbbc80
FS: 00007fa598930740(0000) GS:ffff88885f840000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe83bffd550 CR3: 0000000108f09002 CR4: 0000000000370eb0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<NMI>
? nmi_cpu_backtrace+0x7f/0xe0
? nmi_cpu_backtrace_handler+0xd/0x20
? nmi_handle+0x56/0x150
? default_do_nmi+0x3e/0xd0
? exc_nmi+0xd8/0x100
? end_repeat_nmi+0xf/0x18
? xas_load+0x5/0xa0
? xas_load+0x5/0xa0
? xas_load+0x5/0xa0
</NMI>
<TASK>
filemap_get_read_batch+0x19e/0x2a0
filemap_get_pages+0x97/0x600
? nfs_update_inode+0x4b9/0xb70
filemap_splice_read+0x12b/0x300
? tls_push_sg+0x13e/0x220
? tls_push_data+0x6bd/0xa40
nfs_file_splice_read+0x78/0xa0
splice_direct_to_actor+0xb0/0x230
? splice_file_range_actor+0x40/0x40
do_splice_direct+0x73/0xb0
? propagate_umount+0x560/0x560
do_sendfile+0x33b/0x3e0
__x64_sys_sendfile64+0x5d/0xd0
do_syscall_64+0x4c/0x100
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7fa598705dae
Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 fd fd ff ff 0f 1f 44 00 00 31 c0
c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 4a 40 0f 00 f7 d8 64 89 01 48
RSP: 002b:00007ffc17804728 EFLAGS: 00000206 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 000000003993d090 RCX: 00007fa598705dae
RDX: 00007ffc17804738 RSI: 000000000000002d RDI: 0000000000000019
RBP: 000000000000002d R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000003e80 R11: 0000000000000206 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000003e80 R15: 000000003999f4d8
</TASK>
Sending NMI from CPU 0 to CPUs 2:
NMI backtrace for cpu 2
CPU: 2 PID: 1049 Comm: nginx_openssl_3 Not tainted 6.10.0-bisect+ #21
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:xas_load+0x53/0xa0
Code: 77 08 48 d3 ee 83 e6 3f 89 f0 48 83 c0 04 48 8b 44 c2 08 48 89 57
18 48 89 c1 83 e1 03 48 83 f9 02 74 10 40 88 77 12 80 3a 00 <75> b0 c3
48 83 f9 02 75 f0 48 3d fd 00 00 00 77 e8 48 c1 e8 02 89
RSP: 0018:ffff888103813ae0 EFLAGS: 00000246
RAX: ffffea00046ee800 RBX: ffff888103813c58 RCX: 0000000000000000
RDX: ffff88810c236910 RSI: 0000000000000000 RDI: ffff888103813ae8
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103d30318
R13: ffff888103b08700 R14: 0000000000000000 R15: ffff888117432480
FS: 00007fa598930740(0000) GS:ffff88885f880000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7704001950 CR3: 000000010d823002 CR4: 0000000000370eb0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<NMI>
? nmi_cpu_backtrace+0x7f/0xe0
? nmi_cpu_backtrace_handler+0xd/0x20
? nmi_handle+0x56/0x150
? default_do_nmi+0x3e/0xd0
? exc_nmi+0xd8/0x100
? end_repeat_nmi+0xf/0x18
? xas_load+0x53/0xa0
? xas_load+0x53/0xa0
? xas_load+0x53/0xa0
</NMI>
<TASK>
filemap_get_read_batch+0x19e/0x2a0
filemap_get_pages+0x97/0x600
? nfs_update_inode+0x4b9/0xb70
filemap_splice_read+0x12b/0x300
? tls_push_sg+0x13e/0x220
? tls_push_data+0x6bd/0xa40
nfs_file_splice_read+0x78/0xa0
splice_direct_to_actor+0xb0/0x230
? splice_file_range_actor+0x40/0x40
do_splice_direct+0x73/0xb0
? propagate_umount+0x560/0x560
do_sendfile+0x33b/0x3e0
__x64_sys_sendfile64+0x5d/0xd0
do_syscall_64+0x4c/0x100
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7fa598705dae
Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 fd fd ff ff 0f 1f 44 00 00 31 c0
c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 4a 40 0f 00 f7 d8 64 89 01 48
RSP: 002b:00007ffc17804728 EFLAGS: 00000206 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 0000000039906100 RCX: 00007fa598705dae
RDX: 00007ffc17804738 RSI: 0000000000000034 RDI: 000000000000001c
RBP: 0000000000000034 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000003e80 R11: 0000000000000206 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000003e80 R15: 00000000399b3888
</TASK>
Sending NMI from CPU 0 to CPUs 3:
NMI backtrace for cpu 3
CPU: 3 PID: 1050 Comm: nginx_openssl_3 Not tainted 6.10.0-bisect+ #21
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:xas_start+0x53/0xc0
Code: 83 e1 03 48 83 f9 02 75 08 48 3d 00 10 00 00 77 19 48 85 d2 75 21
48 c7 47 18 00 00 00 00 c3 48 c1 fa 02 85 d2 74 cb 31 c0 c3 <0f> b6 48
fe 48 d3 ea 48 83 fa 3f 76 df 48 c7 47 18 01 00 00 00 31
RSP: 0018:ffff8881328dbad8 EFLAGS: 00000286
RAX: ffff88810c236912 RBX: ffff8881328dbc58 RCX: 0000000000000002
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881328dbae8
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103d30318
R13: ffff88810402e100 R14: 0000000000000000 R15: ffff888104032780
FS: 00007fa598930740(0000) GS:ffff88885f8c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055ddbea4a678 CR3: 0000000108b12001 CR4: 0000000000370eb0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<NMI>
? nmi_cpu_backtrace+0x7f/0xe0
? nmi_cpu_backtrace_handler+0xd/0x20
? nmi_handle+0x56/0x150
? default_do_nmi+0x3e/0xd0
? exc_nmi+0xd8/0x100
? end_repeat_nmi+0xf/0x18
? xas_start+0x53/0xc0
? xas_start+0x53/0xc0
? xas_start+0x53/0xc0
</NMI>
<TASK>
xas_load+0x5/0xa0
filemap_get_read_batch+0x19e/0x2a0
filemap_get_pages+0x97/0x600
? nfs_update_inode+0x4b9/0xb70
filemap_splice_read+0x12b/0x300
? tls_push_sg+0x13e/0x220
? common_interrupt+0xf/0xa0
? asm_common_interrupt+0x22/0x40
? _raw_spin_lock+0x10/0x20
nfs_file_splice_read+0x78/0xa0
splice_direct_to_actor+0xb0/0x230
? splice_file_range_actor+0x40/0x40
do_splice_direct+0x73/0xb0
? propagate_umount+0x560/0x560
do_sendfile+0x33b/0x3e0
__x64_sys_sendfile64+0x5d/0xd0
do_syscall_64+0x4c/0x100
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7fa598705dae
Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 fd fd ff ff 0f 1f 44 00 00 31 c0
c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 4a 40 0f 00 f7 d8 64 89 01 48
RSP: 002b:00007ffc17804728 EFLAGS: 00000206 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 000000003994a6d0 RCX: 00007fa598705dae
RDX: 00007ffc17804738 RSI: 000000000000002f RDI: 0000000000000016
RBP: 000000000000002f R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000003e80 R11: 0000000000000206 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000003e80 R15: 000000003998d548
</TASK>
Example #2:
Oops: general protection fault, probably for non-canonical address
0xdead000000000122: 0000 [#1] SMP
CPU: 4 PID: 0 Comm: swapper/4 Not tainted 6.10.0-bisect+ #23
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:free_pcppages_bulk+0x12f/0x1e0
Code: 89 34 24 e8 a3 ed ff ff 49 8b 14 24 45 31 c9 4c 89 ff 49 89 c0 89
44 24 20 49 8b 44 24 08 8b 4c 24 0c 48 8b 34 24 48 89 42 08 <48> 89 10
48 8b 54 24 18 48 b8 00 01 00 00 00 00 ad de 49 89 04 24
RSP: 0018:ffff88885f905888 EFLAGS: 00010046
RAX: dead000000000122 RBX: ffff88885f932810 RCX: 0000000000000000
RDX: ffff88885f932830 RSI: 00000000001144a0 RDI: ffffea0004512800
RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000001 R11: dead000000000100 R12: ffffea0004512808
R13: 000000000000003a R14: ffff88885f932800 R15: ffffea0004512800
FS: 0000000000000000(0000) GS:ffff88885f900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056063c04b2e8 CR3: 000000000282b003 CR4: 0000000000370eb0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<IRQ>
? die_addr+0x33/0x90
? exc_general_protection+0x1a2/0x390
? asm_exc_general_protection+0x22/0x30
? free_pcppages_bulk+0x12f/0x1e0
? free_pcppages_bulk+0x10d/0x1e0
free_unref_page_commit+0x14d/0x2b0
free_unref_page+0x18a/0x3e0
skb_release_data+0x10d/0x180
__kfree_skb+0x25/0x30
tcp_ack+0x70d/0x14d0
? tcp_v6_rcv+0xf3c/0x1240
tcp_rcv_established+0x5a9/0x760
tcp_v6_do_rcv+0xd3/0x4a0
tcp_v6_rcv+0xf3c/0x1240
? ip6_sublist_rcv+0x231/0x270
ip6_protocol_deliver_rcu+0x56/0x450
ip6_input+0xbf/0xe0
? tcp_v6_early_demux+0xb2/0x190
ip6_sublist_rcv_finish+0x32/0x40
ip6_sublist_rcv+0x231/0x270
? ip6_sublist_rcv+0x270/0x270
ipv6_list_rcv+0xfc/0x120
__netif_receive_skb_list_core+0x180/0x1e0
netif_receive_skb_list_internal+0x1b5/0x2c0
napi_complete_done+0x6f/0x190
mlx5e_napi_poll+0x149/0x6a0 [mlx5_core]
__napi_poll+0x24/0x190
net_rx_action+0x328/0x3b0
? mlx5_eq_comp_int+0x1bc/0x1e0 [mlx5_core]
? notifier_call_chain+0x35/0xa0
handle_softirqs+0xcc/0x270
irq_exit_rcu+0x67/0x90
common_interrupt+0x7f/0xa0
</IRQ>
<TASK>
asm_common_interrupt+0x22/0x40
RIP: 0010:default_idle+0x13/0x20
Code: c0 08 00 00 00 4d 29 c8 4c 01 c7 4c 29 c2 e9 72 ff ff ff cc cc cc
cc 8b 05 ca 29 4e 01 85 c0 7e 07 0f 00 2d f1 5a 25 00 fb f4 <fa> c3 66
66 2e 0f 1f 84 00 00 00 00 00 65 48 8b 35 38 7f 46 7e f0
RSP: 0018:ffff8881018cbee0 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88810189d500 RCX: 7fffffffffffffff
RDX: 0000000000000000 RSI: 000000089ca81700 RDI: 000000000014f654
RBP: 0000000000000004 R08: 7fffffffffffffff R09: 00000000fffeff0f
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
default_idle_call+0x39/0xd0
do_idle+0x1ab/0x1c0
cpu_startup_entry+0x25/0x30
start_secondary+0x105/0x130
common_startup_64+0x129/0x138
</TASK>
Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink
nfnetlink xt_addrtype iptable_nat nf_nat br_netfilter rpcsec_gss_krb5
auth_rpcgss oid_registry overlay mlx5_ib zram zsmalloc mlx5_core rpcrdma
rdma_ucm ib_uverbs ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm
ib_ipoib iw_cm ib_cm fuse ib_core
---[ end trace 0000000000000000 ]---
RIP: 0010:free_pcppages_bulk+0x12f/0x1e0
Code: 89 34 24 e8 a3 ed ff ff 49 8b 14 24 45 31 c9 4c 89 ff 49 89 c0 89
44 24 20 49 8b 44 24 08 8b 4c 24 0c 48 8b 34 24 48 89 42 08 <48> 89 10
48 8b 54 24 18 48 b8 00 01 00 00 00 00 ad de 49 89 04 24
RSP: 0018:ffff88885f905888 EFLAGS: 00010046
RAX: dead000000000122 RBX: ffff88885f932810 RCX: 0000000000000000
RDX: ffff88885f932830 RSI: 00000000001144a0 RDI: ffffea0004512800
RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000001 R11: dead000000000100 R12: ffffea0004512808
R13: 000000000000003a R14: ffff88885f932800 R15: ffffea0004512800
FS: 0000000000000000(0000) GS:ffff88885f900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056063c04b2e8 CR3: 000000000282b003 CR4: 0000000000370eb0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Kernel panic - not syncing: Fatal exception in interrupt
Shutting down cpus with NMI
Kernel Offset: disabled
---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
Example #3:
BUG: kernel NULL pointer dereference, address: 0000000000000008
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
PGD 108898067 P4D 108898067 PUD 108891067 PMD 0
Oops: Oops: 0002 [#1] SMP
CPU: 1 PID: 1157 Comm: nginx_openssl_3 Not tainted 6.10.0-bisect+ #26
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
RIP: 0010:__page_cache_release+0xc7/0x260
Code: 8b 03 48 8b 53 08 48 c1 ed 12 83 e5 01 48 c1 e8 08 83 f5 01 83 e0
01 40 0f b6 ed 01 ed 3c 01 48 8b 43 10 83 dd ff 44 8d 7d 01 <48> 89 42
08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 08 48
RSP: 0018:ffff888110197b78 EFLAGS: 00010013
RAX: dead000000000122 RBX: ffffea0004e9aa00 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff888110197bc8 RDI: ffff8881001e1050
RBP: 0000000000000002 R08: 000000000000005a R09: 00000000000009bc
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881001e1000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
FS: 00007fa91da46740(0000) GS:ffff88885f840000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000010889c001 CR4: 0000000000370eb0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? __die+0x20/0x60
? page_fault_oops+0x150/0x3e0
? exc_page_fault+0x74/0x130
? asm_exc_page_fault+0x22/0x30
? __page_cache_release+0xc7/0x260
? __page_cache_release+0x84/0x260
? folio_activate_fn+0x2d0/0x2d0
folios_put_refs+0x6d/0x170
filemap_splice_read+0x2b8/0x300
? tls_push_sg+0x13e/0x220
? tls_push_data+0x6bd/0xa40
nfs_file_splice_read+0x78/0xa0
splice_direct_to_actor+0xb0/0x230
? splice_file_range_actor+0x40/0x40
do_splice_direct+0x73/0xb0
? propagate_umount+0x560/0x560
do_sendfile+0x33b/0x3e0
__x64_sys_sendfile64+0x5d/0xd0
do_syscall_64+0x4c/0x100
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7fa91d905dae
Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 fd fd ff ff 0f 1f 44 00 00 31 c0
c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 01 c3 48 8b 0d 4a 40 0f 00 f7 d8 64 89 01 48
RSP: 002b:00007ffda039ab98 EFLAGS: 00000202 ORIG_RAX: 0000000000000028
RAX: ffffffffffffffda RBX: 0000000029e45110 RCX: 00007fa91d905dae
RDX: 00007ffda039aba8 RSI: 0000000000000031 RDI: 000000000000001e
RBP: 0000000000000031 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000003e80 R11: 0000000000000202 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000003e80 R15: 0000000029e02c88
</TASK>
Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink
nfnetlink xt_addrtype iptable_nat nf_nat br_netfilter rpcsec_gss_krb5
auth_rpcgss oid_registry overlay mlx5_ib zram zsmalloc mlx5_core rpcrdma
rdma_ucm ib_uverbs ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm
ib_ipoib iw_cm ib_cm fuse ib_core
CR2: 0000000000000008
---[ end trace 0000000000000000 ]---
BUG: kernel NULL pointer dereference, address: 0000000000000008
RIP: 0010:__page_cache_release+0xc7/0x260
#PF: supervisor write access in kernel mode
Code: 8b 03 48 8b 53 08 48 c1 ed 12 83 e5 01 48 c1 e8 08 83 f5 01 83 e0
01 40 0f b6 ed 01 ed 3c 01 48 8b 43 10 83 dd ff 44 8d 7d 01 <48> 89 42
08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 08 48
#PF: error_code(0x0002) - not-present page
RSP: 0018:ffff888110197b78 EFLAGS: 00010013
PGD 1092fc067
P4D 1092fc067
RAX: dead000000000122 RBX: ffffea0004e9aa00 RCX: 0000000000000000
PUD 1092fb067 PMD 0
RDX: 0000000000000000 RSI: ffff888110197bc8 RDI: ffff8881001e1050
RBP: 0000000000000002 R08: 000000000000005a R09: 00000000000009bc
Oops: Oops: 0002 [#2] SMP
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881001e1000
CPU: 3 PID: 1159 Comm: nginx_openssl_3 Tainted: G D 6.10.0-bisect+ #26
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
FS: 00007fa91da46740(0000) GS:ffff88885f840000(0000) knlGS:0000000000000000
RIP: 0010:__page_cache_release+0xc7/0x260
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Code: 8b 03 48 8b 53 08 48 c1 ed 12 83 e5 01 48 c1 e8 08 83 f5 01 83 e0
01 40 0f b6 ed 01 ed 3c 01 48 8b 43 10 83 dd ff 44 8d 7d 01 <48> 89 42
08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 08 48
CR2: 0000000000000008 CR3: 000000010889c001 CR4: 0000000000370eb0
RSP: 0018:ffff888124553cb8 EFLAGS: 00010013
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
RAX: dead000000000122 RBX: ffffea0004e9aa00 RCX: 0000000000000000
note: nginx_openssl_3[1157] exited with irqs disabled
RDX: 0000000000000000 RSI: ffff888124553d08 RDI: ffff888110672850
RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000000 R12: ffff888110672800
R13: 000000000000008e R14: 0000000000000058 R15: 0000000000000003
FS: 00007fa91da46740(0000) GS:ffff88885f8c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 00000001092ff006 CR4: 0000000000370eb0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
<TASK>
? __die+0x20/0x60
? page_fault_oops+0x150/0x3e0
? exc_page_fault+0x74/0x130
? asm_exc_page_fault+0x22/0x30
? __page_cache_release+0xc7/0x260
? __page_cache_release+0x84/0x260
__folio_put+0x43/0xe0
__filemap_get_folio+0x20c/0x2a0
ext4_da_write_begin+0xe1/0x240
generic_perform_write+0xe0/0x2c0
ext4_buffered_write_iter+0x62/0xe0
vfs_write+0x2c8/0x3f0
ksys_write+0x5f/0xe0
do_syscall_64+0x4c/0x100
entry_SYSCALL_64_after_hwframe+0x4b/0x53
RIP: 0033:0x7fa91d9018b7
Code: 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e
fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00
f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
RSP: 002b:00007ffda039af78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000029e45270 RCX: 00007fa91d9018b7
RDX: 0000000000000058 RSI: 0000000029e160f8 RDI: 0000000000000004
RBP: 0000000029ce3700 R08: 00000000cccccccd R09: 0000000000000000
R10: 0000000029e16142 R11: 0000000000000246 R12: 0000000000000058
R13: 0000000029ce35d0 R14: 0000000000000000 R15: 0000000029e45270
</TASK>
Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink
nfnetlink xt_addrtype iptable_nat nf_nat br_netfilter rpcsec_gss_krb5
auth_rpcgss oid_registry overlay mlx5_ib zram zsmalloc mlx5_core rpcrdma
rdma_ucm ib_uverbs ib_iser libiscsi scsi_transport_iscsi ib_umad rdma_cm
ib_ipoib iw_cm ib_cm fuse ib_core
CR2: 0000000000000008
---[ end trace 0000000000000000 ]---
RIP: 0010:__page_cache_release+0xc7/0x260
Code: 8b 03 48 8b 53 08 48 c1 ed 12 83 e5 01 48 c1 e8 08 83 f5 01 83 e0
01 40 0f b6 ed 01 ed 3c 01 48 8b 43 10 83 dd ff 44 8d 7d 01 <48> 89 42
08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 08 48
RSP: 0018:ffff888110197b78 EFLAGS: 00010013
RAX: dead000000000122 RBX: ffffea0004e9aa00 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff888110197bc8 RDI: ffff8881001e1050
RBP: 0000000000000002 R08: 000000000000005a R09: 00000000000009bc
