Re: [Bug report] NFS patch breaks TLS device-offloaded TX zerocopy

[Sagi Grimberg <sagi@xxxxxxxxxxx>]

On 05/08/2024 13:40, Tariq Toukan wrote:
> Hi,
>
> A recent patch [1] to 'fs' broke the TX TLS device-offloaded flow 
> starting from v6.11-rc1.
>
> The kernel crashes. Different runs result in different kernel traces.
> See below [2].
> All of them disappear once patch [1] is reverted.
>
> The issues appears only with "sendfile on and zerocopy on".
> We couldn't repro with "sendfile off", or with "sendfile on and 
> zerocopy off".
>
> The repro test is as simple as a repeated client/server communication 
> (wrk/nginx), with sendfile on and zc on, and with "tls-hw-tx-offload: 
> on".
>
> $ for i in `seq 10`; do wrk -b::2:2:2:3 -t10 -c100 -d15 --timeout 5s 
> https://[::2:2:2:2]:20448/16000b.img; done
>
> We can provide more details if needed, to help with the analysis and 
> debug.

Does tls sw (i.e. no offload) also break?

> Regards,
> Tariq
>
> [1]
> commit 49b29a573da83b65d5f4ecf2db6619bab7aa910c
> Author: Christoph Hellwig <hch@xxxxxx>
> Date:   Mon May 27 18:36:09 2024 +0200
>
>     nfs: add support for large folios
>
>     NFS already is void of folio size assumption, so just pass the 
> chunk size
>     to __filemap_get_folio and set the large folio address_space flag 
> for all
>     regular files.
>
>     Signed-off-by: Christoph Hellwig <hch@xxxxxx>
>     Tested-by: Sagi Grimberg <sagi@xxxxxxxxxxx>
>     Signed-off-by: Anna Schumaker <Anna.Schumaker@xxxxxxxxxx>
>
>  fs/nfs/file.c  | 4 +++-
>  fs/nfs/inode.c | 1 +
>  2 files changed, 4 insertions(+), 1 deletion(-)
>
>
> [2]
>
> Example #1:
>
> rcu: INFO: rcu_sched self-detected stall on CPU
> rcu:     0-....: (5249 ticks this GP) idle=cfb4/1/0x4000000000000000 
> softirq=1809/1813 fqs=2527
> rcu:     (t=5250 jiffies g=2281 q=2004 ncpus=24)
> CPU: 0 PID: 1047 Comm: nginx_openssl_3 Not tainted 6.10.0-bisect+ #21
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> RIP: 0010:xas_start+0x3f/0xc0
> Code: 05 c0 ff ff 77 2d 48 8b 07 48 8b 57 08 48 8b 40 08 48 89 c1 83 
> e1 03 48 83 f9 02 75 08 48 3d 00 10 00 00 77 19 48 85 d2 75 21 <48> c7 
> 47 18 00 00 00 00 c3 48 c1 fa 02 85 d2 74 cb 31 c0 c3 0f b6
> RSP: 0018:ffff888108a4bad8 EFLAGS: 00000293
> RAX: ffff88810c236912 RBX: ffff888108a4bc58 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff888108a4bae8
> RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103d30318
> R13: ffff8881002a3700 R14: 0000000000000000 R15: ffff888105ba2e40
> FS:  00007fa598930740(0000) GS:ffff88885f800000(0000) 
> knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 000055dd8d91fca0 CR3: 0000000108b7e005 CR4: 0000000000370eb0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <IRQ>
>  ? rcu_dump_cpu_stacks+0xc7/0x100
>  ? rcu_sched_clock_irq+0x516/0xb20
>  ? update_process_times+0x69/0xa0
>  ? tick_nohz_handler+0x87/0x110
>  ? tick_do_update_jiffies64+0xd0/0xd0
>  ? __hrtimer_run_queues+0x121/0x270
>  ? hrtimer_interrupt+0x10f/0x260
>  ? __sysvec_apic_timer_interrupt+0x4f/0x110
>  ? sysvec_apic_timer_interrupt+0x6c/0x90
>  </IRQ>
>  <TASK>
>  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
>  ? xas_start+0x3f/0xc0
>  xas_load+0x5/0xa0
>  filemap_get_read_batch+0x19e/0x2a0
>  filemap_get_pages+0x97/0x600
>  ? nfs_update_inode+0x4b9/0xb70
>  filemap_splice_read+0x12b/0x300
>  ? tls_push_sg+0x13e/0x220
>  ? tls_push_data+0x6bd/0xa40
>  nfs_file_splice_read+0x78/0xa0
>  splice_direct_to_actor+0xb0/0x230
>  ? splice_file_range_actor+0x40/0x40
>  do_splice_direct+0x73/0xb0
>  ? propagate_umount+0x560/0x560
>  do_sendfile+0x33b/0x3e0
>  __x64_sys_sendfile64+0x5d/0xd0
>  do_syscall_64+0x4c/0x100
>  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> RIP: 0033:0x7fa598705dae
> Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 fd fd ff ff 0f 1f 44 00 00 31 
> c0 c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 
> 01 f0 ff ff 73 01 c3 48 8b 0d 4a 40 0f 00 f7 d8 64 89 01 48
> RSP: 002b:00007ffc17804728 EFLAGS: 00000206 ORIG_RAX: 0000000000000028
> RAX: ffffffffffffffda RBX: 0000000039960ce0 RCX: 00007fa598705dae
> RDX: 00007ffc17804738 RSI: 0000000000000030 RDI: 0000000000000020
> RBP: 0000000000000030 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000003e80 R11: 0000000000000206 R12: 0000000000000000
> R13: 0000000000000000 R14: 0000000000003e80 R15: 00000000399b8a68
>  </TASK>
> Sending NMI from CPU 0 to CPUs 1:
> NMI backtrace for cpu 1
> CPU: 1 PID: 1048 Comm: nginx_openssl_3 Not tainted 6.10.0-bisect+ #21
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> RIP: 0010:xas_load+0x5/0xa0
> Code: 48 c1 e8 02 0f b6 c0 48 83 c0 04 48 8b 44 c2 08 c3 48 8b 07 48 
> 8b 40 08 c3 66 66 2e 0f 1f 84 00 00 00 00 00 90 e8 3b ff ff ff <48> 89 
> c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 01 c3 0f
> RSP: 0018:ffff888108a3bae0 EFLAGS: 00000293
> RAX: ffff88810c236912 RBX: ffff888108a3bc58 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff888108a3bae8
> RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103d30318
> R13: ffff888103b14700 R14: 0000000000000000 R15: ffff888105fbbc80
> FS:  00007fa598930740(0000) GS:ffff88885f840000(0000) 
> knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007fe83bffd550 CR3: 0000000108f09002 CR4: 0000000000370eb0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <NMI>
>  ? nmi_cpu_backtrace+0x7f/0xe0
>  ? nmi_cpu_backtrace_handler+0xd/0x20
>  ? nmi_handle+0x56/0x150
>  ? default_do_nmi+0x3e/0xd0
>  ? exc_nmi+0xd8/0x100
>  ? end_repeat_nmi+0xf/0x18
>  ? xas_load+0x5/0xa0
>  ? xas_load+0x5/0xa0
>  ? xas_load+0x5/0xa0
>  </NMI>
>  <TASK>
>  filemap_get_read_batch+0x19e/0x2a0
>  filemap_get_pages+0x97/0x600
>  ? nfs_update_inode+0x4b9/0xb70
>  filemap_splice_read+0x12b/0x300
>  ? tls_push_sg+0x13e/0x220
>  ? tls_push_data+0x6bd/0xa40
>  nfs_file_splice_read+0x78/0xa0
>  splice_direct_to_actor+0xb0/0x230
>  ? splice_file_range_actor+0x40/0x40
>  do_splice_direct+0x73/0xb0
>  ? propagate_umount+0x560/0x560
>  do_sendfile+0x33b/0x3e0
>  __x64_sys_sendfile64+0x5d/0xd0
>  do_syscall_64+0x4c/0x100
>  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> RIP: 0033:0x7fa598705dae
> Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 fd fd ff ff 0f 1f 44 00 00 31 
> c0 c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 
> 01 f0 ff ff 73 01 c3 48 8b 0d 4a 40 0f 00 f7 d8 64 89 01 48
> RSP: 002b:00007ffc17804728 EFLAGS: 00000206 ORIG_RAX: 0000000000000028
> RAX: ffffffffffffffda RBX: 000000003993d090 RCX: 00007fa598705dae
> RDX: 00007ffc17804738 RSI: 000000000000002d RDI: 0000000000000019
> RBP: 000000000000002d R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000003e80 R11: 0000000000000206 R12: 0000000000000000
> R13: 0000000000000000 R14: 0000000000003e80 R15: 000000003999f4d8
>  </TASK>
> Sending NMI from CPU 0 to CPUs 2:
> NMI backtrace for cpu 2
> CPU: 2 PID: 1049 Comm: nginx_openssl_3 Not tainted 6.10.0-bisect+ #21
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> RIP: 0010:xas_load+0x53/0xa0
> Code: 77 08 48 d3 ee 83 e6 3f 89 f0 48 83 c0 04 48 8b 44 c2 08 48 89 
> 57 18 48 89 c1 83 e1 03 48 83 f9 02 74 10 40 88 77 12 80 3a 00 <75> b0 
> c3 48 83 f9 02 75 f0 48 3d fd 00 00 00 77 e8 48 c1 e8 02 89
> RSP: 0018:ffff888103813ae0 EFLAGS: 00000246
> RAX: ffffea00046ee800 RBX: ffff888103813c58 RCX: 0000000000000000
> RDX: ffff88810c236910 RSI: 0000000000000000 RDI: ffff888103813ae8
> RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103d30318
> R13: ffff888103b08700 R14: 0000000000000000 R15: ffff888117432480
> FS:  00007fa598930740(0000) GS:ffff88885f880000(0000) 
> knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f7704001950 CR3: 000000010d823002 CR4: 0000000000370eb0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <NMI>
>  ? nmi_cpu_backtrace+0x7f/0xe0
>  ? nmi_cpu_backtrace_handler+0xd/0x20
>  ? nmi_handle+0x56/0x150
>  ? default_do_nmi+0x3e/0xd0
>  ? exc_nmi+0xd8/0x100
>  ? end_repeat_nmi+0xf/0x18
>  ? xas_load+0x53/0xa0
>  ? xas_load+0x53/0xa0
>  ? xas_load+0x53/0xa0
>  </NMI>
>  <TASK>
>  filemap_get_read_batch+0x19e/0x2a0
>  filemap_get_pages+0x97/0x600
>  ? nfs_update_inode+0x4b9/0xb70
>  filemap_splice_read+0x12b/0x300
>  ? tls_push_sg+0x13e/0x220
>  ? tls_push_data+0x6bd/0xa40
>  nfs_file_splice_read+0x78/0xa0
>  splice_direct_to_actor+0xb0/0x230
>  ? splice_file_range_actor+0x40/0x40
>  do_splice_direct+0x73/0xb0
>  ? propagate_umount+0x560/0x560
>  do_sendfile+0x33b/0x3e0
>  __x64_sys_sendfile64+0x5d/0xd0
>  do_syscall_64+0x4c/0x100
>  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> RIP: 0033:0x7fa598705dae
> Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 fd fd ff ff 0f 1f 44 00 00 31 
> c0 c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 
> 01 f0 ff ff 73 01 c3 48 8b 0d 4a 40 0f 00 f7 d8 64 89 01 48
> RSP: 002b:00007ffc17804728 EFLAGS: 00000206 ORIG_RAX: 0000000000000028
> RAX: ffffffffffffffda RBX: 0000000039906100 RCX: 00007fa598705dae
> RDX: 00007ffc17804738 RSI: 0000000000000034 RDI: 000000000000001c
> RBP: 0000000000000034 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000003e80 R11: 0000000000000206 R12: 0000000000000000
> R13: 0000000000000000 R14: 0000000000003e80 R15: 00000000399b3888
>  </TASK>
> Sending NMI from CPU 0 to CPUs 3:
> NMI backtrace for cpu 3
> CPU: 3 PID: 1050 Comm: nginx_openssl_3 Not tainted 6.10.0-bisect+ #21
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> RIP: 0010:xas_start+0x53/0xc0
> Code: 83 e1 03 48 83 f9 02 75 08 48 3d 00 10 00 00 77 19 48 85 d2 75 
> 21 48 c7 47 18 00 00 00 00 c3 48 c1 fa 02 85 d2 74 cb 31 c0 c3 <0f> b6 
> 48 fe 48 d3 ea 48 83 fa 3f 76 df 48 c7 47 18 01 00 00 00 31
> RSP: 0018:ffff8881328dbad8 EFLAGS: 00000286
> RAX: ffff88810c236912 RBX: ffff8881328dbc58 RCX: 0000000000000002
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8881328dbae8
> RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff888103d30318
> R13: ffff88810402e100 R14: 0000000000000000 R15: ffff888104032780
> FS:  00007fa598930740(0000) GS:ffff88885f8c0000(0000) 
> knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 000055ddbea4a678 CR3: 0000000108b12001 CR4: 0000000000370eb0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <NMI>
>  ? nmi_cpu_backtrace+0x7f/0xe0
>  ? nmi_cpu_backtrace_handler+0xd/0x20
>  ? nmi_handle+0x56/0x150
>  ? default_do_nmi+0x3e/0xd0
>  ? exc_nmi+0xd8/0x100
>  ? end_repeat_nmi+0xf/0x18
>  ? xas_start+0x53/0xc0
>  ? xas_start+0x53/0xc0
>  ? xas_start+0x53/0xc0
>  </NMI>
>  <TASK>
>  xas_load+0x5/0xa0
>  filemap_get_read_batch+0x19e/0x2a0
>  filemap_get_pages+0x97/0x600
>  ? nfs_update_inode+0x4b9/0xb70
>  filemap_splice_read+0x12b/0x300
>  ? tls_push_sg+0x13e/0x220
>  ? common_interrupt+0xf/0xa0
>  ? asm_common_interrupt+0x22/0x40
>  ? _raw_spin_lock+0x10/0x20
>  nfs_file_splice_read+0x78/0xa0
>  splice_direct_to_actor+0xb0/0x230
>  ? splice_file_range_actor+0x40/0x40
>  do_splice_direct+0x73/0xb0
>  ? propagate_umount+0x560/0x560
>  do_sendfile+0x33b/0x3e0
>  __x64_sys_sendfile64+0x5d/0xd0
>  do_syscall_64+0x4c/0x100
>  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> RIP: 0033:0x7fa598705dae
> Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 fd fd ff ff 0f 1f 44 00 00 31 
> c0 c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 
> 01 f0 ff ff 73 01 c3 48 8b 0d 4a 40 0f 00 f7 d8 64 89 01 48
> RSP: 002b:00007ffc17804728 EFLAGS: 00000206 ORIG_RAX: 0000000000000028
> RAX: ffffffffffffffda RBX: 000000003994a6d0 RCX: 00007fa598705dae
> RDX: 00007ffc17804738 RSI: 000000000000002f RDI: 0000000000000016
> RBP: 000000000000002f R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000003e80 R11: 0000000000000206 R12: 0000000000000000
> R13: 0000000000000000 R14: 0000000000003e80 R15: 000000003998d548
>  </TASK>
>
>
> Example #2:
>
> Oops: general protection fault, probably for non-canonical address 
> 0xdead000000000122: 0000 [#1] SMP
> CPU: 4 PID: 0 Comm: swapper/4 Not tainted 6.10.0-bisect+ #23
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> RIP: 0010:free_pcppages_bulk+0x12f/0x1e0
> Code: 89 34 24 e8 a3 ed ff ff 49 8b 14 24 45 31 c9 4c 89 ff 49 89 c0 
> 89 44 24 20 49 8b 44 24 08 8b 4c 24 0c 48 8b 34 24 48 89 42 08 <48> 89 
> 10 48 8b 54 24 18 48 b8 00 01 00 00 00 00 ad de 49 89 04 24
> RSP: 0018:ffff88885f905888 EFLAGS: 00010046
> RAX: dead000000000122 RBX: ffff88885f932810 RCX: 0000000000000000
> RDX: ffff88885f932830 RSI: 00000000001144a0 RDI: ffffea0004512800
> RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000
> R10: 0000000000000001 R11: dead000000000100 R12: ffffea0004512808
> R13: 000000000000003a R14: ffff88885f932800 R15: ffffea0004512800
> FS:  0000000000000000(0000) GS:ffff88885f900000(0000) 
> knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 000056063c04b2e8 CR3: 000000000282b003 CR4: 0000000000370eb0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <IRQ>
>  ? die_addr+0x33/0x90
>  ? exc_general_protection+0x1a2/0x390
>  ? asm_exc_general_protection+0x22/0x30
>  ? free_pcppages_bulk+0x12f/0x1e0
>  ? free_pcppages_bulk+0x10d/0x1e0
>  free_unref_page_commit+0x14d/0x2b0
>  free_unref_page+0x18a/0x3e0
>  skb_release_data+0x10d/0x180
>  __kfree_skb+0x25/0x30
>  tcp_ack+0x70d/0x14d0
>  ? tcp_v6_rcv+0xf3c/0x1240
>  tcp_rcv_established+0x5a9/0x760
>  tcp_v6_do_rcv+0xd3/0x4a0
>  tcp_v6_rcv+0xf3c/0x1240
>  ? ip6_sublist_rcv+0x231/0x270
>  ip6_protocol_deliver_rcu+0x56/0x450
>  ip6_input+0xbf/0xe0
>  ? tcp_v6_early_demux+0xb2/0x190
>  ip6_sublist_rcv_finish+0x32/0x40
>  ip6_sublist_rcv+0x231/0x270
>  ? ip6_sublist_rcv+0x270/0x270
>  ipv6_list_rcv+0xfc/0x120
>  __netif_receive_skb_list_core+0x180/0x1e0
>  netif_receive_skb_list_internal+0x1b5/0x2c0
>  napi_complete_done+0x6f/0x190
>  mlx5e_napi_poll+0x149/0x6a0 [mlx5_core]
>  __napi_poll+0x24/0x190
>  net_rx_action+0x328/0x3b0
>  ? mlx5_eq_comp_int+0x1bc/0x1e0 [mlx5_core]
>  ? notifier_call_chain+0x35/0xa0
>  handle_softirqs+0xcc/0x270
>  irq_exit_rcu+0x67/0x90
>  common_interrupt+0x7f/0xa0
>  </IRQ>
>  <TASK>
>  asm_common_interrupt+0x22/0x40
> RIP: 0010:default_idle+0x13/0x20
> Code: c0 08 00 00 00 4d 29 c8 4c 01 c7 4c 29 c2 e9 72 ff ff ff cc cc 
> cc cc 8b 05 ca 29 4e 01 85 c0 7e 07 0f 00 2d f1 5a 25 00 fb f4 <fa> c3 
> 66 66 2e 0f 1f 84 00 00 00 00 00 65 48 8b 35 38 7f 46 7e f0
> RSP: 0018:ffff8881018cbee0 EFLAGS: 00000246
> RAX: 0000000000000000 RBX: ffff88810189d500 RCX: 7fffffffffffffff
> RDX: 0000000000000000 RSI: 000000089ca81700 RDI: 000000000014f654
> RBP: 0000000000000004 R08: 7fffffffffffffff R09: 00000000fffeff0f
> R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
> R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>  default_idle_call+0x39/0xd0
>  do_idle+0x1ab/0x1c0
>  cpu_startup_entry+0x25/0x30
>  start_secondary+0x105/0x130
>  common_startup_64+0x129/0x138
>  </TASK>
> Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink 
> nfnetlink xt_addrtype iptable_nat nf_nat br_netfilter rpcsec_gss_krb5 
> auth_rpcgss oid_registry overlay mlx5_ib zram zsmalloc mlx5_core 
> rpcrdma rdma_ucm ib_uverbs ib_iser libiscsi scsi_transport_iscsi 
> ib_umad rdma_cm ib_ipoib iw_cm ib_cm fuse ib_core
> ---[ end trace 0000000000000000 ]---
> RIP: 0010:free_pcppages_bulk+0x12f/0x1e0
> Code: 89 34 24 e8 a3 ed ff ff 49 8b 14 24 45 31 c9 4c 89 ff 49 89 c0 
> 89 44 24 20 49 8b 44 24 08 8b 4c 24 0c 48 8b 34 24 48 89 42 08 <48> 89 
> 10 48 8b 54 24 18 48 b8 00 01 00 00 00 00 ad de 49 89 04 24
> RSP: 0018:ffff88885f905888 EFLAGS: 00010046
> RAX: dead000000000122 RBX: ffff88885f932810 RCX: 0000000000000000
> RDX: ffff88885f932830 RSI: 00000000001144a0 RDI: ffffea0004512800
> RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000000
> R10: 0000000000000001 R11: dead000000000100 R12: ffffea0004512808
> R13: 000000000000003a R14: ffff88885f932800 R15: ffffea0004512800
> FS:  0000000000000000(0000) GS:ffff88885f900000(0000) 
> knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 000056063c04b2e8 CR3: 000000000282b003 CR4: 0000000000370eb0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Kernel panic - not syncing: Fatal exception in interrupt
> Shutting down cpus with NMI
> Kernel Offset: disabled
> ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
>
>
> Example #3:
>
> BUG: kernel NULL pointer dereference, address: 0000000000000008
> #PF: supervisor write access in kernel mode
> #PF: error_code(0x0002) - not-present page
> PGD 108898067 P4D 108898067 PUD 108891067 PMD 0
> Oops: Oops: 0002 [#1] SMP
> CPU: 1 PID: 1157 Comm: nginx_openssl_3 Not tainted 6.10.0-bisect+ #26
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> RIP: 0010:__page_cache_release+0xc7/0x260
> Code: 8b 03 48 8b 53 08 48 c1 ed 12 83 e5 01 48 c1 e8 08 83 f5 01 83 
> e0 01 40 0f b6 ed 01 ed 3c 01 48 8b 43 10 83 dd ff 44 8d 7d 01 <48> 89 
> 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 08 48
> RSP: 0018:ffff888110197b78 EFLAGS: 00010013
> RAX: dead000000000122 RBX: ffffea0004e9aa00 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffff888110197bc8 RDI: ffff8881001e1050
> RBP: 0000000000000002 R08: 000000000000005a R09: 00000000000009bc
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881001e1000
> R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
> FS:  00007fa91da46740(0000) GS:ffff88885f840000(0000) 
> knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000008 CR3: 000000010889c001 CR4: 0000000000370eb0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <TASK>
>  ? __die+0x20/0x60
>  ? page_fault_oops+0x150/0x3e0
>  ? exc_page_fault+0x74/0x130
>  ? asm_exc_page_fault+0x22/0x30
>  ? __page_cache_release+0xc7/0x260
>  ? __page_cache_release+0x84/0x260
>  ? folio_activate_fn+0x2d0/0x2d0
>  folios_put_refs+0x6d/0x170
>  filemap_splice_read+0x2b8/0x300
>  ? tls_push_sg+0x13e/0x220
>  ? tls_push_data+0x6bd/0xa40
>  nfs_file_splice_read+0x78/0xa0
>  splice_direct_to_actor+0xb0/0x230
>  ? splice_file_range_actor+0x40/0x40
>  do_splice_direct+0x73/0xb0
>  ? propagate_umount+0x560/0x560
>  do_sendfile+0x33b/0x3e0
>  __x64_sys_sendfile64+0x5d/0xd0
>  do_syscall_64+0x4c/0x100
>  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> RIP: 0033:0x7fa91d905dae
> Code: c3 0f 1f 00 4c 89 d2 4c 89 c6 e9 fd fd ff ff 0f 1f 44 00 00 31 
> c0 c3 0f 1f 44 00 00 f3 0f 1e fa 49 89 ca b8 28 00 00 00 0f 05 <48> 3d 
> 01 f0 ff ff 73 01 c3 48 8b 0d 4a 40 0f 00 f7 d8 64 89 01 48
> RSP: 002b:00007ffda039ab98 EFLAGS: 00000202 ORIG_RAX: 0000000000000028
> RAX: ffffffffffffffda RBX: 0000000029e45110 RCX: 00007fa91d905dae
> RDX: 00007ffda039aba8 RSI: 0000000000000031 RDI: 000000000000001e
> RBP: 0000000000000031 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000003e80 R11: 0000000000000202 R12: 0000000000000000
> R13: 0000000000000000 R14: 0000000000003e80 R15: 0000000029e02c88
>  </TASK>
> Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink 
> nfnetlink xt_addrtype iptable_nat nf_nat br_netfilter rpcsec_gss_krb5 
> auth_rpcgss oid_registry overlay mlx5_ib zram zsmalloc mlx5_core 
> rpcrdma rdma_ucm ib_uverbs ib_iser libiscsi scsi_transport_iscsi 
> ib_umad rdma_cm ib_ipoib iw_cm ib_cm fuse ib_core
> CR2: 0000000000000008
> ---[ end trace 0000000000000000 ]---
> BUG: kernel NULL pointer dereference, address: 0000000000000008
> RIP: 0010:__page_cache_release+0xc7/0x260
> #PF: supervisor write access in kernel mode
> Code: 8b 03 48 8b 53 08 48 c1 ed 12 83 e5 01 48 c1 e8 08 83 f5 01 83 
> e0 01 40 0f b6 ed 01 ed 3c 01 48 8b 43 10 83 dd ff 44 8d 7d 01 <48> 89 
> 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 08 48
> #PF: error_code(0x0002) - not-present page
> RSP: 0018:ffff888110197b78 EFLAGS: 00010013
> PGD 1092fc067
>
> P4D 1092fc067
> RAX: dead000000000122 RBX: ffffea0004e9aa00 RCX: 0000000000000000
> PUD 1092fb067 PMD 0
> RDX: 0000000000000000 RSI: ffff888110197bc8 RDI: ffff8881001e1050
>
> RBP: 0000000000000002 R08: 000000000000005a R09: 00000000000009bc
> Oops: Oops: 0002 [#2] SMP
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff8881001e1000
> CPU: 3 PID: 1159 Comm: nginx_openssl_3 Tainted: G      D 
> 6.10.0-bisect+ #26
> R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> FS:  00007fa91da46740(0000) GS:ffff88885f840000(0000) 
> knlGS:0000000000000000
> RIP: 0010:__page_cache_release+0xc7/0x260
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Code: 8b 03 48 8b 53 08 48 c1 ed 12 83 e5 01 48 c1 e8 08 83 f5 01 83 
> e0 01 40 0f b6 ed 01 ed 3c 01 48 8b 43 10 83 dd ff 44 8d 7d 01 <48> 89 
> 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 08 48
> CR2: 0000000000000008 CR3: 000000010889c001 CR4: 0000000000370eb0
> RSP: 0018:ffff888124553cb8 EFLAGS: 00010013
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> RAX: dead000000000122 RBX: ffffea0004e9aa00 RCX: 0000000000000000
> note: nginx_openssl_3[1157] exited with irqs disabled
> RDX: 0000000000000000 RSI: ffff888124553d08 RDI: ffff888110672850
> RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000001
> R10: 0000000000000001 R11: 0000000000000000 R12: ffff888110672800
> R13: 000000000000008e R14: 0000000000000058 R15: 0000000000000003
> FS:  00007fa91da46740(0000) GS:ffff88885f8c0000(0000) 
> knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000008 CR3: 00000001092ff006 CR4: 0000000000370eb0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <TASK>
>  ? __die+0x20/0x60
>  ? page_fault_oops+0x150/0x3e0
>  ? exc_page_fault+0x74/0x130
>  ? asm_exc_page_fault+0x22/0x30
>  ? __page_cache_release+0xc7/0x260
>  ? __page_cache_release+0x84/0x260
>  __folio_put+0x43/0xe0
>  __filemap_get_folio+0x20c/0x2a0
>  ext4_da_write_begin+0xe1/0x240
>  generic_perform_write+0xe0/0x2c0
>  ext4_buffered_write_iter+0x62/0xe0
>  vfs_write+0x2c8/0x3f0
>  ksys_write+0x5f/0xe0
>  do_syscall_64+0x4c/0x100
>  entry_SYSCALL_64_after_hwframe+0x4b/0x53
> RIP: 0033:0x7fa91d9018b7
> Code: 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 
> 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 
> 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
> RSP: 002b:00007ffda039af78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
> RAX: ffffffffffffffda RBX: 0000000029e45270 RCX: 00007fa91d9018b7
> RDX: 0000000000000058 RSI: 0000000029e160f8 RDI: 0000000000000004
> RBP: 0000000029ce3700 R08: 00000000cccccccd R09: 0000000000000000
> R10: 0000000029e16142 R11: 0000000000000246 R12: 0000000000000058
> R13: 0000000029ce35d0 R14: 0000000000000000 R15: 0000000029e45270
>  </TASK>
> Modules linked in: xt_conntrack xt_MASQUERADE nf_conntrack_netlink 
> nfnetlink xt_addrtype iptable_nat nf_nat br_netfilter rpcsec_gss_krb5 
> auth_rpcgss oid_registry overlay mlx5_ib zram zsmalloc mlx5_core 
> rpcrdma rdma_ucm ib_uverbs ib_iser libiscsi scsi_transport_iscsi 
> ib_umad rdma_cm ib_ipoib iw_cm ib_cm fuse ib_core
> CR2: 0000000000000008
> ---[ end trace 0000000000000000 ]---
> RIP: 0010:__page_cache_release+0xc7/0x260
> Code: 8b 03 48 8b 53 08 48 c1 ed 12 83 e5 01 48 c1 e8 08 83 f5 01 83 
> e0 01 40 0f b6 ed 01 ed 3c 01 48 8b 43 10 83 dd ff 44 8d 7d 01 <48> 89 
> 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 43 08 48
> RSP: 0018:ffff888110197b78 EFLAGS: 00010013
> RAX: dead000000000122 RBX: ffffea0004e9aa00 RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffff888110197bc8 RDI: ffff8881001e1050
> RBP: 0000000000000002 R08: 000000000000005a R09: 00000000000009bc