On 2023/06/30 19:18, Ard Biesheuvel wrote: > On Fri, 30 Jun 2023 at 12:11, Alexander Potapenko <glider@xxxxxxxxxx> wrote: >> >> On Fri, Jun 30, 2023 at 12:02 PM Ard Biesheuvel <ardb@xxxxxxxxxx> wrote: >>> >>> On Fri, 30 Jun 2023 at 11:53, Tetsuo Handa >>> <penguin-kernel@xxxxxxxxxxxxxxxxxxx> wrote: >>>> >>>> On 2023/06/30 18:36, Ard Biesheuvel wrote: >>>>> Why are you sending this now? >>>> >>>> Just because this is currently top crasher and I can reproduce locally. >>>> >>>>> Do you have a reproducer for this issue? >>>> >>>> Yes. https://syzkaller.appspot.com/text?tag=ReproC&x=12931621900000 works. >>>> >>> >>> Could you please share your kernel config and the resulting kernel log >>> when running the reproducer? I'll try to reproduce locally as well, >>> and see if I can figure out what is going on in the crypto layer >> >> The config together with the repro is available at >> https://syzkaller.appspot.com/bug?extid=828dfc12440b4f6f305d, see the >> latest row of the "Crashes" table that contains a C repro. Kernel is commit e6bc8833d80f of https://github.com/google/kmsan/commits/master . Config is available in the dashboard page, but a smaller one is available at https://I-love.SAKURA.ne.jp/tmp/config-6.4.0-rc7-kmsan . I'm using a debug printk() patch shown below. ---------------------------------------- diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 1a53c8f481e9..b32bb015995c 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -1210,7 +1210,8 @@ static int tls_sw_do_sendpage(struct sock *sk, struct page *page, if (!sk_stream_memory_free(sk)) goto wait_for_sndbuf; alloc_payload: - ret = tls_alloc_encrypted_msg(sk, required_size); + ret = tls_alloc_encrypted_msg(sk, required_size); ///// + pr_info("required_size=%d ret=%d\n", required_size, ret); if (ret) { if (ret != -ENOSPC) goto wait_for_memory; @@ -1232,7 +1233,9 @@ static int tls_sw_do_sendpage(struct sock *sk, struct page *page, tls_ctx->pending_open_record_frags = true; if (full_record || eor || sk_msg_full(msg_pl)) { - ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, + pr_info("full_record=%d eor=%d sk_msg_full(msg_pl)=%d copied=%d\n", + full_record, eor, sk_msg_full(msg_pl), copied); + ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, ///// record_type, &copied, flags); if (ret) { if (ret == -EINPROGRESS) ---------------------------------------- Output (on Ubuntu 22.04 on Oracle VM VirtualBox) is shown below. Please check tendency of the sum of required_size= values up to the full_record= line. It seems that the value of required_size= might vary depending on the timings, but the sum of the values seems to have some rule. 4125+8221+12317+16413=41076 (the lower 4 bits are 0100) 2461+6557+10653+14749+16413=50833 (the lower 4 bits are 0001) 2461+6573+10669+14765+16413=50881 (the lower 4 bits are 0001) KMSAN reports this problem when the lower 4 bits became 0001 for the second time. Unless KMSAN's reporting is asynchronous, maybe the reason of "for the second time" part is that the previous state is relevant... ---------------------------------------- [ 157.471712][ T3414] required_size=4125 ret=0 [ 157.475879][ T3414] required_size=8221 ret=0 [ 157.480471][ T3414] required_size=12317 ret=0 [ 157.484604][ T3414] required_size=16413 ret=0 [ 157.490499][ T3414] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=4096 [ 157.513772][ T3414] required_size=4125 ret=0 [ 157.523782][ T3414] required_size=8221 ret=0 [ 157.533658][ T3414] required_size=12317 ret=0 [ 157.539579][ T3414] required_size=16413 ret=0 [ 157.543785][ T3414] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=4096 [ 157.572869][ T3414] required_size=4125 ret=0 [ 157.579350][ T3414] required_size=8221 ret=0 [ 157.584699][ T3414] required_size=12317 ret=0 [ 157.591756][ T3414] required_size=16413 ret=0 [ 157.595891][ T3414] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=4096 [ 157.790734][ T3424] required_size=2461 ret=0 [ 157.800725][ T3424] required_size=6557 ret=0 [ 157.804560][ T3424] required_size=10653 ret=0 [ 157.808433][ T3424] required_size=14749 ret=0 [ 157.810125][ T3424] required_size=16413 ret=0 [ 157.829564][ T3424] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=1664 [ 157.848397][ T3424] required_size=2461 ret=0 [ 157.854875][ T3424] required_size=6573 ret=0 [ 157.860883][ T3424] required_size=10669 ret=0 [ 157.865463][ T3424] required_size=14765 ret=0 [ 157.871794][ T3424] required_size=16413 ret=0 [ 157.877333][ T3424] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=1648 [ 157.885187][ T3424] ===================================================== [ 157.887262][ T3424] BUG: KMSAN: uninit-value in aes_encrypt+0x1692/0x1fa0 [ 157.887262][ T3424] aes_encrypt+0x1692/0x1fa0 [ 157.887262][ T3424] aesti_encrypt+0xe1/0x160 [ 157.887262][ T3424] crypto_cipher_encrypt_one+0x1d1/0x2e0 [ 157.887262][ T3424] crypto_cbcmac_digest_update+0x3ff/0x5a0 [ 157.887262][ T3424] shash_ahash_finup+0x79d/0xd00 [ 157.887262][ T3424] shash_async_finup+0xbf/0x110 [ 157.887262][ T3424] crypto_ahash_finup+0x244/0x500 [ 157.887262][ T3424] crypto_ccm_auth+0x14df/0x15a0 [ 157.887262][ T3424] crypto_ccm_encrypt+0x2ad/0x8b0 [ 157.887262][ T3424] crypto_aead_encrypt+0x116/0x1a0 [ 157.887262][ T3424] tls_push_record+0x2bbe/0x3bf0 [ 157.887262][ T3424] bpf_exec_tx_verdict+0x5ba/0x2530 [ 157.887262][ T3424] tls_sw_do_sendpage+0x1779/0x21f0 [ 157.887262][ T3424] tls_sw_sendpage+0x247/0x2b0 [ 157.887262][ T3424] inet_sendpage+0x1de/0x2f0 [ 157.887262][ T3424] kernel_sendpage+0x4cc/0x940 [ 158.004827][ T3424] sock_sendpage+0x162/0x220 [ 158.004827][ T3424] pipe_to_sendpage+0x3df/0x4f0 [ 158.004827][ T3424] __splice_from_pipe+0x5c7/0x1010 [ 158.004827][ T3424] generic_splice_sendpage+0x1c6/0x2a0 [ 158.004827][ T3424] do_splice+0x26d8/0x32f0 [ 158.004827][ T3424] __se_sys_splice+0x81f/0xba0 [ 158.004827][ T3424] __x64_sys_splice+0x1a1/0x200 [ 158.004827][ T3424] do_syscall_64+0x41/0x90 [ 158.004827][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 158.004827][ T3424] [ 158.004827][ T3424] Uninit was stored to memory at: [ 158.004827][ T3424] __crypto_xor+0x285/0x1700 [ 158.004827][ T3424] crypto_cbcmac_digest_update+0x2ba/0x5a0 [ 158.004827][ T3424] shash_ahash_finup+0x79d/0xd00 [ 158.004827][ T3424] shash_async_finup+0xbf/0x110 [ 158.004827][ T3424] crypto_ahash_finup+0x244/0x500 [ 158.004827][ T3424] crypto_ccm_auth+0x14df/0x15a0 [ 158.004827][ T3424] crypto_ccm_encrypt+0x2ad/0x8b0 [ 158.004827][ T3424] crypto_aead_encrypt+0x116/0x1a0 [ 158.004827][ T3424] tls_push_record+0x2bbe/0x3bf0 [ 158.004827][ T3424] bpf_exec_tx_verdict+0x5ba/0x2530 [ 158.004827][ T3424] tls_sw_do_sendpage+0x1779/0x21f0 [ 158.004827][ T3424] tls_sw_sendpage+0x247/0x2b0 [ 158.004827][ T3424] inet_sendpage+0x1de/0x2f0 [ 158.004827][ T3424] kernel_sendpage+0x4cc/0x940 [ 158.004827][ T3424] sock_sendpage+0x162/0x220 [ 158.004827][ T3424] pipe_to_sendpage+0x3df/0x4f0 [ 158.004827][ T3424] __splice_from_pipe+0x5c7/0x1010 [ 158.004827][ T3424] generic_splice_sendpage+0x1c6/0x2a0 [ 158.004827][ T3424] do_splice+0x26d8/0x32f0 [ 158.004827][ T3424] __se_sys_splice+0x81f/0xba0 [ 158.004827][ T3424] __x64_sys_splice+0x1a1/0x200 [ 158.004827][ T3424] do_syscall_64+0x41/0x90 [ 158.004827][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 158.004827][ T3424] [ 158.004827][ T3424] Uninit was created at: [ 158.004827][ T3424] __alloc_pages+0x925/0x1050 [ 158.004827][ T3424] alloc_pages+0xe30/0x11b0 [ 158.004827][ T3424] skb_page_frag_refill+0x362/0x910 [ 158.004827][ T3424] sk_page_frag_refill+0xa2/0x1c0 [ 158.004827][ T3424] sk_msg_alloc+0x278/0x1560 [ 158.004827][ T3424] tls_sw_do_sendpage+0xbec/0x21f0 [ 158.004827][ T3424] tls_sw_sendpage+0x247/0x2b0 [ 158.004827][ T3424] inet_sendpage+0x1de/0x2f0 [ 158.004827][ T3424] kernel_sendpage+0x4cc/0x940 [ 158.004827][ T3424] sock_sendpage+0x162/0x220 [ 158.004827][ T3424] pipe_to_sendpage+0x3df/0x4f0 [ 158.004827][ T3424] __splice_from_pipe+0x5c7/0x1010 [ 158.004827][ T3424] generic_splice_sendpage+0x1c6/0x2a0 [ 158.260226][ T3424] do_splice+0x26d8/0x32f0 [ 158.260226][ T3424] __se_sys_splice+0x81f/0xba0 [ 158.260226][ T3424] __x64_sys_splice+0x1a1/0x200 [ 158.260226][ T3424] do_syscall_64+0x41/0x90 [ 158.260226][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 158.260226][ T3424] [ 158.260226][ T3424] CPU: 7 PID: 3424 Comm: a.out Not tainted 6.4.0-rc7-ge6bc8833d80f-dirty #26 [ 158.260226][ T3424] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 158.260226][ T3424] ===================================================== [ 158.260226][ T3424] Disabling lock debugging due to kernel taint [ 158.260226][ T3424] Kernel panic - not syncing: kmsan.panic set ... [ 158.260226][ T3424] CPU: 7 PID: 3424 Comm: a.out Tainted: G B 6.4.0-rc7-ge6bc8833d80f-dirty #26 [ 158.320898][ T3424] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 158.334186][ T3424] Call Trace: [ 158.334186][ T3424] <TASK> [ 158.334186][ T3424] dump_stack_lvl+0x1f6/0x280 [ 158.334186][ T3424] dump_stack+0x29/0x30 [ 158.334186][ T3424] panic+0x4e7/0xc60 [ 158.334186][ T3424] ? add_taint+0x185/0x210 [ 158.334186][ T3424] kmsan_report+0x2d1/0x2e0 [ 158.334186][ T3424] ? __msan_warning+0x98/0x120 [ 158.334186][ T3424] ? aes_encrypt+0x1692/0x1fa0 [ 158.334186][ T3424] ? aesti_encrypt+0xe1/0x160 [ 158.334186][ T3424] ? crypto_cipher_encrypt_one+0x1d1/0x2e0 [ 158.334186][ T3424] ? crypto_cbcmac_digest_update+0x3ff/0x5a0 [ 158.334186][ T3424] ? shash_ahash_finup+0x79d/0xd00 [ 158.334186][ T3424] ? shash_async_finup+0xbf/0x110 [ 158.334186][ T3424] ? crypto_ahash_finup+0x244/0x500 [ 158.334186][ T3424] ? crypto_ccm_auth+0x14df/0x15a0 [ 158.334186][ T3424] ? crypto_ccm_encrypt+0x2ad/0x8b0 [ 158.334186][ T3424] ? crypto_aead_encrypt+0x116/0x1a0 [ 158.334186][ T3424] ? tls_push_record+0x2bbe/0x3bf0 [ 158.334186][ T3424] ? bpf_exec_tx_verdict+0x5ba/0x2530 [ 158.334186][ T3424] ? tls_sw_do_sendpage+0x1779/0x21f0 [ 158.334186][ T3424] ? tls_sw_sendpage+0x247/0x2b0 [ 158.334186][ T3424] ? inet_sendpage+0x1de/0x2f0 [ 158.334186][ T3424] ? kernel_sendpage+0x4cc/0x940 [ 158.334186][ T3424] ? sock_sendpage+0x162/0x220 [ 158.334186][ T3424] ? pipe_to_sendpage+0x3df/0x4f0 [ 158.334186][ T3424] ? __splice_from_pipe+0x5c7/0x1010 [ 158.334186][ T3424] ? generic_splice_sendpage+0x1c6/0x2a0 [ 158.334186][ T3424] ? do_splice+0x26d8/0x32f0 [ 158.334186][ T3424] ? __se_sys_splice+0x81f/0xba0 [ 158.334186][ T3424] ? __x64_sys_splice+0x1a1/0x200 [ 158.334186][ T3424] ? do_syscall_64+0x41/0x90 [ 158.334186][ T3424] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 158.334186][ T3424] ? filter_irq_stacks+0xb9/0x230 [ 158.334186][ T3424] ? __stack_depot_save+0x22/0x490 [ 158.334186][ T3424] ? kmsan_internal_set_shadow_origin+0x66/0xe0 [ 158.334186][ T3424] ? kmsan_internal_chain_origin+0x110/0x120 [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0 [ 158.334186][ T3424] __msan_warning+0x98/0x120 [ 158.334186][ T3424] aes_encrypt+0x1692/0x1fa0 [ 158.334186][ T3424] aesti_encrypt+0xe1/0x160 [ 158.334186][ T3424] crypto_cipher_encrypt_one+0x1d1/0x2e0 [ 158.334186][ T3424] ? aesti_set_key+0xb0/0xb0 [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0 [ 158.334186][ T3424] crypto_cbcmac_digest_update+0x3ff/0x5a0 [ 158.334186][ T3424] ? crypto_cbcmac_digest_init+0x140/0x140 [ 158.334186][ T3424] shash_ahash_finup+0x79d/0xd00 [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0 [ 158.334186][ T3424] shash_async_finup+0xbf/0x110 [ 158.334186][ T3424] crypto_ahash_finup+0x244/0x500 [ 158.334186][ T3424] ? shash_async_final+0x3d0/0x3d0 [ 158.334186][ T3424] crypto_ccm_auth+0x14df/0x15a0 [ 158.334186][ T3424] crypto_ccm_encrypt+0x2ad/0x8b0 [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0 [ 158.334186][ T3424] ? crypto_ccm_setauthsize+0x100/0x100 [ 158.334186][ T3424] crypto_aead_encrypt+0x116/0x1a0 [ 158.653332][ T3424] tls_push_record+0x2bbe/0x3bf0 [ 158.653332][ T3424] bpf_exec_tx_verdict+0x5ba/0x2530 [ 158.653332][ T3424] ? _printk+0x181/0x1b0 [ 158.653332][ T3424] ? tls_sw_do_sendpage+0xc81/0x21f0 [ 158.653332][ T3424] tls_sw_do_sendpage+0x1779/0x21f0 [ 158.653332][ T3424] tls_sw_sendpage+0x247/0x2b0 [ 158.653332][ T3424] ? tls_sw_do_sendpage+0x21f0/0x21f0 [ 158.653332][ T3424] inet_sendpage+0x1de/0x2f0 [ 158.653332][ T3424] ? inet_sendmsg+0x1d0/0x1d0 [ 158.653332][ T3424] kernel_sendpage+0x4cc/0x940 [ 158.653332][ T3424] sock_sendpage+0x162/0x220 [ 158.653332][ T3424] pipe_to_sendpage+0x3df/0x4f0 [ 158.653332][ T3424] ? sock_fasync+0x240/0x240 [ 158.653332][ T3424] __splice_from_pipe+0x5c7/0x1010 [ 158.653332][ T3424] ? generic_splice_sendpage+0x2a0/0x2a0 [ 158.653332][ T3424] generic_splice_sendpage+0x1c6/0x2a0 [ 158.653332][ T3424] ? iter_file_splice_write+0x1a30/0x1a30 [ 158.653332][ T3424] do_splice+0x26d8/0x32f0 [ 158.653332][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0 [ 158.653332][ T3424] ? __se_sys_splice+0x292/0xba0 [ 158.653332][ T3424] ? __msan_metadata_ptr_for_load_8+0x24/0x40 [ 158.653332][ T3424] ? filter_irq_stacks+0xb9/0x230 [ 158.653332][ T3424] __se_sys_splice+0x81f/0xba0 [ 158.870673][ T3424] __x64_sys_splice+0x1a1/0x200 [ 158.870673][ T3424] do_syscall_64+0x41/0x90 [ 158.870673][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 158.870673][ T3424] RIP: 0033:0x7f6bbd51ea3d [ 158.895223][ T3424] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c3 a3 0f 00 f7 d8 64 89 01 48 [ 158.895223][ T3424] RSP: 002b:00007f6bbd731e08 EFLAGS: 00000246 ORIG_RAX: 0000000000000113 [ 158.895223][ T3424] RAX: ffffffffffffffda RBX: 000055ccd9ea6080 RCX: 00007f6bbd51ea3d [ 158.895223][ T3424] RDX: 0000000000000004 RSI: 0000000000000000 RDI: 0000000000000003 [ 158.895223][ T3424] RBP: 000055ccd9ea41f4 R08: 00080000fffffffc R09: 0000000000000000 [ 158.895223][ T3424] R10: 0000000000000000 R11: 0000000000000246 R12: 0100000000000000 [ 158.895223][ T3424] R13: e65b75b4ec4292eb R14: f2300cdb85a45425 R15: 000055ccd9ea6088 [ 159.041467][ T3424] </TASK> [ 159.041467][ T3424] Kernel Offset: disabled [ 159.041467][ T3424] Rebooting in 10 seconds.. ---------------------------------------- > > Could you explain why that bug contains ~50 reports that seem entirely > unrelated? AIUI, this actual issue has not been reproduced since > 2020?? Multiple different bugs are reported as the same problem. Reproducer is available for only bpf_exec_tx_verdict() one, and the reproducer still works. > > >> Config: https://syzkaller.appspot.com/text?tag=KernelConfig&x=ee5f7a0b2e48ed66 >> Report: https://syzkaller.appspot.com/text?tag=CrashReport&x=1325260d900000 >> Syz repro: https://syzkaller.appspot.com/text?tag=ReproSyz&x=11af973e900000 >> C repro: https://syzkaller.appspot.com/text?tag=ReproC&x=163a1e45900000 >> >> The bug is reproducible for me locally as well (and Tetsuo's patch >> makes it disappear, although I have no opinion on its correctness). > > What I'd like to do is run a kernel plus initrd locally in OVMF and > reproduce the issue - can I do that without all the syzkaller > machinery? I'm using Ubuntu 22.04 on Oracle VM VirtualBox. I don't know if this can be reproduced with kernel plus initrd only. But since the C reproducer is standalone, syzkaller machinery is not involved.
