On Fri, 30 Jun 2023 at 13:12, Tetsuo Handa <penguin-kernel@xxxxxxxxxxxxxxxxxxx> wrote: > > On 2023/06/30 19:18, Ard Biesheuvel wrote: > > On Fri, 30 Jun 2023 at 12:11, Alexander Potapenko <glider@xxxxxxxxxx> wrote: > >> > >> On Fri, Jun 30, 2023 at 12:02 PM Ard Biesheuvel <ardb@xxxxxxxxxx> wrote: > >>> > >>> On Fri, 30 Jun 2023 at 11:53, Tetsuo Handa > >>> <penguin-kernel@xxxxxxxxxxxxxxxxxxx> wrote: > >>>> > >>>> On 2023/06/30 18:36, Ard Biesheuvel wrote: > >>>>> Why are you sending this now? > >>>> > >>>> Just because this is currently top crasher and I can reproduce locally. > >>>> > >>>>> Do you have a reproducer for this issue? > >>>> > >>>> Yes. https://syzkaller.appspot.com/text?tag=ReproC&x=12931621900000 works. > >>>> > >>> > >>> Could you please share your kernel config and the resulting kernel log > >>> when running the reproducer? I'll try to reproduce locally as well, > >>> and see if I can figure out what is going on in the crypto layer > >> > >> The config together with the repro is available at > >> https://syzkaller.appspot.com/bug?extid=828dfc12440b4f6f305d, see the > >> latest row of the "Crashes" table that contains a C repro. > > Kernel is commit e6bc8833d80f of https://github.com/google/kmsan/commits/master . That commit does not exist in that repo. Does it matter? > Config is available in the dashboard page, but a smaller one is available at > https://I-love.SAKURA.ne.jp/tmp/config-6.4.0-rc7-kmsan . > Thanks - I'll try to rebuild v6.4-rc7 with that. > I'm using a debug printk() patch shown below. > > ---------------------------------------- > diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c > index 1a53c8f481e9..b32bb015995c 100644 > --- a/net/tls/tls_sw.c > +++ b/net/tls/tls_sw.c > @@ -1210,7 +1210,8 @@ static int tls_sw_do_sendpage(struct sock *sk, struct page *page, > if (!sk_stream_memory_free(sk)) > goto wait_for_sndbuf; > alloc_payload: > - ret = tls_alloc_encrypted_msg(sk, required_size); > + ret = tls_alloc_encrypted_msg(sk, required_size); ///// > + pr_info("required_size=%d ret=%d\n", required_size, ret); > if (ret) { > if (ret != -ENOSPC) > goto wait_for_memory; > @@ -1232,7 +1233,9 @@ static int tls_sw_do_sendpage(struct sock *sk, struct page *page, > > tls_ctx->pending_open_record_frags = true; > if (full_record || eor || sk_msg_full(msg_pl)) { > - ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, > + pr_info("full_record=%d eor=%d sk_msg_full(msg_pl)=%d copied=%d\n", > + full_record, eor, sk_msg_full(msg_pl), copied); > + ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, ///// > record_type, &copied, flags); > if (ret) { > if (ret == -EINPROGRESS) > ---------------------------------------- > > Output (on Ubuntu 22.04 on Oracle VM VirtualBox) is shown below. > Please check tendency of the sum of required_size= values up to the full_record= line. > It seems that the value of required_size= might vary depending on the timings, but > the sum of the values seems to have some rule. > > 4125+8221+12317+16413=41076 (the lower 4 bits are 0100) > 2461+6557+10653+14749+16413=50833 (the lower 4 bits are 0001) > 2461+6573+10669+14765+16413=50881 (the lower 4 bits are 0001) > > KMSAN reports this problem when the lower 4 bits became 0001 for the second time. > Unless KMSAN's reporting is asynchronous, maybe the reason of "for the second time" > part is that the previous state is relevant... > > ---------------------------------------- > [ 157.471712][ T3414] required_size=4125 ret=0 > [ 157.475879][ T3414] required_size=8221 ret=0 > [ 157.480471][ T3414] required_size=12317 ret=0 > [ 157.484604][ T3414] required_size=16413 ret=0 > [ 157.490499][ T3414] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=4096 > [ 157.513772][ T3414] required_size=4125 ret=0 > [ 157.523782][ T3414] required_size=8221 ret=0 > [ 157.533658][ T3414] required_size=12317 ret=0 > [ 157.539579][ T3414] required_size=16413 ret=0 > [ 157.543785][ T3414] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=4096 > [ 157.572869][ T3414] required_size=4125 ret=0 > [ 157.579350][ T3414] required_size=8221 ret=0 > [ 157.584699][ T3414] required_size=12317 ret=0 > [ 157.591756][ T3414] required_size=16413 ret=0 > [ 157.595891][ T3414] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=4096 > [ 157.790734][ T3424] required_size=2461 ret=0 > [ 157.800725][ T3424] required_size=6557 ret=0 > [ 157.804560][ T3424] required_size=10653 ret=0 > [ 157.808433][ T3424] required_size=14749 ret=0 > [ 157.810125][ T3424] required_size=16413 ret=0 > [ 157.829564][ T3424] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=1664 > [ 157.848397][ T3424] required_size=2461 ret=0 > [ 157.854875][ T3424] required_size=6573 ret=0 > [ 157.860883][ T3424] required_size=10669 ret=0 > [ 157.865463][ T3424] required_size=14765 ret=0 > [ 157.871794][ T3424] required_size=16413 ret=0 > [ 157.877333][ T3424] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=1648 > [ 157.885187][ T3424] ===================================================== > [ 157.887262][ T3424] BUG: KMSAN: uninit-value in aes_encrypt+0x1692/0x1fa0 > [ 157.887262][ T3424] aes_encrypt+0x1692/0x1fa0 > [ 157.887262][ T3424] aesti_encrypt+0xe1/0x160 > [ 157.887262][ T3424] crypto_cipher_encrypt_one+0x1d1/0x2e0 > [ 157.887262][ T3424] crypto_cbcmac_digest_update+0x3ff/0x5a0 > [ 157.887262][ T3424] shash_ahash_finup+0x79d/0xd00 > [ 157.887262][ T3424] shash_async_finup+0xbf/0x110 > [ 157.887262][ T3424] crypto_ahash_finup+0x244/0x500 > [ 157.887262][ T3424] crypto_ccm_auth+0x14df/0x15a0 > [ 157.887262][ T3424] crypto_ccm_encrypt+0x2ad/0x8b0 > [ 157.887262][ T3424] crypto_aead_encrypt+0x116/0x1a0 > [ 157.887262][ T3424] tls_push_record+0x2bbe/0x3bf0 > [ 157.887262][ T3424] bpf_exec_tx_verdict+0x5ba/0x2530 > [ 157.887262][ T3424] tls_sw_do_sendpage+0x1779/0x21f0 > [ 157.887262][ T3424] tls_sw_sendpage+0x247/0x2b0 > [ 157.887262][ T3424] inet_sendpage+0x1de/0x2f0 > [ 157.887262][ T3424] kernel_sendpage+0x4cc/0x940 > [ 158.004827][ T3424] sock_sendpage+0x162/0x220 > [ 158.004827][ T3424] pipe_to_sendpage+0x3df/0x4f0 > [ 158.004827][ T3424] __splice_from_pipe+0x5c7/0x1010 > [ 158.004827][ T3424] generic_splice_sendpage+0x1c6/0x2a0 > [ 158.004827][ T3424] do_splice+0x26d8/0x32f0 > [ 158.004827][ T3424] __se_sys_splice+0x81f/0xba0 > [ 158.004827][ T3424] __x64_sys_splice+0x1a1/0x200 > [ 158.004827][ T3424] do_syscall_64+0x41/0x90 > [ 158.004827][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd > [ 158.004827][ T3424] > [ 158.004827][ T3424] Uninit was stored to memory at: > [ 158.004827][ T3424] __crypto_xor+0x285/0x1700 > [ 158.004827][ T3424] crypto_cbcmac_digest_update+0x2ba/0x5a0 > [ 158.004827][ T3424] shash_ahash_finup+0x79d/0xd00 > [ 158.004827][ T3424] shash_async_finup+0xbf/0x110 > [ 158.004827][ T3424] crypto_ahash_finup+0x244/0x500 > [ 158.004827][ T3424] crypto_ccm_auth+0x14df/0x15a0 > [ 158.004827][ T3424] crypto_ccm_encrypt+0x2ad/0x8b0 > [ 158.004827][ T3424] crypto_aead_encrypt+0x116/0x1a0 > [ 158.004827][ T3424] tls_push_record+0x2bbe/0x3bf0 > [ 158.004827][ T3424] bpf_exec_tx_verdict+0x5ba/0x2530 > [ 158.004827][ T3424] tls_sw_do_sendpage+0x1779/0x21f0 > [ 158.004827][ T3424] tls_sw_sendpage+0x247/0x2b0 > [ 158.004827][ T3424] inet_sendpage+0x1de/0x2f0 > [ 158.004827][ T3424] kernel_sendpage+0x4cc/0x940 > [ 158.004827][ T3424] sock_sendpage+0x162/0x220 > [ 158.004827][ T3424] pipe_to_sendpage+0x3df/0x4f0 > [ 158.004827][ T3424] __splice_from_pipe+0x5c7/0x1010 > [ 158.004827][ T3424] generic_splice_sendpage+0x1c6/0x2a0 > [ 158.004827][ T3424] do_splice+0x26d8/0x32f0 > [ 158.004827][ T3424] __se_sys_splice+0x81f/0xba0 > [ 158.004827][ T3424] __x64_sys_splice+0x1a1/0x200 > [ 158.004827][ T3424] do_syscall_64+0x41/0x90 > [ 158.004827][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd > [ 158.004827][ T3424] > [ 158.004827][ T3424] Uninit was created at: > [ 158.004827][ T3424] __alloc_pages+0x925/0x1050 > [ 158.004827][ T3424] alloc_pages+0xe30/0x11b0 > [ 158.004827][ T3424] skb_page_frag_refill+0x362/0x910 > [ 158.004827][ T3424] sk_page_frag_refill+0xa2/0x1c0 > [ 158.004827][ T3424] sk_msg_alloc+0x278/0x1560 > [ 158.004827][ T3424] tls_sw_do_sendpage+0xbec/0x21f0 > [ 158.004827][ T3424] tls_sw_sendpage+0x247/0x2b0 > [ 158.004827][ T3424] inet_sendpage+0x1de/0x2f0 > [ 158.004827][ T3424] kernel_sendpage+0x4cc/0x940 > [ 158.004827][ T3424] sock_sendpage+0x162/0x220 > [ 158.004827][ T3424] pipe_to_sendpage+0x3df/0x4f0 > [ 158.004827][ T3424] __splice_from_pipe+0x5c7/0x1010 > [ 158.004827][ T3424] generic_splice_sendpage+0x1c6/0x2a0 > [ 158.260226][ T3424] do_splice+0x26d8/0x32f0 > [ 158.260226][ T3424] __se_sys_splice+0x81f/0xba0 > [ 158.260226][ T3424] __x64_sys_splice+0x1a1/0x200 > [ 158.260226][ T3424] do_syscall_64+0x41/0x90 > [ 158.260226][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd > [ 158.260226][ T3424] > [ 158.260226][ T3424] CPU: 7 PID: 3424 Comm: a.out Not tainted 6.4.0-rc7-ge6bc8833d80f-dirty #26 > [ 158.260226][ T3424] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 > [ 158.260226][ T3424] ===================================================== > [ 158.260226][ T3424] Disabling lock debugging due to kernel taint > [ 158.260226][ T3424] Kernel panic - not syncing: kmsan.panic set ... > [ 158.260226][ T3424] CPU: 7 PID: 3424 Comm: a.out Tainted: G B 6.4.0-rc7-ge6bc8833d80f-dirty #26 > [ 158.320898][ T3424] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 > [ 158.334186][ T3424] Call Trace: > [ 158.334186][ T3424] <TASK> > [ 158.334186][ T3424] dump_stack_lvl+0x1f6/0x280 > [ 158.334186][ T3424] dump_stack+0x29/0x30 > [ 158.334186][ T3424] panic+0x4e7/0xc60 > [ 158.334186][ T3424] ? add_taint+0x185/0x210 > [ 158.334186][ T3424] kmsan_report+0x2d1/0x2e0 > [ 158.334186][ T3424] ? __msan_warning+0x98/0x120 > [ 158.334186][ T3424] ? aes_encrypt+0x1692/0x1fa0 > [ 158.334186][ T3424] ? aesti_encrypt+0xe1/0x160 > [ 158.334186][ T3424] ? crypto_cipher_encrypt_one+0x1d1/0x2e0 > [ 158.334186][ T3424] ? crypto_cbcmac_digest_update+0x3ff/0x5a0 > [ 158.334186][ T3424] ? shash_ahash_finup+0x79d/0xd00 > [ 158.334186][ T3424] ? shash_async_finup+0xbf/0x110 > [ 158.334186][ T3424] ? crypto_ahash_finup+0x244/0x500 > [ 158.334186][ T3424] ? crypto_ccm_auth+0x14df/0x15a0 > [ 158.334186][ T3424] ? crypto_ccm_encrypt+0x2ad/0x8b0 > [ 158.334186][ T3424] ? crypto_aead_encrypt+0x116/0x1a0 > [ 158.334186][ T3424] ? tls_push_record+0x2bbe/0x3bf0 > [ 158.334186][ T3424] ? bpf_exec_tx_verdict+0x5ba/0x2530 > [ 158.334186][ T3424] ? tls_sw_do_sendpage+0x1779/0x21f0 > [ 158.334186][ T3424] ? tls_sw_sendpage+0x247/0x2b0 > [ 158.334186][ T3424] ? inet_sendpage+0x1de/0x2f0 > [ 158.334186][ T3424] ? kernel_sendpage+0x4cc/0x940 > [ 158.334186][ T3424] ? sock_sendpage+0x162/0x220 > [ 158.334186][ T3424] ? pipe_to_sendpage+0x3df/0x4f0 > [ 158.334186][ T3424] ? __splice_from_pipe+0x5c7/0x1010 > [ 158.334186][ T3424] ? generic_splice_sendpage+0x1c6/0x2a0 > [ 158.334186][ T3424] ? do_splice+0x26d8/0x32f0 > [ 158.334186][ T3424] ? __se_sys_splice+0x81f/0xba0 > [ 158.334186][ T3424] ? __x64_sys_splice+0x1a1/0x200 > [ 158.334186][ T3424] ? do_syscall_64+0x41/0x90 > [ 158.334186][ T3424] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd > [ 158.334186][ T3424] ? filter_irq_stacks+0xb9/0x230 > [ 158.334186][ T3424] ? __stack_depot_save+0x22/0x490 > [ 158.334186][ T3424] ? kmsan_internal_set_shadow_origin+0x66/0xe0 > [ 158.334186][ T3424] ? kmsan_internal_chain_origin+0x110/0x120 > [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0 > [ 158.334186][ T3424] __msan_warning+0x98/0x120 > [ 158.334186][ T3424] aes_encrypt+0x1692/0x1fa0 > [ 158.334186][ T3424] aesti_encrypt+0xe1/0x160 > [ 158.334186][ T3424] crypto_cipher_encrypt_one+0x1d1/0x2e0 > [ 158.334186][ T3424] ? aesti_set_key+0xb0/0xb0 > [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0 > [ 158.334186][ T3424] crypto_cbcmac_digest_update+0x3ff/0x5a0 > [ 158.334186][ T3424] ? crypto_cbcmac_digest_init+0x140/0x140 > [ 158.334186][ T3424] shash_ahash_finup+0x79d/0xd00 > [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0 > [ 158.334186][ T3424] shash_async_finup+0xbf/0x110 > [ 158.334186][ T3424] crypto_ahash_finup+0x244/0x500 > [ 158.334186][ T3424] ? shash_async_final+0x3d0/0x3d0 > [ 158.334186][ T3424] crypto_ccm_auth+0x14df/0x15a0 > [ 158.334186][ T3424] crypto_ccm_encrypt+0x2ad/0x8b0 > [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0 > [ 158.334186][ T3424] ? crypto_ccm_setauthsize+0x100/0x100 > [ 158.334186][ T3424] crypto_aead_encrypt+0x116/0x1a0 > [ 158.653332][ T3424] tls_push_record+0x2bbe/0x3bf0 > [ 158.653332][ T3424] bpf_exec_tx_verdict+0x5ba/0x2530 > [ 158.653332][ T3424] ? _printk+0x181/0x1b0 > [ 158.653332][ T3424] ? tls_sw_do_sendpage+0xc81/0x21f0 > [ 158.653332][ T3424] tls_sw_do_sendpage+0x1779/0x21f0 > [ 158.653332][ T3424] tls_sw_sendpage+0x247/0x2b0 > [ 158.653332][ T3424] ? tls_sw_do_sendpage+0x21f0/0x21f0 > [ 158.653332][ T3424] inet_sendpage+0x1de/0x2f0 > [ 158.653332][ T3424] ? inet_sendmsg+0x1d0/0x1d0 > [ 158.653332][ T3424] kernel_sendpage+0x4cc/0x940 > [ 158.653332][ T3424] sock_sendpage+0x162/0x220 > [ 158.653332][ T3424] pipe_to_sendpage+0x3df/0x4f0 > [ 158.653332][ T3424] ? sock_fasync+0x240/0x240 > [ 158.653332][ T3424] __splice_from_pipe+0x5c7/0x1010 > [ 158.653332][ T3424] ? generic_splice_sendpage+0x2a0/0x2a0 > [ 158.653332][ T3424] generic_splice_sendpage+0x1c6/0x2a0 > [ 158.653332][ T3424] ? iter_file_splice_write+0x1a30/0x1a30 > [ 158.653332][ T3424] do_splice+0x26d8/0x32f0 > [ 158.653332][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0 > [ 158.653332][ T3424] ? __se_sys_splice+0x292/0xba0 > [ 158.653332][ T3424] ? __msan_metadata_ptr_for_load_8+0x24/0x40 > [ 158.653332][ T3424] ? filter_irq_stacks+0xb9/0x230 > [ 158.653332][ T3424] __se_sys_splice+0x81f/0xba0 > [ 158.870673][ T3424] __x64_sys_splice+0x1a1/0x200 > [ 158.870673][ T3424] do_syscall_64+0x41/0x90 > [ 158.870673][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd > [ 158.870673][ T3424] RIP: 0033:0x7f6bbd51ea3d > [ 158.895223][ T3424] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c3 a3 0f 00 f7 d8 64 89 01 48 > [ 158.895223][ T3424] RSP: 002b:00007f6bbd731e08 EFLAGS: 00000246 ORIG_RAX: 0000000000000113 > [ 158.895223][ T3424] RAX: ffffffffffffffda RBX: 000055ccd9ea6080 RCX: 00007f6bbd51ea3d > [ 158.895223][ T3424] RDX: 0000000000000004 RSI: 0000000000000000 RDI: 0000000000000003 > [ 158.895223][ T3424] RBP: 000055ccd9ea41f4 R08: 00080000fffffffc R09: 0000000000000000 > [ 158.895223][ T3424] R10: 0000000000000000 R11: 0000000000000246 R12: 0100000000000000 > [ 158.895223][ T3424] R13: e65b75b4ec4292eb R14: f2300cdb85a45425 R15: 000055ccd9ea6088 > [ 159.041467][ T3424] </TASK> > [ 159.041467][ T3424] Kernel Offset: disabled > [ 159.041467][ T3424] Rebooting in 10 seconds.. > ---------------------------------------- > > > > > Could you explain why that bug contains ~50 reports that seem entirely > > unrelated? AIUI, this actual issue has not been reproduced since > > 2020?? > > Multiple different bugs are reported as the same problem. > Reproducer is available for only bpf_exec_tx_verdict() one, and the reproducer still works. > > > > > > >> Config: https://syzkaller.appspot.com/text?tag=KernelConfig&x=ee5f7a0b2e48ed66 > >> Report: https://syzkaller.appspot.com/text?tag=CrashReport&x=1325260d900000 > >> Syz repro: https://syzkaller.appspot.com/text?tag=ReproSyz&x=11af973e900000 > >> C repro: https://syzkaller.appspot.com/text?tag=ReproC&x=163a1e45900000 > >> > >> The bug is reproducible for me locally as well (and Tetsuo's patch > >> makes it disappear, although I have no opinion on its correctness). > > > > What I'd like to do is run a kernel plus initrd locally in OVMF and > > reproduce the issue - can I do that without all the syzkaller > > machinery? > > I'm using Ubuntu 22.04 on Oracle VM VirtualBox. > I don't know if this can be reproduced with kernel plus initrd only. But > since the C reproducer is standalone, syzkaller machinery is not involved. >
