On Sun, Oct 13, 2024 at 10:22:11PM -0400, Liam R. Howlett wrote: > * Sasha Levin <sashal@xxxxxxxxxx> [241013 09:29]: > > On Thu, Oct 10, 2024 at 04:28:18PM +0100, Lorenzo Stoakes wrote: > > > On Thu, Oct 10, 2024 at 08:19:28AM -0700, syzbot wrote: > > > > Hello, > > > > > > > > syzbot found the following issue on: > > > > > > > > HEAD commit: d3d1556696c1 Merge tag 'mm-hotfixes-stable-2024-10-09-15-4.. > > > > git tree: upstream > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=10416fd0580000 > > > > kernel config: https://syzkaller.appspot.com/x/.config?x=7a3fccdd0bb995 > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=39bc767144c55c8db0ea > > > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > > > Downloadable assets: > > > > disk image: https://storage.googleapis.com/syzbot-assets/0600b551e610/disk-d3d15566.raw.xz > > > > vmlinux: https://storage.googleapis.com/syzbot-assets/d59d43ed3976/vmlinux-d3d15566.xz > > > > kernel image: https://storage.googleapis.com/syzbot-assets/e686a3e7e0d6/bzImage-d3d15566.xz > > > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > > Reported-by: syzbot+39bc767144c55c8db0ea@xxxxxxxxxxxxxxxxxxxxxxxxx > > > > > > > > INFO: task syz.3.917:7739 blocked for more than 146 seconds. > > > > Not tainted 6.12.0-rc2-syzkaller-00074-gd3d1556696c1 #0 > > > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > > task:syz.3.917 state:D stack:23808 pid:7739 tgid:7739 ppid:5232 flags:0x00004000 > > > > Call Trace: > > > > <TASK> > > > > context_switch kernel/sched/core.c:5322 [inline] > > > > __schedule+0x1843/0x4ae0 kernel/sched/core.c:6682 > > > > __schedule_loop kernel/sched/core.c:6759 [inline] > > > > schedule+0x14b/0x320 kernel/sched/core.c:6774 > > > > schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6831 > > > > rwsem_down_write_slowpath+0xeee/0x13b0 kernel/locking/rwsem.c:1176 > > > > __down_write_common kernel/locking/rwsem.c:1304 [inline] > > > > __down_write kernel/locking/rwsem.c:1313 [inline] > > > > down_write+0x1d7/0x220 kernel/locking/rwsem.c:1578 > > > > mmap_write_lock include/linux/mmap_lock.h:106 [inline] > > > > exit_mmap+0x2bd/0xc40 mm/mmap.c:1872 > > > > > > Hmm, task freezing up or system becoming unstable/locked up is reminsecent > > > of the maple tree bug I fixed in [0], which is still in the unstable hotfix > > > branch. > > > > > > This is likely not going to repro as it's quite heisenbug-ish to trigger > > > and the failures are like this - somewhat disconnected from the cause, so > > > not sure if there is any case to speed this to Linus's tree. > > > > > > On the other hand it's a pretty serious problem for stability and likely to > > > continue to manifest in nasty ways like this. > > > > > > Can't be 100% sure this is the cause, but seems likely. > > > > > > [0]:https://lore.kernel.org/linux-mm/48b349a2a0f7c76e18772712d0997a5e12ab0a3b.1728314403.git.lorenzo.stoakes@xxxxxxxxxx/ > > > > On my Debian build box, running a 6.1 kernel, I've started hitting a > > similar issue: > > > > Oct 12 17:24:01 debian kernel: INFO: task sed:3557356 blocked for more than 1208 seconds. > > Oct 12 17:24:01 debian kernel: Not tainted 6.1.0-26-amd64 #1 Debian 6.1.112-1 > > Oct 12 17:24:01 debian kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > Oct 12 17:24:01 debian kernel: task:sed state:D stack:0 pid:3557356 ppid:1 flags:0x00000002 > > Oct 12 17:24:01 debian kernel: Call Trace: > > Oct 12 17:24:01 debian kernel: <TASK> > > Oct 12 17:24:01 debian kernel: __schedule+0x34d/0x9e0 > > Oct 12 17:24:01 debian kernel: schedule+0x5a/0xd0 > > Oct 12 17:24:01 debian kernel: rwsem_down_write_slowpath+0x311/0x6d0 > > Oct 12 17:24:01 debian kernel: exit_mmap+0xf6/0x2f0 > > Oct 12 17:24:01 debian kernel: __mmput+0x3e/0x130 > > Oct 12 17:24:01 debian kernel: do_exit+0x2fc/0xaf0 > > Oct 12 17:24:01 debian kernel: do_group_exit+0x2d/0x80 > > Oct 12 17:24:01 debian kernel: __x64_sys_exit_group+0x14/0x20 > > Oct 12 17:24:01 debian kernel: do_syscall_64+0x55/0xb0 > > Oct 12 17:24:01 debian kernel: ? do_fault+0x1a4/0x410 > > Oct 12 17:24:01 debian kernel: ? __handle_mm_fault+0x660/0xfa0 > > Oct 12 17:24:01 debian kernel: ? exit_to_user_mode_prepare+0x40/0x1e0 > > Oct 12 17:24:01 debian kernel: ? handle_mm_fault+0xdb/0x2d0 > > Oct 12 17:24:01 debian kernel: ? do_user_addr_fault+0x1b0/0x550 > > Oct 12 17:24:01 debian kernel: ? exit_to_user_mode_prepare+0x40/0x1e0 > > Oct 12 17:24:01 debian kernel: entry_SYSCALL_64_after_hwframe+0x6e/0xd8 > > Oct 12 17:24:01 debian kernel: RIP: 0033:0x7f797d75a349 > > Oct 12 17:24:01 debian kernel: RSP: 002b:00007fff37f0d3c8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7 > > Oct 12 17:24:01 debian kernel: RAX: ffffffffffffffda RBX: 00007f797d8549e0 RCX: 00007f797d75a349 > > Oct 12 17:24:01 debian kernel: RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000 > > Oct 12 17:24:01 debian kernel: RBP: 0000000000000000 R08: fffffffffffffe98 R09: 00007fff37f0d2df > > Oct 12 17:24:01 debian kernel: R10: 00007fff37f0d240 R11: 0000000000000246 R12: 00007f797d8549e0 > > Oct 12 17:24:01 debian kernel: R13: 00007f797d85a2e0 R14: 0000000000000002 R15: 00007f797d85a2c8 > > Oct 12 17:24:01 debian kernel: </TASK> > > > > It reproduces fairly easily during a kernel build... > > > > It doesn't sound like the same issue you're pointing out, right Lorenzo? > > It could be. I suspect there has been a change recently that has > made the bug possible - although, I've not put effort into finding out > yet if that is true. If the bug existed for a long time (probably since > I fixed the live locking issue in 6.4 that was backported), then you > could be hitting it. > > It is a single line fix. If it happens frequently enough, you could try > it - this fix will go through the backporting route once it lands. > > Although, I am not sure it has much to do with the maple tree as I don't > think anyone should have the mm to take the mmap write lock. If we were > stuck in the maple tree somehow, the mm wouldn't reach the exit_mmap() > path - unless I missed something? I think this is the same bug, as the problem is, once it manifests, the actual problems it causes are delayed down the line until we hit a situation that is _caused by_ the bug but somewhat detached from it. In fact Bert and Mikhail hit the same thing with a process locking up in this path AND in that a lock is held that really makes no sense. As Liam says, this is a one-line change [0], could you try taking it and seeing? It has already been taken by Andrew, but we delay our hotfixes by a week or two so isn't in -rc yet. [0]: https://lore.kernel.org/linux-mm/48b349a2a0f7c76e18772712d0997a5e12ab0a3b.1728314403.git.lorenzo.stoakes@xxxxxxxxxx/ > > If you can dump the running tasks when you hit it, we could get a clue > from the (probably numerous) backtraces? I suggest first trying [0] :) if that doesn't fix things then we will definitely want to dig in further. > > Thanks, > Liam Thanks!
