On Tue, 12 Mar 2024 at 13:47, Jakub Kicinski <kuba@xxxxxxxxxx> wrote: > > With your tree as of 65d287c7eb1d it gets to prompt but dies soon after > when prod services kick in (dunno what rpm Kdump does but says iocost > so adding Tejun): Both of your traces are timers that seem to either lock up in ioc_now(): https://lore.kernel.org/all/20240312133427.1a744844@xxxxxxxxxx/ and now it looks like ioc_timer_fn(): https://lore.kernel.org/all/20240312134739.248e6bd3@xxxxxxxxxx/ But in neither case does it actually look like it's a lockup on a *lock*. IOW, the NMI isn't happening on some spin_lock sequence or anything like that. Yes, ioc_now() could have been looping on the seq read-lock if the sequence number was odd. But the writers do seem to be done with interrupts disabled, plus then you wouldn't have this lockup in ioc_timer_fn, so it's probably not that. And yes, ioc_timer_fn() does take locks, but again, that doesn't seem to be where it is hanging. So it smells like it's an endless loop in ioc_timer_fn() to me, or perhaps retriggering the timer itself infinitely. Which would then explain both of those traces (that endless loop would call ioc_now() as part of it). The blk-iocost.c code itself hasn't changed, but the timer code has gone through big changes. That said, there's a more blk-related change: da4c8c3d0975 ("block: cache current nsec time in struct blk_plug"). *And* your second dump is from that period_vtime = now.vnow - ioc->period_at_vtime; if (WARN_ON_ONCE(!period_vtime)) { so it smells like the blk-iocost code is just completely confused by the time caching. Jens? Jakub, it might be worth seeing if just reverting that commit da4c8c3d0975 makes the problem go away. Otherwise a bisect might be needed... Linus
