On Thu, Nov 02, 2023 at 03:27:58PM +0000, bugzilla-daemon@xxxxxxxxxx wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=217572 > > --- Comment #18 from Christian Theune (ct@xxxxxxxxxxxxxxx) --- > We've updated a while ago and our fleet is not seeing improved results. They've > actually seemed to have gotten worse according to the number of alerts we've > seen. This is still an unreproducable, unfixed bug in upstream kernels. There is no known reproducer, so actually triggering it and hence performing RCA is extremely difficult at this point in time. We don't really even know what workload triggers it. > We've had a multitude of crashes in the last weeks with the following > statistics: > > 6.1.31 - 2 affected machines > 6.1.35 - 1 affected machine > 6.1.37 - 1 affected machine > 6.1.51 - 5 affected machines > 6.1.55 - 2 affected machines > 6.1.57 - 2 affected machines Do these machines have ECC memory? > Here's the more detailed behaviour of one of the machines with 6.1.57. > > $ uptime > 16:10:23 up 13 days 19:00, 1 user, load average: 3.21, 1.24, 0.57 Yeah, that's the problem - such a rare, one off issue that we don't really even know where to begin looking. :( Given you seem to have a workload that occasionally triggers it, could you try to craft a reproducer workload that does stuff similar to your production workload and see if you can find out something that makes this easier to trigger? > $ uname -a > Linux ts00 6.1.57 #1-NixOS SMP PREEMPT_DYNAMIC Tue Oct 10 20:00:46 UTC 2023 > x86_64 GNU/Linux > > And here' the stall: .... > [654042.645101] <TASK> > [654042.645353] ? asm_sysvec_apic_timer_interrupt+0x16/0x20 > [654042.645956] ? xas_descend+0x22/0x90 > [654042.646366] xas_load+0x30/0x40 > [654042.646738] filemap_get_read_batch+0x16e/0x250 > [654042.647253] filemap_get_pages+0xa9/0x630 > [654042.647714] filemap_read+0xd2/0x340 > [654042.648124] ? __mod_memcg_lruvec_state+0x6e/0xd0 > [654042.648670] xfs_file_buffered_read+0x4f/0xd0 [xfs] This implies you are using memcg to constrain memory footprint of the applications? Are these workloads running in memcgs that experience random memcg OOM conditions? Or maybe the failure correlates with global OOM conditions triggering memcg reclaim? Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
