Roman Kisel <romank@xxxxxxxxxxxxxxxxxxx> writes: > On 6/17/2024 11:18 PM, Sebastian Andrzej Siewior wrote: >> On 2024-06-17 16:41:30 [-0700], Roman Kisel wrote: >>> Missing, failed, or corrupted core dumps might impede crash >>> investigations. To improve reliability of that process and consequently >>> the programs themselves, one needs to trace the path from producing >>> a core dumpfile to analyzing it. That path starts from the core dump file >>> written to the disk by the kernel or to the standard input of a user >>> mode helper program to which the kernel streams the coredump contents. >>> There are cases where the kernel will interrupt writing the core out or >>> produce a truncated/not-well-formed core dump. >> How much of this happened and how much of this is just "let me handle >> everything that could go wrong". > Some of that must be happening as there are truncated dump files. Haven't run > the logging code at large scale yet with the systems being stressed a lot by the > customer workloads to hit all edge cases. Sent the changes to the kernel mail > list out of abundance of caution first, and being ecstatic about that: on the > other thread Kees noticed I didn't use the ratelimited logging. That has > absolutely made me day and whole week, just glowing :) Might've been a close > call due to something in a crash loop. Another reason you could have truncated coredumps is the coredumping process being killed. I suspect if you want reasons why the coredump is truncated you are going to want to instrument dump_interrupted, dump_skip and dump_emit rather than their callers. As they don't actually report why the failed. Are you using systemd-coredump? Or another pipe based coredump collector? It might be the dump collector is truncating things. Do you know if your application uses io_uring? There were some weird issues with io_uring and coredumps that were causing things to get truncation at one point. As I recall a hack was put in the coredump code so that it worked but maybe there is another odd case that still needs to be handled. > > I think it'd be fair to say that I am asking to please "let me handle (log) > everything that could go wrong", ratelimited, as these error cases are present > in the code, and logging can give a clue why the core dump collection didn't > succeed and what one would need to explore to increase reliability of the > system. If you are looking for reasons you definitely want to instrument fs/coredump.c much more than fs/binfmt_elf.c. As fs/coredump.c is the code that actually performs the writes. One of these days if someone is ambitious we should probably merge the coredump code from fs/binfmt_elf.c and fs/binfmt_elf_fdpic.c and just hardcode the coredump code to always produce an elf format coredump. Just for the simplicity of it all. Eric
