On 6/18/2024 2:21 PM, Eric W. Biederman wrote:
Roman Kisel <romank@xxxxxxxxxxxxxxxxxxx> writes:
On 6/17/2024 11:18 PM, Sebastian Andrzej Siewior wrote:
On 2024-06-17 16:41:30 [-0700], Roman Kisel wrote:
Missing, failed, or corrupted core dumps might impede crash
investigations. To improve reliability of that process and consequently
the programs themselves, one needs to trace the path from producing
a core dumpfile to analyzing it. That path starts from the core dump file
written to the disk by the kernel or to the standard input of a user
mode helper program to which the kernel streams the coredump contents.
There are cases where the kernel will interrupt writing the core out or
produce a truncated/not-well-formed core dump.
How much of this happened and how much of this is just "let me handle
everything that could go wrong".
Some of that must be happening as there are truncated dump files. Haven't run
the logging code at large scale yet with the systems being stressed a lot by the
customer workloads to hit all edge cases. Sent the changes to the kernel mail
list out of abundance of caution first, and being ecstatic about that: on the
other thread Kees noticed I didn't use the ratelimited logging. That has
absolutely made me day and whole week, just glowing :) Might've been a close
call due to something in a crash loop.
Another reason you could have truncated coredumps is the coredumping
process being killed.
I suspect if you want reasons why the coredump is truncated you are
going to want to instrument dump_interrupted, dump_skip and dump_emit
rather than their callers. As they don't actually report why the
failed.
I'll add logging there as well, thanks for the great idea!
Are you using systemd-coredump? Or another pipe based coredump
collector? It might be the dump collector is truncating things.
There is a collector program set via core_pattern so that the core dump
is streamed to its standard input. That is a very simple memcpy-like
bytes-in..bytes-out code. It logs how many bytes it receives and how
many bytes it writes, and no bytes are lost in this path. Of the system
itself, it is built out of the latest stable LTS kernel and a small user
land, not based on any distribution and packet management. One might say
it resembles an appliance.
Do you know if your application uses io_uring? There were some weird
issues with io_uring and coredumps that were causing things to get
truncation at one point. As I recall a hack was put in the coredump
code so that it worked but maybe there is another odd case that still
needs to be handled.
Couldn't appreciate the pointer more! There are cases when the user land
reaches out to io_uring, not the work horse though.
I think it'd be fair to say that I am asking to please "let me handle (log)
everything that could go wrong", ratelimited, as these error cases are present
in the code, and logging can give a clue why the core dump collection didn't
succeed and what one would need to explore to increase reliability of the
system.
If you are looking for reasons you definitely want to instrument
fs/coredump.c much more than fs/binfmt_elf.c. As fs/coredump.c is the
code that actually performs the writes.
Understood, thank you very much!
One of these days if someone is ambitious we should probably merge the
coredump code from fs/binfmt_elf.c and fs/binfmt_elf_fdpic.c and just
hardcode the coredump code to always produce an elf format coredump.
Just for the simplicity of it all.
I've had loads of experience with collecting and analyzing ELF core dump
files, including a tool that parses machine state, rebuilds the
necessary Linux kernel structures and produces ELF core dump files for
the user land processes from that. Perhaps I could embark on that
ambitious journey if no one else has time :)
Eric
--
Thank you,
Roman
