On Sat, Jul 06, 2024 at 01:55:11PM -0700, David Rientjes wrote:
> Rather than hacky scripts that collect things like vmstat, memory.stat,
> buddyinfo, etc, at regular intervals, it would be preferable to hand off
> something more complete. Idea is an open source tool that can be run in
> the background to collect metrics for the system, NUMA nodes, and memcg
> hierarchies, as well as potentially from subsystems in the kernel like
> delay accounting. IOW, I want to be able to say "install ${tool} and send
> over the log file."
>
> Are thre any open source tools that do a good job of this today that I can
> latch onto? If not, sounds like I'll be writing one from scratch. Let me
> know if there's interest in this as well.
>
> Thanks!
>
Hi David,
At meta we have built and deployed Below[1] for this purpose. It's a
tool similar to `top` or others, but can record system state
periodically and allow for replaying. We run this on our production
fleet, periodically recording system state to the local disk. When we
need to debug a machine at a point in the past, we can log in and
replay the state. This uses a TUI (see the link for a demo) to make
navigating the data more natural.
I'm aware of a few other organizations who have also deployed Below,
but tend to run it more in the manner you suggest - have it record
data but then use the snapshot command to export the state (e.g. as if
it was a log file) that can then be viewed off-host. Some
organizations eschew the TUI altogether and export the data to
Prometheus/Grafana.
I'll caution though that having the data is one thing, being able to
interpret it is entirely different. While we try and put the most
useful and easily-understood metrics front-and-center in the TUI,
debugging an issue like you describe would probably require some
domain-expertise.
[1] https://github.com/facebookincubator/below
