This series proposes a set of improvements and new features for the
tracing subsystem to facilitate the debugging of low latency
deployments.
Currently, hwlat runs on a single CPU at a time, migrating across a
set of CPUs in a round-robin fashion. The first three patches are
changes made to allow hwlat to run on multiple CPUs in parallel,
increasing the chances of detecting a hardware latency.
The fourth patch is a helper to print a timestamp in a u64 in
seconds.nanoseconds format on tracepoints.
The fifth patch proposes a new tracer named osnoise and aims to help
users of isolcpus= (or a similar method) to measure how much noise the
OS and the hardware add to the isolated application. The osnoise tracer
bases on the hwlat detector code. The difference is that, instead of
sampling with interrupts disabled, the osnoise tracer samples the CPU with
interrupts and preemption enabled. In this way, the sampling thread will
suffer any source of noise from the OS. The detection and classification
of the type of noise are then made by observing the entry points of NMIs,
IRQs, SoftIRQs, and threads. If none of these sources of noise is detected,
the tool associates the noise with the hardware. The tool periodically
prints a status, printing the total noise of the period, the max single
noise observed, the percentage of CPU available for the task, along with
the counters of each source of the noise. To debug the sources of noise,
the tracer also adds a set of tracepoints that print any NMI, IRQ, SofIRQ,
and thread occurrence. These tracepoints print the starting time and the
noise's net duration at the end of the noise. In this way, it reduces the
number of tracepoints (one instead of two) and the need to manually
accounting the contribution of each noise independently.
Daniel Bristot de Oliveira (4):
tracing/hwlat: Add a cpus file specific for hwlat_detector
tracing/hwlat: Implement the mode config option
tracing/hwlat: Implement the per-cpu mode
tracing: Add the osnoise tracer
Steven Rostedt (1):
tracing: Add __print_ns_to_secs() and __print_ns_without_secs()
helpers
Documentation/trace/hwlat_detector.rst | 29 +-
Documentation/trace/osnoise_tracer.rst | 149 ++
include/linux/ftrace_irq.h | 16 +
include/trace/events/osnoise.h | 141 ++
include/trace/trace_events.h | 25 +
kernel/trace/Kconfig | 34 +
kernel/trace/Makefile | 1 +
kernel/trace/trace.h | 9 +-
kernel/trace/trace_entries.h | 27 +
kernel/trace/trace_hwlat.c | 445 +++++-
kernel/trace/trace_osnoise.c | 1714 ++++++++++++++++++++++++
kernel/trace/trace_output.c | 72 +-
12 files changed, 2604 insertions(+), 58 deletions(-)
create mode 100644 Documentation/trace/osnoise_tracer.rst
create mode 100644 include/trace/events/osnoise.h
create mode 100644 kernel/trace/trace_osnoise.c
--
2.30.2
