Background
==========
In the RFC for Kernel Support of Memory Error Detection [1], one advantage
of software-based scanning over hardware patrol scrubber is the ability
to make statistics visible to system administrators. The statistics
include 2 categories:
* Memory error statistics, for example, how many memory error are
encountered, how many of them are recovered by the kernel. Note these
memory errors are non-fatal to kernel: during the machine check
exception (MCE) handling kernel already classified MCE's severity to
be unnecessary to panic (but either action required or optional).
* Scanner statistics, for example how many times the scanner have fully
scanned a NUMA node, how many errors are first detected by the scanner.
The memory error statistics are useful to userspace and actually not
specific to scanner detected memory errors, and are the focus of this RFC.
Motivation
==========
Memory error stats are important to userspace but insufficient in kernel
today. Datacenter administrators can better monitor a machine's memory
health with the visible stats. For example, while memory errors are
inevitable on servers with 10+ TB memory, starting server maintenance
when there are only 1~2 recovered memory errors could be overreacting;
in cloud production environment maintenance usually means live migrate
all the workload running on the server and this usually causes nontrivial
disruption to the customer. Providing insight into the scope of memory
errors on a system helps to determine the appropriate follow-up action.
In addition, the kernel's existing memory error stats need to be
standardized so that userspace can reliably count on their usefulness.
Today kernel provides following memory error info to userspace, but they
are not sufficient or have disadvantages:
* HardwareCorrupted in /proc/meminfo: number of bytes poisoned in total,
not per NUMA node stats though
* ras:memory_failure_event: only available after explicitly enabled
* /dev/mcelog provides many useful info about the MCEs, but doesn't
capture how memory_failure recovered memory MCEs
* kernel logs: userspace needs to process log text
Exposing memory error stats is also a good start for the in-kernel memory
error detector. Today the data source of memory error stats are either
direct memory error consumption, or hardware patrol scrubber detection
(when signaled as UCNA; these signaled as SRAO are not handled by
memory_failure). Once in-kernel memory scanner is implemented, it will be
the main source as it is usually configured to scan memory DIMMs constantly
and faster than hardware patrol scrubber.
How Implemented
===============
As Naoya pointed out [2], exposing memory error statistics to userspace
is useful independent of software or hardware scanner. Therefore we
implement the memory error statistics independent of the in-kernel memory
error detector. It exposes the following per NUMA node memory error
counters:
/sys/devices/system/node/node${X}/memory_failure/pages_poisoned
/sys/devices/system/node/node${X}/memory_failure/pages_recovered
/sys/devices/system/node/node${X}/memory_failure/pages_ignored
/sys/devices/system/node/node${X}/memory_failure/pages_failed
/sys/devices/system/node/node${X}/memory_failure/pages_delayed
These counters describe how many raw pages are poisoned and after the
attempted recoveries by the kernel, their resolutions: how many are
recovered, ignored, failed, or delayed respectively. This approach can be
easier to extend for future use cases than /proc/meminfo, trace event,
and log. The following math holds for the statistics:
* pages_poisoned = pages_recovered + pages_ignored + pages_failed +
pages_delayed
* pages_poisoned * page_size = /proc/meminfo/HardwareCorrupted
These memory error stats are reset during machine boot.
The 1st commit introduces these sysfs entries. The 2nd commit populates
memory error stats every time memory_failure finishes memory error
recovery. The 3rd commit adds documentations for introduced stats.
[1] https://lore.kernel.org/linux-mm/7E670362-C29E-4626-B546-26530D54F937@xxxxxxxxx/T/#mc22959244f5388891c523882e61163c6e4d703af
[2] https://lore.kernel.org/linux-mm/7E670362-C29E-4626-B546-26530D54F937@xxxxxxxxx/T/#m52d8d7a333d8536bd7ce74253298858b1c0c0ac6
Jiaqi Yan (3):
mm: memory-failure: Add memory failure stats to sysfs
mm: memory-failure: Bump memory failure stats to pglist_data
mm: memory-failure: Document memory failure stats
Documentation/ABI/stable/sysfs-devices-node | 39 +++++++++++
drivers/base/node.c | 3 +
include/linux/mm.h | 5 ++
include/linux/mmzone.h | 28 ++++++++
mm/memory-failure.c | 71 +++++++++++++++++++++
5 files changed, 146 insertions(+)
--
2.39.0.314.g84b9a713c41-goog
