On Wed, Sep 27, 2023 at 9:03 AM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: > > On Fri, Sep 22, 2023 at 10:46:36AM +0800, Shuai Xue wrote: > > ... > > > Actually, this is a question from my colleague from firmware team. > > The original question is that: > > > > "Should I set CPER_SEV_FATAL for Generic Error Status Block when a > > PCIe fatal error is detected? If set, kernel will always panic. > > Otherwise, kernel will always not panic." > > > > So I pull a question about desired behavior of Linux kernel first :) > > From the perspective of the kernel, CPER_SEV_FATAL for Generic Error > > Status Block is not reasonable. The kernel will attempt to recover > > Fatal errors, although recovery may fail. > > I don't know the semantics of CPER_SEV_FATAL or why it's there. > With CPER, we have *two* error severities: a "native" one defined by > the PCIe spec and another defined by the platform via CPER. > > I speculate that the reason for the CPER severity could be to provide > a severity for error sources that don't have a "native" severity like > AER does, or for the vendor to force the OS to restart (for > CPER_SEV_FATAL, anyway) in cases where it might not otherwise. > > In the native case, we only have the PCIe severity and don't have the > CPER severity at all, and I suspect that unless there's uncontained > data corruption, we would rather handle even the most severe PCIe > fatal error by disabling the specific device(s) instead of panicking > and restarting the whole machine. >From a user's point of view disabling a device is often worse than a reboot. If you get a fatal error from a system's only network card then disabling the card may result in the system being uncontactable until someone manually recovers it. Similarly, if the disk hosting the root filesystem disappears the system may not crash immediately (most of what it needs will be in page cache), but there's no guarantee that it can do anything useful in that state. In both cases forcing a reboot will probably bring the system back into a usable state. > So for PCIe errors, I'm not sure setting CPER_SEV_FATAL is beneficial > unless the platform wants to force the OS to panic, e.g., maybe the > platform knows about data corruption and/or the vendor wants the OS to > panic as part of a reliability story. The PCIe spec is (intentionally?) vague about the causes of fatal errors. For all we know a device is reporting one because the embedded OS it was running crashed and as a side effect it's been DMAing junk into system memory for the past hour. If you know something about the device in question maybe you can make those assumptions, but in general it's impossible to assess the actual severity of an error. Even in the case of a noisy link causing bit flips (it's possible, LCRC is only 16bit and ECEC is optional) if we get corruption of the address bits of the TLP header then the DMA might have overwritten something important to the OS. From a hardware vendor's point of view just forcing a reboot makes a lot of sense. Paranoia aside, in a lot of cases PCI device errors are nothing major and can be resolved by just restarting the device. However, there's no way for generic kernel code to make that assessment and we probably shouldn't have the kernel guess. I'd say the safest option would be to punt that decision to userspace and provide some way to whitelist devices that we can ignore errors from. I'm not familiar enough with the ACPI to know if we have enough details in the notification it sends to actually implement that though. Oliver
