[+cc Jonathan, author of 068c29a248b6]
On Wed, Mar 15, 2023 at 04:54:49PM -0700, Kuppuswamy Sathyanarayanan wrote:
> Commit 068c29a248b6 ("PCI/ERR: Clear PCIe Device Status errors only if
> OS owns AER") adds support to clear error status in the Device Status
> Register(DEVSTA) only if OS owns the AER support. But this change
> breaks the requirement of the EDR feature which requires OS to cleanup
> the error registers even if firmware owns the control of AER support.
>
> More details about this requirement can be found in PCIe Firmware
> specification v3.3, Table 4-6 Interpretation of the _OSC Control Field.
> If the OS supports the Error Disconnect Recover (EDR) feature and
> firmware sends the EDR event, then during the EDR recovery window, OS
> is responsible for the device error recovery and holds the ownership of
> the following error registers.
>
> • Device Status Register
> • Uncorrectable Error Status Register
> • Correctable Error Status Register
> • Root Error Status Register
> • RP PIO Status Register
>
> So call pcie_clear_device_status() in edr_handle_event() if the error
> recovery is successful.
IIUC, after ac1c8e35a326 ("PCI/DPC: Add Error Disconnect Recover (EDR)
support") appeared in v5.7-rc1, DEVSTA was always cleared in this path:
edr_handle_event
pcie_do_recovery
pcie_clear_device_status
After 068c29a248b6 ("PCI/ERR: Clear PCIe Device Status errors only if
OS owns AER") appeared in v5.9-rc1, we only clear DEVSTA if the OS
owns the AER Capability:
edr_handle_event
pcie_do_recovery
if (pcie_aer_is_native(dev)) # <-- new test
pcie_clear_device_status
So in the case where the OS does *not* own AER, and it receives an EDR
notification, DEVSTA is not cleared when it should be. Right?
I assume we should have a Fixes: tag here, since this patch should be
backported to every kernel that contains 068c29a248b6. Possibly even
a stable tag, although it's arguable whether it's "critical" per
Documentation/process/stable-kernel-rules.rst.
> Reported-by: Tsaur Erwin <erwin.tsaur@xxxxxxxxx>
I assume this report was internal, and there's no mailing list post or
bugzilla issue URL we can include here?
> Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@xxxxxxxxxxxxxxx>
> ---
>
> Changes since v1:
> * Rebased on top of v6.3-rc1.
> * Fixed a typo in pcie_clear_device_status().
>
> drivers/pci/pcie/edr.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/drivers/pci/pcie/edr.c b/drivers/pci/pcie/edr.c
> index a6b9b479b97a..87734e4c3c20 100644
> --- a/drivers/pci/pcie/edr.c
> +++ b/drivers/pci/pcie/edr.c
> @@ -193,6 +193,7 @@ static void edr_handle_event(acpi_handle handle, u32 event, void *data)
> */
> if (estate == PCI_ERS_RESULT_RECOVERED) {
> pci_dbg(edev, "DPC port successfully recovered\n");
> + pcie_clear_device_status(edev);
It's a little weird to work around a change inside pcie_do_recovery()
by clearing it here, and that means we clear it twice in the AER
native case, but I don't see any simpler way to do this, so this seems
fine as the fix for the current issue.
Question though: in the AER native case, pcie_do_recovery() calls
both:
pcie_clear_device_status() and
pci_aer_clear_nonfatal_status()
In this patch, you only call pcie_clear_device_status(). Do you care
about pci_aer_clear_nonfatal_status(), too?
The overall design for clearing status has gotten pretty complicated
as we've added error handling methods (firmware-first, DPC, EDR), and
there are so many different places and cases that it's hard to be sure
we do them all correctly.
I don't really know how to clean this up, so I'm just attaching my
notes about the current state:
- AER native handling:
handle_error_source
if (info->severity == AER_CORRECTABLE)
clear PCI_ERR_COR_STATUS <--
if (pcie_aer_is_native(dev))
pdrv->err_handler->cor_error_detected()
pcie_clear_device_status
clear PCI_EXP_DEVSTA <--
else
pcie_do_recovery
pcie_clear_device_status
clear PCI_EXP_DEVSTA <--
pci_aer_clear_nonfatal_status
clear PCI_ERR_UNCOR_STATUS <--
- Firmware-first handling: status is cleared by firmware before
event is reported to OS via HEST
- DPC native handling:
dpc_handler
dpc_process_error
if (rp_extensions)
dpc_process_rp_pio_error
clear PCI_EXP_DPC_RP_PIO_STATUS <--
else if (...)
pci_aer_clear_nonfatal_status
clear PCI_ERR_UNCOR_STATUS <--
pci_aer_clear_fatal_status
clear PCI_ERR_UNCOR_STATUS <--
pcie_do_recovery
if (AER native)
pcie_clear_device_status
clear PCI_EXP_DEVSTA <--
pci_aer_clear_nonfatal_status
clear PCI_ERR_UNCOR_STATUS <--
- EDR event handling:
edr_handle_event
dpc_process_error
if (rp_extensions)
dpc_process_rp_pio_error
clear PCI_EXP_DPC_RP_PIO_STATUS <--
else if (...)
pci_aer_clear_nonfatal_status
clear PCI_ERR_UNCOR_STATUS <--
pci_aer_clear_fatal_status
clear PCI_ERR_UNCOR_STATUS <--
pci_aer_raw_clear_status
clear PCI_ERR_ROOT_STATUS <--
clear PCI_ERR_COR_STATUS <--
clear PCI_ERR_UNCOR_STATUS <--
pcie_do_recovery
if (AER native)
pcie_clear_device_status
clear PCI_EXP_DEVSTA <--
pci_aer_clear_nonfatal_status
clear PCI_ERR_UNCOR_STATUS <--
if (PCI_ERS_RESULT_RECOVERED)
pcie_clear_device_status
clear PCI_EXP_DEVSTA <--
> acpi_send_edr_status(pdev, edev, EDR_OST_SUCCESS);
> } else {
> pci_dbg(edev, "DPC port recovery failed\n");
> --
> 2.34.1
>
