On Tue, Apr 09, 2024 at 10:05:40AM -0500, Bjorn Helgaas wrote: > On Tue, Apr 09, 2024 at 04:35:28PM +0900, Kobayashi,Daisuke wrote: > > Add sysfs attribute for CXL 1.1 device link status to the cxl pci device. > > > > In CXL1.1, the link status of the device is included in the RCRB mapped to > > the memory mapped register area. Critically, that arrangement makes the link > > status and control registers invisible to existing PCI user tooling. > > Idle thought: PCIe does define RCRB, even pre-CXL. Maybe the PCI core > should be enhanced to comprehend RCRB directly? The way CXL 1.1 (ab)uses the RCRB differs from what the PCIe Base Spec envisions: Per PCIe r6.2 sec 7.2.3, the RCRB contains additional Extended Capabilities of a Root Port -- in addition to those in the Root Port's Config Space. What we could do in the PCI core to support this is to amend our helpers which search for Extended Capabilities to also search for them in the RCRB. In fact, two years ago I cooked up a patch which does exactly that: https://github.com/l1k/linux/commit/3eb94f042527 And I cooked up another patch to fetch the RCRB's address from the CXL Early Discovery ACPI table: https://github.com/l1k/linux/commit/d9d3cf45cf8c The reason I never submitted the patches? I realized after the fact that CXL 1.1 uses the RCRB in a completely different way: Per CXL r3.0 sec 8.2.1, RCH Downstream and RCD Upstream Ports do not actually possess a Config Space. Instead, they possess *only* an RCRB. And that RCRB contains a Type 1 Configuration Space Header. But because the PCIe Base Spec prescribes that there has to be an Extended Capability at offset 0 of the RCRB, the CXL spec puts a Null Extended Capability at offset 0 so that the Type 1 Config Space Header is skipped. However this means that the first dword of the Type 1 Config Space Header does not contain a Vendor ID and Device ID. So what we could do is create a fake pci_dev for each RCH Downstream and RCD Upstream Port plus a specially crafted struct pci_ops whose ->read() and write() callbacks access the RCRB. But how do we know which Vendor and Device ID to return from a ->read()? There is none in the RCRB! The CXL Consortium seems to have realized the mess they made with CXL 1.1 and from CXL 2.0 onwards everything is now a proper PCI device. I talked to Dan about my findings and his decision was basically to not enable any of that legacy CXL 1.1 RCRB functionality in the kernel. Thanks, Lukas
