On 2/11/2025 4:33 PM, Dan Williams wrote:
> Terry Bowman wrote:
>> CXL and AER drivers need the ability to identify CXL devices and CXL port
>> devices.
>>
>> First, add set_pcie_cxl() with logic checking for CXL Flexbus DVSEC
>> presence. The CXL Flexbus DVSEC presence is used because it is required
>> for all the CXL PCIe devices.[1]
>>
>> Add boolean 'struct pci_dev::is_cxl' with the purpose to cache the CXL
>> Flexbus presence.
>>
>> Add pcie_is_cxl() as a macro to return 'struct pci_dev::is_cxl'.
>>
>> Add pcie_is_cxl_port() to check if a device is a CXL Root Port, CXL
>> Upstream Switch Port, or CXL Downstream Switch Port. Also, verify the
>> CXL Extensions DVSEC for Ports is present.[1]
>>
>> [1] CXL 3.1 Spec, 8.1.1 PCIe Designated Vendor-Specific Extended
>> Capability (DVSEC) ID Assignment, Table 8-2
>>
>> Signed-off-by: Terry Bowman <terry.bowman@xxxxxxx>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@xxxxxxxxxx>
>> Reviewed-by: Dave Jiang <dave.jiang@xxxxxxxxx>
>> Reviewed-by: Fan Ni <fan.ni@xxxxxxxxxxx>
>> ---
>> drivers/pci/pci.c | 13 +++++++++++++
>> drivers/pci/probe.c | 10 ++++++++++
>> include/linux/pci.h | 5 +++++
>> include/uapi/linux/pci_regs.h | 3 ++-
>> 4 files changed, 30 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
>> index 869d204a70a3..a2d8b41dd043 100644
>> --- a/drivers/pci/pci.c
>> +++ b/drivers/pci/pci.c
>> @@ -5032,6 +5032,19 @@ static u16 cxl_port_dvsec(struct pci_dev *dev)
>> PCI_DVSEC_CXL_PORT);
>> }
>>
>> +inline bool pcie_is_cxl(struct pci_dev *pci_dev)
>> +{
>> + return pci_dev->is_cxl;
>> +}
>> +
>> +bool pcie_is_cxl_port(struct pci_dev *dev)
>> +{
>> + if (!pcie_is_cxl(dev))
>> + return false;
>> +
>> + return (cxl_port_dvsec(dev) > 0);
> At first I was concerned that this adds a capability list walk during
> error handling, but patch 17 takes pcie_is_cxl_port() out of the
> handles_cxl_errors() path.
>
> It is still used in the aer_probe() path which means enumeration can
> potentially race a CXL link up event.
>
> I think this is fine for now because the CXL core has the same top-down
> vs bottom-up race, and the CXL SBR code also shares the same race
> problem.
>
> A follow-on change needs to arrange for cxl_port_probe() to
> enable/disable internal errors, because that path knows that a link has
> been negotiated with an endpoint and that the CXL link details should be
> stable.
>
>> +}
>> +
>> static bool cxl_sbr_masked(struct pci_dev *dev)
>> {
>> u16 dvsec, reg;
>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>> index b6536ed599c3..7737b9ce7a83 100644
>> --- a/drivers/pci/probe.c
>> +++ b/drivers/pci/probe.c
>> @@ -1676,6 +1676,14 @@ static void set_pcie_thunderbolt(struct pci_dev *dev)
>> dev->is_thunderbolt = 1;
>> }
>>
>> +static void set_pcie_cxl(struct pci_dev *dev)
>> +{
>> + u16 dvsec = pci_find_dvsec_capability(dev, PCI_VENDOR_ID_CXL,
>> + PCI_DVSEC_CXL_FLEXBUS);
>> + if (dvsec)
>> + dev->is_cxl = 1;
>> +}
> Similar race problem here as it is premature to check for this DVSEC on
> disconnected ports.
>
> For now, lets add a comment to include/uapi/linux/pci_regs.h along the
> lines of:
>
> diff --git a/include/uapi/linux/pci_regs.h b/include/uapi/linux/pci_regs.h
> index 3445c4970e4d..32df7abdd23c 100644
> --- a/include/uapi/linux/pci_regs.h
> +++ b/include/uapi/linux/pci_regs.h
> @@ -1208,7 +1208,13 @@
> #define PCI_DOE_DATA_OBJECT_DISC_RSP_3_PROTOCOL 0x00ff0000
> #define PCI_DOE_DATA_OBJECT_DISC_RSP_3_NEXT_INDEX 0xff000000
>
> -/* Compute Express Link (CXL r3.1, sec 8.1.5) */
> +/*
> + * Compute Express Link (CXL r3.1, sec 8.1)
> + *
> + * Note that CXL DVSEC id 3 and 7 to be ignored when the CXL link state
> + * is "disconnected" (CXL r3.1, sec 9.12.3). Re-enumerate these
> + * registers on downstream link-up events.
> + */
> #define PCI_DVSEC_CXL_PORT 3
> #define PCI_DVSEC_CXL_PORT_CTL 0x0c
> #define PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR 0x00000001
>
> ...to at least remind our future selves that there is work to do here to
> make the implementation robust against hot-plug scenarios.
>
> With that you can add:
>
> Reviewed-by: Dan Williams <dan.j.williams@xxxxxxxxx>
Ok, I will add the comment.
Would you like for me to add the enable/disable internal error logic to cxl_port_probe()? I can but want to confirm.
Thanks for reviewing.
Terry
