In subject: PCI/ASPM: Fix L1.2 parameters before enabling L1.2 On Fri, Feb 16, 2024 at 02:26:44PM +0800, Jian-Hong Pan wrote: > Currently, when enable link's L1.2 features with __pci_enable_link_state(), > it configs the link directly without ensuring related L1.2 parameters, such > as T_POWER_ON, Common_Mode_Restore_Time, and LTR_L1.2_THRESHOLD have been > programmed. > > This leads VMD enabled systems' L1.2 of the link between VMD remapped PCIe > Root Port and NVMe gets wrong configs when a caller tries to enabled it. This is not VMD-specific. > Here is a failed example on ASUS B1400CEAE with enabled VMD: > > 10000:e0:06.0 PCI bridge: Intel Corporation 11th Gen Core Processor PCIe Controller (rev 01) (prog-if 00 [Normal decode]) > ... > Capabilities: [200 v1] L1 PM Substates > L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ > PortCommonModeRestoreTime=45us PortTPowerOnTime=50us > L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- > T_CommonMode=45us LTR1.2_Threshold=101376ns > L1SubCtl2: T_PwrOn=50us > > 10000:e1:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01) (prog-if 02 [NVM Express]) > ... > Capabilities: [900 v1] L1 PM Substates > L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+ > PortCommonModeRestoreTime=32us PortTPowerOnTime=10us > L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- > T_CommonMode=0us LTR1.2_Threshold=0ns > L1SubCtl2: T_PwrOn=10us > > According to PCI Express Base Specification Revision 6.0, Section 5.5.4, > before enable ASPM L1.2 on the PCIe Root Port and the NVMe, they should be > programmed with the same LTR1.2_Threshold value. However, they have > different values in this case. "PCIe r6.0, sec 5.5.4" s/before enable/before enabling/ > This patch invokes aspm_calc_l12_info() to program the L1.2 parameters > properly before enable L1.2 bits of L1 PM Substates Control Register in > __pci_enable_link_state(). s/This patch invokes/Invoke/ https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst?id=v6.7#n94 > Also, introduces aspm_get_l1ss_cap() shared > into aspm_l1ss_init() and __pci_enable_link_state() to get the PCIe > devices' L1SS capability for aspm_calc_l12_info(). "Also" is always a good clue that something should be split to a separate patch :) > Link: https://bugzilla.kernel.org/show_bug.cgi?id=218394 > Signed-off-by: Jian-Hong Pan <jhp@xxxxxxxxxxxxx> > --- > v2: > - Prepare the PCIe LTR parameters before enable L1 Substates > > v3: > - Only enable supported features for the L1 Substates part > > v4: > - Focus on fixing L1.2 parameters, instead of re-initializing whole L1SS > > drivers/pci/pcie/aspm.c | 35 ++++++++++++++++++++++++++--------- > 1 file changed, 26 insertions(+), 9 deletions(-) > > diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c > index a39d2ee744cb..42a8c4c194c1 100644 > --- a/drivers/pci/pcie/aspm.c > +++ b/drivers/pci/pcie/aspm.c > @@ -588,6 +588,18 @@ static void pcie_aspm_check_latency(struct pci_dev *endpoint) > } > } > > +static u32 aspm_get_l1ss_cap(struct pci_dev *pdev) > +{ > + u32 l1ss_cap; > + > + pci_read_config_dword(pdev, pdev->l1ss + PCI_L1SS_CAP, &l1ss_cap); > + > + if (!(l1ss_cap & PCI_L1SS_CAP_L1_PM_SS)) > + l1ss_cap = 0; > + > + return l1ss_cap; > +} This is nice but should be a separate patch because it just factors out existing code, and having it in the same patch obscures the real point of *this* patch. > /* Calculate L1.2 PM substate timing parameters */ > static void aspm_calc_l12_info(struct pcie_link_state *link, > u32 parent_l1ss_cap, u32 child_l1ss_cap) > @@ -698,15 +710,8 @@ static void aspm_l1ss_init(struct pcie_link_state *link) > return; > > /* Setup L1 substate */ > - pci_read_config_dword(parent, parent->l1ss + PCI_L1SS_CAP, > - &parent_l1ss_cap); > - pci_read_config_dword(child, child->l1ss + PCI_L1SS_CAP, > - &child_l1ss_cap); > - > - if (!(parent_l1ss_cap & PCI_L1SS_CAP_L1_PM_SS)) > - parent_l1ss_cap = 0; > - if (!(child_l1ss_cap & PCI_L1SS_CAP_L1_PM_SS)) > - child_l1ss_cap = 0; > + parent_l1ss_cap = aspm_get_l1ss_cap(parent); > + child_l1ss_cap = aspm_get_l1ss_cap(child); > > /* > * If we don't have LTR for the entire path from the Root Complex > @@ -1367,6 +1372,8 @@ EXPORT_SYMBOL(pci_disable_link_state); > static int __pci_enable_link_state(struct pci_dev *pdev, int state, bool locked) > { > struct pcie_link_state *link = pcie_aspm_get_link(pdev); > + struct pci_dev *child = link->downstream, *parent = link->pdev; > + u32 parent_l1ss_cap, child_l1ss_cap; > > if (!link) > return -EINVAL; > @@ -1398,6 +1405,16 @@ static int __pci_enable_link_state(struct pci_dev *pdev, int state, bool locked) > link->aspm_default |= ASPM_STATE_L1_1_PCIPM | ASPM_STATE_L1; > if (state & PCIE_LINK_STATE_L1_2_PCIPM) > link->aspm_default |= ASPM_STATE_L1_2_PCIPM | ASPM_STATE_L1; > + /* > + * Ensure L1.2 paramters: Common_Mode_Restore_Times, T_POWER_ON and > + * LTR_L1.2_THRESHOLD are programmed properly before enable bits for > + * L1.2, per PCIe r6.0, sec 5.5.4. s/paramters/parameters/ > + if (state & link->aspm_capable & ASPM_STATE_L1_2_MASK) { > + parent_l1ss_cap = aspm_get_l1ss_cap(parent); > + child_l1ss_cap = aspm_get_l1ss_cap(child); > + aspm_calc_l12_info(link, parent_l1ss_cap, child_l1ss_cap); Why doesn't this happen already via normal enumeration? It looks like this path should do it even without this patch: pcie_aspm_init_link_state pcie_aspm_cap_init aspm_l1ss_init > + } > pcie_config_aspm_link(link, policy_to_aspm_state(link)); > > link->clkpm_default = (state & PCIE_LINK_STATE_CLKPM) ? 1 : 0; > -- > 2.43.2 >
