On Thu, Oct 06, 2022 at 08:14:48AM +0000, Elad Nachman wrote: > Hi, > > Regarding number #1 : > The H/W erratum means strong ordering mode is not properly > implemented in the H/W. The workaround is to set DIS_ORD_CHK to > disable the ordered check, to prevent the issues Pali mentioned. > > Leaving it without the workaround means encountering the crashes > Pali mentioned. The H/W was originally designed to have the strong > ordering mode, per what is expected of PCI Express, but due to this > erratum, it cannot be supported by the taped-out H/W. There is > nothing to do about it now in the H/W, as this is a years old SOC. > Software wise the only option is the workaround above, or to accept > sporadic crashes of the kernel. > > Regarding number #2 : > The impact of the workaround is that the DMA done status could be > set before the last data is written to the host memory, causing the > host to read incomplete data. This is the reason why I recommended > to add a memory barrier. If this is bothering, the memory barrier > can be removed from the patch, although I suggest to thoroughly test > the data integrity if implementing the patch without the memory > barrier. Thank you very much for these details, Elad. If I understand correctly, this patch avoids many issues (or at least makes them much less likely), but there is still a small possibility of corner cases where a correctly-written driver would work on other platforms but not on this hardware. Since this definitely improves things dramatically, I think it's worth doing even if it's not absolutely 100%. The commit log and code comments should be enough of a hint. > -----Original Message----- > From: Pali Rohár <pali@xxxxxxxxxx> > Sent: Tuesday, October 4, 2022 11:40 AM > To: Elad Nachman <enachman@xxxxxxxxxxx>; Lorenzo Pieralisi <lpieralisi@xxxxxxxxxx> > Cc: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>; Krzysztof Wilczyński <kw@xxxxxxxxx>; linux-pci@xxxxxxxxxxxxxxx; linux-arm-kernel@xxxxxxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx > Subject: Re: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround for PCIe Completion Timeout > > + Elad > > Could you please look at Lorenzo's comments and help with this fix? > > On Tuesday 04 October 2022 10:02:32 Lorenzo Pieralisi wrote: > > On Mon, Oct 03, 2022 at 11:14:12PM +0200, Pali Rohár wrote: > > > Lorenzo, is something more needed for this patch? As it workarounds > > > crashing it is really needed to have it in mainline and backports. > > > > Yes, a clear explanation from Marvell about what this is actually > > fixing - it took me a while to go through the whole thread but I still > > don't understand what this patch actually does and why. > > > > An Erratum workaround (if there is any) should define and explain a SW > > workaround. > > > > (1) Bjorn's concerns in relation to PCI memory model weren't addressed > > (2) We don't add undocumented memory barriers to the kernel to "minimize > > risks". Either we fix a bug or we don't. If we do, write that down > > and document why the barrier is there and the issue it solves. > > > > I understand that basically you are reverse engineering a HW bug but I > > am afraid we can't fix the kernel this way - more so with patches > > going to be backported to stable kernels. > > > > Lorenzo > > > > > On Wednesday 28 September 2022 14:05:10 Elad Nachman wrote: > > > > Reviewed-by: Elad Nachman <enachman@xxxxxxxxxxx> > > > > > > > > Thanks, > > > > > > > > Elad. > > > > > > > > -----Original Message----- > > > > From: Pali Rohár <pali@xxxxxxxxxx> > > > > Sent: Monday, September 26, 2022 3:35 PM > > > > To: Elad Nachman <enachman@xxxxxxxxxxx> > > > > Cc: Thomas Petazzoni <thomas.petazzoni@xxxxxxxxxxx>; Lorenzo > > > > Pieralisi <lpieralisi@xxxxxxxxxx>; Bjorn Helgaas > > > > <bhelgaas@xxxxxxxxxx>; Krzysztof Wilczyński <kw@xxxxxxxxx>; Rob > > > > Herring <robh@xxxxxxxxxx>; linux-pci@xxxxxxxxxxxxxxx; > > > > linux-arm-kernel@xxxxxxxxxxxxxxxxxxx; > > > > linux-kernel@xxxxxxxxxxxxxxx; Gregory Clement > > > > <gregory.clement@xxxxxxxxxxx>; Marek Behún <kabel@xxxxxxxxxx>; > > > > Remi Pommarel <repk@xxxxxxxxxxxx>; Xogium <contact@xxxxxxxxx>; > > > > Tomasz Maciej Nowak <tmn505@xxxxxxxxx> > > > > Subject: [EXT] Re: [PATCH v2] PCI: aardvark: Implement workaround > > > > for PCIe Completion Timeout > > > > > > > > External Email > > > > > > > > ------------------------------------------------------------------ > > > > ---- Hello Elad, could you please review this patch? I have > > > > implemented it according your instructions, including that full memory barrier as you described. > > > > > > > > On Tuesday 02 August 2022 14:38:16 Pali Rohár wrote: > > > > > Marvell Armada 3700 Functional Errata, Guidelines, and > > > > > Restrictions document describes in erratum 3.12 PCIe Completion Timeout (Ref #: > > > > > 251), that PCIe IP does not support a strong-ordered model for inbound posted vs. > > > > > outbound completion. > > > > > > > > > > As a workaround for this erratum, DIS_ORD_CHK flag in Debug Mux > > > > > Control register must be set. It disables the ordering check in > > > > > the core between Completions and Posted requests received from the link. > > > > > > > > > > Marvell also suggests to do full memory barrier at the beginning > > > > > of aardvark summary interrupt handler before calling interrupt > > > > > handlers of endpoint drivers in order to minimize the risk for > > > > > the race condition documented in the Erratum between the DMA > > > > > done status reading and the completion of writing to the host memory. > > > > > > > > > > More details about this issue and suggested workarounds are in discussion: > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lore.kernel > > > > > .org_l > > > > > inux-2Dpci_BN9PR18MB425154FE5019DCAF2028A1D5DB8D9-40BN9PR18MB425 > > > > > 1.namp > > > > > rd18.prod.outlook.com_t_-23u&d=DwIDaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r > > > > > =eTeNT > > > > > LEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=bjgkhgPgOjqCEsbHYHONCZM > > > > > iFDX72 > > > > > MztWaE0AvWBktQVn3zKEDtUdn02Kx_KJ14B&s=SToGsDGEObwbZGilVtVZPyME8j > > > > > NiRgrq > > > > > 4SDYvqqT0TA&e= > > > > > > > > > > It was reported that enabling this workaround fixes instability > > > > > issues and "Unhandled fault" errors when using 60 GHz WiFi > > > > > 802.11ad card with Qualcomm > > > > > QCA6335 chip under significant load which were caused by > > > > > interrupt status stuck in the outbound CMPLT queue traced back to this erratum. > > > > > > > > > > This workaround fixes also kernel panic triggered after some > > > > > minutes of usage 5 GHz WiFi 802.11ax card with Mediatek MT7915 chip: > > > > > > > > > > Internal error: synchronous external abort: 96000210 [#1] SMP > > > > > Kernel panic - not syncing: Fatal exception in interrupt > > > > > > > > > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@xxxxxxxxxxx> > > > > > Signed-off-by: Pali Rohár <pali@xxxxxxxxxx> > > > > > Fixes: 8c39d710363c ("PCI: aardvark: Add Aardvark PCI host > > > > > controller > > > > > driver") > > > > > Cc: stable@xxxxxxxxxxxxxxx > > > > > --- > > > > > drivers/pci/controller/pci-aardvark.c | 10 ++++++++++ > > > > > 1 file changed, 10 insertions(+) > > > > > > > > > > diff --git a/drivers/pci/controller/pci-aardvark.c > > > > > b/drivers/pci/controller/pci-aardvark.c > > > > > index 060936ef01fe..3ae8a85ec72e 100644 > > > > > --- a/drivers/pci/controller/pci-aardvark.c > > > > > +++ b/drivers/pci/controller/pci-aardvark.c > > > > > @@ -210,6 +210,8 @@ enum { > > > > > }; > > > > > > > > > > #define VENDOR_ID_REG (LMI_BASE_ADDR + 0x44) > > > > > +#define DEBUG_MUX_CTRL_REG (LMI_BASE_ADDR + 0x208) > > > > > +#define DIS_ORD_CHK BIT(30) > > > > > > > > > > /* PCIe core controller registers */ > > > > > #define CTRL_CORE_BASE_ADDR 0x18000 > > > > > @@ -558,6 +560,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie) > > > > > PCIE_CORE_CTRL2_TD_ENABLE; > > > > > advk_writel(pcie, reg, PCIE_CORE_CTRL2_REG); > > > > > > > > > > + /* Disable ordering checks, workaround for erratum 3.12 "PCIe completion timeout" */ > > > > > + reg = advk_readl(pcie, DEBUG_MUX_CTRL_REG); > > > > > + reg |= DIS_ORD_CHK; > > > > > + advk_writel(pcie, reg, DEBUG_MUX_CTRL_REG); > > > > > + > > > > > /* Set lane X1 */ > > > > > reg = advk_readl(pcie, PCIE_CORE_CTRL0_REG); > > > > > reg &= ~LANE_CNT_MSK; > > > > > @@ -1581,6 +1588,9 @@ static irqreturn_t advk_pcie_irq_handler(int irq, void *arg) > > > > > struct advk_pcie *pcie = arg; > > > > > u32 status; > > > > > > > > > > + /* Full memory barrier (ARM dsb sy), workaround for erratum 3.12 "PCIe completion timeout" */ > > > > > + mb(); > > > > > + > > > > > status = advk_readl(pcie, HOST_CTRL_INT_STATUS_REG); > > > > > if (!(status & PCIE_IRQ_CORE_INT)) > > > > > return IRQ_NONE; > > > > > -- > > > > > 2.20.1 > > > > > > > > > > > _______________________________________________ > > > linux-arm-kernel mailing list > > > linux-arm-kernel@xxxxxxxxxxxxxxxxxxx > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead. > > > org_mailman_listinfo_linux-2Darm-2Dkernel&d=DwIDaQ&c=nKjWec2b6R0mOyP > > > az7xtfQ&r=eTeNTLEK5-TxXczjOcKPhANIFtlB9pP4lq9qhdlFrwQ&m=2NzkT9KLO26k > > > efUOw2nIeSeRnJVZLxEiBXqEoRvDQ0ueww6n4YaXWgAN1uCJX20o&s=nczACS_2jERbA > > > -c4Gfar0-HTA4PtvZdJmsBv8jhW8G0&e= > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@xxxxxxxxxxxxxxxxxxx > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
