| From: Lucas Stach <l.stach@xxxxxxxxxxxxxx> | Sent: Friday, April 28, 2017 1:51 AM | | Am Donnerstag, den 27.04.2017, 12:19 -0500 schrieb Bjorn Helgaas: | > | > | > I thought Relaxed Ordering was an optimization. Are there cases where | > it is actually required for correct behavior? | | Yes, at least the Tegra 2 TRM claims that RO needs to be enabled on the | device side for correct operation with the following language: | | "Tegra 2 requires relaxed ordering for responses to downstream requests | (responses can pass writes). It is possible in some circumstances for PCIe | transfers from an external bus masters (i.e. upstream transfers) to become | blocked by a downstream read or non-posted write. The responses to these | downstream requests are blocked by upstream posted writes only when PCIe | strict ordering is imposed. It is therefore necessary to never impose strict | ordering that would block a response to a downstream NPW/read request and | always set the relaxed ordering bit to 1. Only devices that are capable of | relaxed ordering may be used with Tegra 2 devices." (woof) Reading through the above paragraph is difficult because the author seems to shift language and terminology mid sentence and isn't following standard PCI terminology conventions. The Root Complex is "Upstream", a non-Root Complex Node in the PCIe Fabric is "Downstream", Requests that a Downstream Device (End Point) send to the Root Complex are called "Upstream Requests", responses that the Root Complex send to a Device are called "Downstream Responses" (or, even more pedantically, "Responses sent Downstream for an earlier Upstream Request"). Because a Root Complex is Upstream, but the Requests it sent Downstream, and Downstream Devices send their Requests Upstream, it's very important that we use exceedingly precise language. So, it ~sounds like~ the nVidia Tegra 2 document is talking about the need for Downstream Devices to echo the Relaxed Ordering Attribute in their Responses directed Upstream to Requests sent Downstream from the Root Complex. Moreover, there's code in drivers/pci/host/pci-tegra.c: tegra_pcie_relax_enable() which appears to set the PCIe Capability Device Control[Enable Relaxed Ordering] bit on all PCIe Fabric Nodes. If my reading of the intent of the nVidia document is correct -- and that's a Big If because of the extremely imprecise language used -- that means that the tegra_pcie_relax_enable() is completely bogus. The PCIe 3.0 Specification states that Responses MUST reflect the Relaxed Ordering and No Snoop Attributes of the Requests for which they are responding. Section 2.2.9 of PCI Express(r) Base Specification Revision 3.0 November 10, 2010: "Completion headers must supply the same values for the Attribute as were supplied in the header of the corresponding Request, except as explicitly allowed when IDO is used." And, specifically, the PCIe Capability Device Control[Enable Relaxed Ordering] bit _only_ affects the ability of that Device to originate Transaction Layer Packet Requests with the Relaxed Ordering Attribute set. Thus, tegra_pcie_relax_enable() setting those bits on all the Downstream Devices (and intervening Bridges) does not _cause_ those Devices to generate Requests with Relaxed Ordering set. And, if the Devices are PCIe 3.0 compliant, it also doesn't affect the Responses that they send back Upstream to the Root Complex. I apologize for the incredibly detailed nature of these responses, but it's very easy for people new to PCIe to get these things wrong and/or misinterpret the PCIe Specifications. Casey
