Okay, thanks for the note Alexander. I'll have to look more closely at the patch on Monday and try it out on one of the targeted systems to verify the semantics you describe. However, that said, there is no way to tell a priori where a device will send TLPs. To simply assume that all TLPs will be directed towards the Root Complex is a big assumption. Only the device and the code controlling it know where the TLPs will be directed. That's why there are changes required in the cxgb4 driver. For instance, the code in drivers/net/ethernet/chelsio./cxgb4/sge.c: t4_sge_alloc_rxq() knows that it's allocating Free List Buffers in Host Memory and that the RX Queues that it's allocating in the Hardware will eventually send Ingress Data to those Free List Buffers. (And similarly for the Free List Buffer Pointer Queue with respect to DMA Reads from the host.) In that routine we explicitly configure the Hardware to use/not-use the Relaxed Ordering Attribute via the FW_IQ_CMD_FL0FETCHRO and FW_IQ_CMD_FL0DATARO flags. Basically we're conditionally setting them based on the desirability of sending Relaxed Ordering TLPs to the Root Complex. (And we would perform the same kind of check for an nVME application ... which brings us to ...) And what would be the code using these patch APIs to set up a Peer-to-Peer nVME-style application? In that case we'd need the Chelsio adapter's PCIe Capability Device Control[Relaxed Ordering Enable] set for the nVME application ... and we would avoid programming the Chelsio Hardware to use Relaxed Ordering for TLPs directed at the Root Complex. Thus we would be in a position where some TLPs being emitted by the device to Peer devices would have Relaxed Ordering set and some directed at the Root Complex would not. And the only way for that to work is if the source device's Device Control[Relaxed Ordering Enable] is set ... Finally, setting aside my disagreements with the patch, we still have the code in the cxgb4 driver which explicitly turns on its own Device Control[Relaxed Ordering Enable] in cxgb4_main.c: enable_pcie_relaxed_ordering(). So the patch is something of a loop if all we're doing is testing our own Relaxed Ordering Enable state ... Casey
