On Fri, Jul 7, 2017 at 5:30 PM, Casey Leedom <leedom@xxxxxxxxxxx> wrote: > By the way, it ~seems~ like the patch set confuses the idea of the PCIe Capability Device Control[Relaxed Ordering Enable] with the device's ability to handle incoming TLPs with the Relaxed Ordering Attribute set. These are completely different things. The PCIe Capability Device Control[Relaxed Ordering Enable] solely governs the ability of the device to _send_ TLPs with the Relaxed Ordering Attribute set. It has nothing whatsoever to do with the handling of incoming TLPs with the Relaxed Ordering Attribute set. In fact, there is any standard way to disable receipt processing of such TLPs. That's kind of the whole point of the majority of the patch. If there was a separate "Ignore Incoming Relaxed Ordering Attributes" then we'd mostly just have a quirk which would set that for problematic devices. > > Worse yet, if I'm reading the patch right, it _is_ turning off the PCIe Capability Device Control[Relaxed Ordering Enable] for the devices that we've identified with problematic receive handling. Not only does this not solve the problem that we've been talking about, it actually almost certainly introduces a huge Graphics Performance Bug. The Relaxed Ordering Attribute was originally developed for Graphics Device support in order to download textures, etc. to a graphics devices as fast as possible and only ensure ordering at points where needed. For instance, as far as I know, the Intel Root Complexes that we've been talking about have no issues whatsoever in generating downstream TLPs with the Relaxed Ordering Attribute set. By turning off the PCIe Capability Device Control[Relaxed Ordering Enable] on these Root Complexes, this patch prevents the generation of downstream TLPs with the Relaxed Ordering Attribute set targeted at devices like graphics adapters. > > Casey The patches should be disabling the relaxed ordering on upstream facing ports that would be sending requests to the root complex. So if the root complex cannot handle receiving a message with relaxed ordering enabled we should be clearing the relaxed ordering bit in the PCIe configuration on those device that could send TLPs to that root complex. By doing that the devices hosted on that root complex cannot generate requests that contain relaxed ordering TLPs. I'll have to do some double checking of the patches, but I thought we were only clearing the relaxed ordering bits on the devices that would be sending requests to the root complex, not the downstream ports of the root complex itself. As such we should be able to generate relaxed ordering TLPs from the root complex, but the devices hosted on the bus shouldn't be able to generate relaxed ordering TLPs. The bit in the configuration space controls the ability to generate content, not the ability to receive it so clearing the bit on the device should be the correct behavior for this. As far as your suggestions for the cxgb4 patch it has the same problem it originally had. We want to disable the relaxed ordering bit on the device since the device should not be generating relaxed ordering requests. This is why I was trying to get your input on this as I know you have a peer-to-peer configuration that you wanted to support. What we need to do is identify what platforms cannot support relaxed ordering to the root complex and prevent that from happening which is why we clear the relaxed ordering bit on the device. In that setup we need to have a good way to identify when this has occurred and instead setup a side channel type configuration where we then re-enable relaxed ordering, but only allow for sending directly to the peer and not the root complex. - Alex
