Hi Casey On Wed, Aug 02, 2017 at 05:53:52PM +0000, Casey Leedom wrote: > Okay, here you go. As you can tell, it's almost a trivial copy of the > cxgb4 patch. > > By the way, I realized that we have yet another hole which is likely not > to be fixable. If we're dealing with a problematic Root Complex, and we > instantiate Virtual Functions and attach them to a Virtual Machine along > with an NVMe device which can deal with Relaxed Ordering TLPs, the VF driver > in the VM will be able to tell that it shouldn't attempt to send RO TLPs to > the RC because it will see the state of its own PCIe Capability Device > Control[Relaxed Ordering Enable] (a copy of the setting in the VF's > corresponding PF), but it won't be able to change that and send non-RO TLPs > to the RC, and RO TLPs to the NVMe device. Oh well. I don't understand this completely.. So your driver would know not to send RO TLP's to root complex. But you want to send RO to the NVMe device? This is the peer-2-peer case correct? The issue in the current patchset is that we device to turn off the device capability for all devices in the hierarchy so one would expect that the NVMe also would have RO turned off i suppose. The other approach is to not turn off the device capabilty, but let the driver do the right thing. i.e for transactions towards system memory vs. peer-2-peer? But since we wanted to take a big hammer approach because some platforms there can be data-corruption and we can't let trust guest drivers to do the right thing. This isn't something we can fix in this current version. One possible approach is to provide a strict flag, where we use this heavy hammer approach only on platforms that have a serious implication, and the other is we let the driver do the right thing depending on the platform. Worst case if the driver doesn't do the right thing, you would see perf issues but nothing bad would happen. It would allow you to select when to turn on RO and when to turn it off. Cheers, Ashok
