On Tue, 20 Oct 2020 17:15:42 +1100 Benjamin Herrenschmidt wrote: > On Mon, 2020-10-19 at 19:57 -0700, Jakub Kicinski wrote: > > > I suspect the problem is that the HW (and yes this would be a HW bug) > > > doesn't order the CPU -> memory and the CPU -> MMIO path. > > > > > > What I think happens is that the store to txde0 is potentially still in > > > a buffer somewhere on its way to memory, gets bypassed by the store to > > > MMIO, causing the MAC to try to read the descriptor, and getting the > > > "old" data from memory. > > > > I see, but in general this sort of a problem should be resolved by > > adding an appropriate memory barrier. And in fact such barrier should > > (these days) be implied by a writel (I'm not 100% clear on why this > > driver uses iowrite, and if it matters). > > No, a barrier won't solve this I think. > > This is a coherency problem at the fabric/interconnect level. I has to > do with the way they implemented the DMA path from memory to the > ethernet controller using a different "port" of the memory controller > than the one used by the CPU, separately from the MMIO path, with no > proper ordering between those busses. Old school design .... and > broken. > > By doing a read back, they probably force the previous write to memory > to get past the point where it will be visible to a subsequent DMA read > by the ethernet controller. Thanks for the explanation. How wonderful :/ It'd still be highly, highly preferable if the platform was conforming to the Linux memory model. IO successors (iowrite32 / writel) must ensure previous DRAM writes had completed. For performance sensitive ops, which don't require ordering we have writel_relaxed etc. I assume the DRAM controller queue is a straight FIFO and we don't have to worry about hitting the same address, so how about we add a read of some known uncached address in iowrite32 / writel?
