[resending as plain-text. :( gmail is such a PITA - it should just know by now that anything going to vger needs to be plain text. :/ ] On Wed, May 22, 2019 at 6:09 PM John David Anglin <dave.anglin@xxxxxxxx> wrote: > > On 2019-05-22 8:38 p.m., Grant Grundler wrote: > > While PCI bus type is the same, most problems with device drivers are > > at DMA Coherency/Memory ordering level. C3600 has PA8600 processor > > and you might learn more about PA-8600 processor, Astro (IOMMU), and > > Elroy (PCI host controller) from > > https://www.openpa.net/systems/hp-visualize_b1000_c3000_c3600.html > > > > If you can try some experiments, start adding mb() calls after the > > driver adds or removes an IO request from any list or queue. > That's an interesting comment. I was also being very lazy and imprecise. :) More specifically, I was thinking the mb() should be placed AFTER adding any IOs to data structure in memory the device will read but BEFORE the driver tells the device more IO requests. I didn't look if such sequences even exist in the drivers mentioned. If the devices use "mail boxes", completely different issues around ordering can come up. > On a UP kernel, mb() is currently just a compiler > memory barrier. On a SMP kernel, mb() generates a "sync" instruction. We also > use "ldcw" as a barrier in spinlocks. Yeah, I'm not sure how strong the mb() needs to be and maybe I'm giving the wrong advice: use dma_wmb() for the case I've described above. Then use dma_rmb() before reading data structures updated by the device. See examples in the existing code: https://elixir.bootlin.com/linux/v4.20/ident/dma_wmb [It's interesting that mostly networking drivers are using these memory barriers and very few block devices do.] OTOH, limiting the compiler is often sufficient to get "expected behavior" since the caches and memory controllers usually aren't that busy/backlogged that memory accesses can get very much out of order - especially memory reads. Especially for drivers only tested on x86 where everything is pretty strongly ordered and compiler doesn't have many registers to work with. These older drivers are more likely to have issues with RISC compiler ordering accesses into sequences that are "unexpected" (even if perfectly correct from compilers PoV). > I'm thinking dma_rmb() and dma_wmb() may need to be stronger. Is "sync" or "syncdma" > a better choice for these defines? Hrm, maybe but seems like we should be using dma_*mb() calls since they are available. cheers, grant > > Cheers, > Dave > > -- > John David Anglin dave.anglin@xxxxxxxx >
