On Fri, 1 Jul 2005, Alan Cox wrote: > > No it's not. You need to insert appropriate barriers, one of: wmb(), > > mb() or rmb(). In rare cases you may need to use iob(), which is > > currently non-portable (which reminds me I should really push it > > upstream). > > Its even more complicated than that 8) > > writeb/writel may be merged in some cases (but not re-ordered) for I/O Is that non-reordering specified anywhere for the API or does it just happen to be satisfied by most implementations? Ours (for MIPS, that is) for example does nothing to ensure that. > devices but a simple mb() will only synchronize them as viewed from > cpu/memory interface. There are two other synchronization points. From That's true -- which is why I mentioned bridge-specific operations may be required. > the bridge with the I/O device (typically the PCI root bridge) which is > not enforced automatically across processors on some large numa boxes > but is not usually a problem and on the PCI bus itself. What if the host I/O bus is not PCI? For this kind of stuff I tend to think in the terms of TURBOchannel systems, just to be sure not to get influenced by the most common hardware. ;-) E.g. I have this R4400-based TURBOchannel system with aggressive buffering in the CPU's MB (memory buffer) ASIC which requires a read-back (RAM is OK for that) after a write and a memory barrier only to make writes propagate to the I/O bridge. It may be worse yet with TURBOchannel Alpha and VAX systems. With the latters TURBOchannel is behind two bridges, with two intermediate buses on the way. > PCI permits posting (delaying writes) and some forms of merging (but not > re-ordering). Thus if you need an I/O to hit a device on the PCI bus and > know it arrived you must follow it by a read from the same device. So > for example if you want to shut down a DMA transfer and free the buffer > for a PCI device you > need to do > > writel(TURN_DMA_OFF, dev->control); > readl(dev->something); > /* Only now is the free safe */ Again, the I/O bus your host is attached to need not be PCI and you may need a bridge specific operation to make your write be completed, possibly combined with your quoted sequence (if there is actually PCI somewhere in the system; think AlphaServer 8400). Maciej
