Hi Arnd, On Tue, Feb 12, 2019 at 02:03:04PM +0100, Arnd Bergmann wrote: > On Mon, Feb 11, 2019 at 6:29 PM Will Deacon <will.deacon@xxxxxxx> wrote: > > > + __iomem pointers obtained with non-default attributes (e.g. those returned > > + by ioremap_wc()) are unlikely to provide many of these guarantees. If > > + ordering is required for such mappings, then the mandatory barriers should > > + be used in conjunction with the _relaxed() accessors defined below. > > I wonder if we are even able to guarantee consistent behavior across > architectures > in the last case here (wc mapping with relaxed accessors and barriers). > > Fortunately, there are only five implementations that actually differ from > ioremap_nocache(): arm32, arm64, ppc32, ppc64 and x86 (both 32/64), so > that is something we can probably figure out between the people on Cc. I'd be keen to try to document this as a follow-up patch, otherwise there's the risk of biting off more than I can chew, which is easily done with this stuff! For arm32 (v7) and arm64, ioremap_wc() returns "normal, non-cacheable memory". Some notable differences between this and the memory type returned by ioremap() are: * ioremap_wc() allows speculative reads * ioremap_wc() allows unaligned access * ioremap_wc() allows reordering of accesses to different addresses * ioremap_wc() allows merging of accesses so for us, you really only want to use it to map things that look an awful lot like memory. > The problem with recommending *_relaxed() + barrier() is that it ends > up being more expensive than the non-relaxed accessors whenever > _relaxed() implies the barrier already (true on most architectures other > than arm). > > ioremap_wc() in turn is used almost exclusively to map RAM behind > a bus, (typically for frame buffers) and we may be better off not > assuming any particular MMIO barrier semantics for it at all, but possibly > audit the few uses that are not frame buffers. Right, my expectation is actually that you very rarely need ordering guarantees for wc mappings, and so saying "relaxed + mandatory barriers" is the best thing to say for portable driver code. I'm deliberately /not/ trying to enumerate arch or device-specific behaviours. > > + Since many CPU architectures ultimately access these peripherals via an > > + internal virtual memory mapping, the portable ordering guarantees provided > > + by inX() and outX() are the same as those provided by readX() and writeX() > > + respectively when accessing a mapping with the default I/O attributes. > > This is notably weaker than the PCI mandated non-posted write semantics. > As I said earlier, not all architectures or PCI host implementations can provide > non-posted writes though, but maybe we can document that fact here, e.g. > > | Device drivers may expect outX() to be a non-posted write, i.e. waiting for > | a completion response from the I/O device, which may not be possible > | on a particular hardware. I can add something along these lines, since this seems like it could be a bit of a "gotcha" given the macro names and implicit x86 heritage. > > (*) ioreadX(), iowriteX() > > > > These will perform appropriately for the type of access they're actually > > doing, be it inX()/outX() or readX()/writeX(). > > This probably needs clarification as well then: On architectures that > have a stronger barrier after outX() than writeX() but that use memory > mapped access for both, the statement is currently not true. We could > either strengthen the requirement by requiring CONFIG_GENERIC_IOMAP > on such architectures, or we could document the current behavior > as intentional and explicitly not allow iowriteX() on I/O ports to be posted. At least on arm and arm64, the heavy barrier in outX() is *before* the I/O access, and so it does nothing to prevent the access from being posted. It looks like the asm-generic/io.h behaviour is the same in the case that none of the __io_* barriers are provided by the architecture. Do you think this is something we actually need to strengthen, or are drivers that rely on this going to be x86-specific anyway? > > +All of these accessors assume that the underlying peripheral is little-endian, > > +and will therefore perform byte-swapping operations on big-endian architectures. > > This sounds like a useful addition and the only sane way to do it IMHO, but > I think at least traditionally we've had architectures that do not work like > this but that make readX()/writeX() do native big-endian loads and stores, with > a hardware byteswap on the PCI bus. Sure, hence my disclaimer at the beginning about non-portable drivers :) My goal here is really to document the portable semantics for the common architectures, so that driver developers and reviewers can get the usual case right. Will
