> > Broadly speaking, providers are not using the existing macros > consistently and the existing macros are very poorly defined. > > Due to this poor definition we struggled to implement a sensible > barrier for ARM64 and just went with the strongest barriers instead. > > Split wmb/wmb_wc into several cases: > udma_to_device_barrier - Think dma_map(TO_DEVICE) in kernel terms > udma_ordering_write_barrier - Weaker than wmb() in the kernel > mmio_flush_writes - Special to help work with WC memory > mmio_wc_start - Special to help work with WC memory I think you left out the mmio_wc_start() implementation? > mmio_ordered_writes_hack - Stand in for lack an ordered writel() > > rmb becomes: > udma_from_device_barrier - Think dmap_unamp(FROM_DEVICE) in kernel terms > > The split forces provider authors to think about what they are doing more > carefully and the comments provide a solid explanation for when the barrier > is actually supposed to be used and when to use it with the common idioms > all drivers seem to have. > > NOTE: do not assume that the existing asm optimally implements the defined > semantics. The required semantics were derived primarily from what the > existing providers do. > > Signed-off-by: Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> > --- > util/udma_barrier.h | 107 > ++++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 107 insertions(+) > > diff --git a/util/udma_barrier.h b/util/udma_barrier.h > index 57ab0f76cbe33e..f9b8291db20210 100644 > --- a/util/udma_barrier.h > +++ b/util/udma_barrier.h > @@ -122,4 +122,111 @@ > > #endif > > +/* Barriers for DMA. > + > + These barriers are expliclty only for use with user DMA operations. If you > + are looking for barriers to use with cache-coherent multi-threaded > + consitency then look in stdatomic.h. If you need both kinds of synchronicity > + for the same address then use an atomic operation followed by one > + of these barriers. > + > + When reasoning about these barriers there are two objects: > + - CPU attached address space (the CPU memory could be a range of things: > + cached/uncached/non-temporal CPU DRAM, uncached MMIO space in > another > + device, pMEM). Generally speaking the ordering is only relative > + to the local CPU's view of the system. Eg if the local CPU > + is not guarenteed to see a write from another CPU then it is also > + OK for the DMA device to also not see the write after the barrier. > + - A DMA initiator on a bus. For instance a PCI-E device issuing > + MemRd/MemWr TLPs. > + > + The ordering guarentee is always stated between those two streams. Eg what > + happens if a MemRd TLP is sent in via PCI-E relative to a CPU WRITE to the > + same memory location. > +*/ > + > +/* Ensure that the device's view of memory matches the CPU's view of memory. > + This should be placed before any MMIO store that could trigger the device > + to begin doing DMA, such as a device doorbell ring. > + > + eg > + *dma_buf = 1; > + udma_to_device_barrier(); > + mmio_write(DO_DMA_REG, dma_buf); > + Must ensure that the device sees the '1'. > + > + This is required to fence writes created by the libibverbs user. Those > + writes could be to any CPU mapped memory object with any cachability mode. > + > + NOTE: x86 has historically used a weaker semantic for this barrier, and > + only fenced normal stores to normal memory. libibverbs users using other > + memory types or non-temporal stores are required to use SFENCE in their own > + code prior to calling verbs to start a DMA. > +*/ > +#define udma_to_device_barrier() wmb() > + > +/* Ensure that all ordered stores from the device are observable from the > + CPU. This only makes sense after something that observes an ordered store > + from the device - eg by reading a MMIO register or seeing that CPU memory is > + updated. > + > + This guarentees that all reads that follow the barrier see the ordered > + stores that preceded the observation. > + > + For instance, this would be used after testing a valid bit in a memory > + that is a DMA target, to ensure that the following reads see the > + data written before the MemWr TLP that set the valid bit. > +*/ > +#define udma_from_device_barrier() rmb() > + > +/* Order writes to CPU memory so that a DMA device cannot view writes after > + the barrier without also seeing all writes before the barrier. This does > + not guarentee any writes are visible to DMA. > + > + This would be used in cases where a DMA buffer might have a valid bit and > + data, this barrier is placed after writing the data but before writing the > + valid bit to ensure the DMA device cannot observe a set valid bit with > + unwritten data. > + > + Compared to udma_to_device_barrier() this barrier is not required to fence > + anything but normal stores to normal malloc memory. Usage should be: > + > + write_wqe > + udma_to_device_barrier(); // Get user memory ready for DMA > + wqe->addr = ...; > + wqe->flags = ...; > + udma_ordering_write_barrier(); // Guarantee WQE written in order > + wqe->valid = 1; > +*/ > +#define udma_ordering_write_barrier() wmb() > + > +/* Promptly flush writes, possibly in a write buffer, to MMIO backed memory. > + This is not required to have any effect on CPU memory. If done while > + holding a lock then the ordering of MMIO writes across CPUs must be > + guarenteed to follow the natural ordering implied by the lock. > + > + This must also act as a barrier that prevents write combining, eg > + *wc_mem = 1; > + mmio_flush_writes(); > + *wc_mem = 2; > + Must always produce two MemWr TLPs, the '2' cannot be combined with and > + supress the '1'. > + > + This is intended to be used in conjunction with write combining memory > + to generate large PCI-E MemWr TLPs from the CPU. > +*/ > +#define mmio_flush_writes() wc_wmb() > + > +/* Keep MMIO writes in order. > + Currently we lack writel macros that universally guarentee MMIO > + writes happen in order, like the kernel does. Even worse many > + providers haphazardly open code writes to MMIO memory omitting even > + volatile. > + > + Until this can be fixed with a proper writel macro, this barrier > + is a stand in to indicate places where MMIO writes should be switched > + to some future writel. > +*/ > +#define mmio_ordered_writes_hack() mmio_flush_writes() > + > #endif > -- > 2.7.4 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
