> On 2 Jan 2020, at 21:29, Jason Gunthorpe <jgg@xxxxxxxx> wrote: > > On Thu, Jan 02, 2020 at 07:44:36PM +0200, Liran Alon wrote: >> Currently, mlx5e_notify_hw() executes wmb() to complete writes to cache-coherent >> memory before ringing doorbell. Doorbell is written to by mlx5_write64() >> which use __raw_writeX(). >> >> This is semantically correct but executes reduntant wmb() in some architectures. >> For example, in x86, a write to UC memory guarantees that any previous write to >> WB memory will be globally visible before the write to UC memory. Therefore, there >> is no need to also execute wmb() before write to doorbell which is mapped as UC memory. >> >> The consideration regarding this between different architectures is handled >> properly by the writeX() macro. Which is defined differently for different >> architectures. E.g. On x86, it is just a memory write. However, on ARM, it >> is defined as __iowmb() folowed by a memory write. __iowmb() is defined >> as wmb(). > > This reasoning seems correct, though I would recommend directly > refering to locking/memory-barriers.txt which explains this. I find memory-barriers.txt not explicit enough on the semantics of writeX(). (For example: Should it flush write-combined buffers before writing to the UC memory?) That’s why I preferred to explicitly state here how I perceive it. But I don’t mind of course adding a pointer to the memory-barriers.txt file. > >> Therefore, change mlx5_write64() to use writeX() and remove wmb() from >> it's callers. > > Yes, wmb(); writel(); is always redundant Well, unfortunately not… See: https://marc.info/?l=linux-netdev&m=157798859215697&w=2 (See my suggestion to add flush_wc_writeX()) > >> diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h >> index 40748fc1b11b..28744a725e64 100644 >> +++ b/include/linux/mlx5/cq.h >> @@ -162,11 +162,6 @@ static inline void mlx5_cq_arm(struct mlx5_core_cq *cq, u32 cmd, >> >> *cq->arm_db = cpu_to_be32(sn << 28 | cmd | ci); >> >> - /* Make sure that the doorbell record in host memory is >> - * written before ringing the doorbell via PCI MMIO. >> - */ >> - wmb(); >> - > > Why did this one change? The doorbell memory here is not a writel(): Well, it’s not seen in the diff but actually the full code is: /* Make sure that the doorbell record in host memory is * written before ringing the doorbell via PCI MMIO. */ wmb(); doorbell[0] = cpu_to_be32(sn << 28 | cmd | ci); doorbell[1] = cpu_to_be32(cq->cqn); mlx5_write64(doorbell, uar_page + MLX5_CQ_DOORBELL); i.e. doorbell is not the write to the doorbell itself. It’s still done via mlx5_write64(). > >> doorbell[0] = cpu_to_be32(sn << 28 | cmd | ci); >> doorbell[1] = cpu_to_be32(cq->cqn); > >> static inline void mlx5_write64(__be32 val[2], void __iomem *dest) >> { >> #if BITS_PER_LONG == 64 >> - __raw_writeq(*(u64 *)val, dest); >> + writeq(*(u64 *)val, dest); > > I want to say this might cause problems with endian swapping as writeq > also does some swaps that __raw does not? Is this true? Hmm... Looking at ARM64 version, writeq() indeed calls cpu_to_le64() on parameter before passing it to __raw_writeq(). Quite surprising from API perspective to be honest. So should I change this instead to iowrite64be(*(u64 *)val, dest)? -Liran > > ie writeq does not accept a be32 > > Some time ago I reworked this similar code in userspace to use a u64 > and remove the swapping from the caller. > > Jason
