On Thu, Jan 02, 2020 at 07:44:36PM +0200, Liran Alon wrote:
> Currently, mlx5e_notify_hw() executes wmb() to complete writes to cache-coherent
> memory before ringing doorbell. Doorbell is written to by mlx5_write64()
> which use __raw_writeX().
>
> This is semantically correct but executes reduntant wmb() in some architectures.
> For example, in x86, a write to UC memory guarantees that any previous write to
> WB memory will be globally visible before the write to UC memory. Therefore, there
> is no need to also execute wmb() before write to doorbell which is mapped as UC memory.
>
> The consideration regarding this between different architectures is handled
> properly by the writeX() macro. Which is defined differently for different
> architectures. E.g. On x86, it is just a memory write. However, on ARM, it
> is defined as __iowmb() folowed by a memory write. __iowmb() is defined
> as wmb().
This reasoning seems correct, though I would recommend directly
refering to locking/memory-barriers.txt which explains this.
> Therefore, change mlx5_write64() to use writeX() and remove wmb() from
> it's callers.
Yes, wmb(); writel(); is always redundant
> diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
> index 40748fc1b11b..28744a725e64 100644
> +++ b/include/linux/mlx5/cq.h
> @@ -162,11 +162,6 @@ static inline void mlx5_cq_arm(struct mlx5_core_cq *cq, u32 cmd,
>
> *cq->arm_db = cpu_to_be32(sn << 28 | cmd | ci);
>
> - /* Make sure that the doorbell record in host memory is
> - * written before ringing the doorbell via PCI MMIO.
> - */
> - wmb();
> -
Why did this one change? The doorbell memory here is not a writel():
> doorbell[0] = cpu_to_be32(sn << 28 | cmd | ci);
> doorbell[1] = cpu_to_be32(cq->cqn);
> static inline void mlx5_write64(__be32 val[2], void __iomem *dest)
> {
> #if BITS_PER_LONG == 64
> - __raw_writeq(*(u64 *)val, dest);
> + writeq(*(u64 *)val, dest);
I want to say this might cause problems with endian swapping as writeq
also does some swaps that __raw does not? Is this true?
ie writeq does not accept a be32
Some time ago I reworked this similar code in userspace to use a u64
and remove the swapping from the caller.
Jason
