On Fri, Jan 03, 2020 at 07:52:07PM +0200, Liran Alon wrote:
> diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
> index 40748fc1b11b..4631ad35da53 100644
> +++ b/include/linux/mlx5/cq.h
> @@ -162,13 +162,8 @@ static inline void mlx5_cq_arm(struct mlx5_core_cq *cq, u32 cmd,
>
> *cq->arm_db = cpu_to_be32(sn << 28 | cmd | ci);
>
> - /* Make sure that the doorbell record in host memory is
> - * written before ringing the doorbell via PCI MMIO.
> - */
> - wmb();
> -
> - doorbell[0] = cpu_to_be32(sn << 28 | cmd | ci);
> - doorbell[1] = cpu_to_be32(cq->cqn);
> + doorbell[0] = sn << 28 | cmd | ci;
> + doorbell[1] = cq->cqn;
This does actually have to change to a u64 otherwise it is not the
same.
On x86 LE, it was
db[0] = swab(a)
db[1] = swab(b)
__raw_writel(db)
Now it is
db[0] = a
db[1] = b
__raw_writel(swab(db))
Putting the swab around the u64 swaps the order of a/b in the TLP.
It might be tempting to swap db[0]/db[1] but IIRC this messed it up on
BE.
The sanest, simplest solution is to use a u64 natively, as the example
I gave did.
There is also the issue of casting a u32 to a u64 and possibly
triggering a unaligned kernel access, presumably this doesn't happen
today only by some lucky chance..
> mlx5_write64(doorbell, uar_page + MLX5_CQ_DOORBELL);
> }
> diff --git a/include/linux/mlx5/doorbell.h b/include/linux/mlx5/doorbell.h
> index 5c267707e1df..9c1d35777323 100644
> +++ b/include/linux/mlx5/doorbell.h
> @@ -43,17 +43,15 @@
> * Note that the write is not atomic on 32-bit systems! In contrast to 64-bit
> * ones, it requires proper locking. mlx5_write64 doesn't do any locking, so use
> * it at your own discretion, protected by some kind of lock on 32 bits.
> - *
> - * TODO: use write{q,l}_relaxed()
> */
>
> -static inline void mlx5_write64(__be32 val[2], void __iomem *dest)
> +static inline void mlx5_write64(u32 val[2], void __iomem *dest)
> {
So this should accept a straight u64, the goofy arrays have to go away
> #if BITS_PER_LONG == 64
> - __raw_writeq(*(u64 *)val, dest);
> + iowrite64be(*(u64 *)val, dest);
> #else
> - __raw_writel((__force u32) val[0], dest);
> - __raw_writel((__force u32) val[1], dest + 4);
> + iowrite32be(val[0], dest);
> + iowrite32be(val[1], dest + 4);
With a u64 input this fallback is written as
iowrite32be(val >> 32, dest)
iowrite32be((u32)val, dest + 4)
Which matches the definition for how write64 must construct a TLP.
And arguably the first one should be _relaxed (but nobody cares about
this code path)
Jason
