On Mon, Mar 12, 2018 at 9:15 PM, Jason Gunthorpe <jgg@xxxxxxxx> wrote: > On Mon, Mar 12, 2018 at 08:21:25PM -0500, Rohit Zambre wrote: >> On Mon, Mar 12, 2018 at 6:15 PM, Rohit Zambre <rzambre@xxxxxxx> wrote: >> > On Mon, Mar 12, 2018 at 2:37 PM, Jason Gunthorpe <jgg@xxxxxxxx> wrote: >> >> On Sun, Mar 11, 2018 at 08:07:43PM -0500, Rohit Zambre wrote: >> >> >> >>> the different bfregs. However, the Mellanox PRM states that doorbells >> >>> to the same UAR page must be serialized. >> >> >> >> This seems like a nonsense statement to me. doorbell rings are >> >> indivisible 64 bit writes, there is no concept of serialization of >> >> those writes to a PCI-E device. >> > >> > In the "Sharing UARs" section, the PRM states "No other DoorBell can >> > be rung (or even start ringing) in the midst of an on-going write of a >> > DoorBell over a given UAR page... the access to a UAR must be >> > synchronized unless an atomic write of 64 bits in a single bus >> > operation is guaranteed." >> >> Since this statement is talking about the 64-bit DoorBell writes, I >> took a look at where the DoorBell is being written. mlx5_bf_copy calls >> the COPY_64B_NT macro which, for x86, writes 64 bytes into the blue >> flame buffer using double quadword (128-bits) move-instructions. > > mlx5_bf_copy is not the 64 bit doorbell write, it is doing something > else. > > 64 bit doorbell writes are executed atomicly at the PCI-E level or > globally locked. You should not look at this stuff in MOFED, it > doesn't handle the case where 64 bit writes are split, rdma-core does. I see that the 64-bit doorbell write in rdma-core happens with mmio_write64_be() which is atomic. The doorbell write is either the first mmio_write64_be() in mlx5_bf_copy or the one called in post_send_db() (when the conditions for mlx5_bf_copy() are not satisfied). Thanks! -Rohit -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
