On Mon, Feb 20, 2017 at 07:46:02PM +0200, Yishai Hadas wrote: > > pthread_spin_lock(&qp->sq.lock); > > > >+ /* Get all user DMA buffers ready to go */ > >+ udma_to_device_barrier(); > >+ > > /* XXX check that state is OK to post send */ > > Not clear why do we need here an extra barrier ? what is the future > optimization that you pointed on ? Writes to different memory types are not guarenteed to be strongly ordered. This is narrowly true on x86-64 too apparently. The purpose of udma_to_device_barrier is to serialize all memory types. This allows the follow on code to use the weaker 'udma_ordering_write_barrier' when it knows it is working exclusively with cached memory. Eg in an ideal world on x86 udma_to_device_barrier should be SFENCE and 'udma_ordering_write_barrier' should be compiler_barrier() Since the barriers have been so ill defined and wonky on x86 other arches use much stronger barriers than they actually need for each cases. Eg ARM64 can switch to much weaker barriers. Since there is no cost on x86-64 to this barrier I would like to leave it here as it lets us actually optimize the ARM and other barriers. If you take it out then 'udma_ordering_write_barrier' is forced to be the strongest barrier on all arches. > >@@ -478,7 +481,7 @@ out: > > * Make sure that descriptor is written to memory > > * before writing to BlueFlame page. > > */ > >- wmb(); > >+ mmio_wc_start(); > > Why to make this change which affects at least X86_64 ? the data was > previously written to the host memory so we expect that wmb is > enough here. Same as above, writes to different memory types are not strongly ordered and wmb() is not the right barrier to serialize writes to DMA'able ctrl with the WC memcpy. If this is left wrong then again other arches are again required to adopt the strongest barrier for everything which hurts them. Even on x86, it is very questionable to not have the SFENCE in that spot. AFAIK it is not defined to be strongly ordered. mlx5 has the SFENCE here, for instance. Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
