On Tue, Jul 09, 2019 at 12:45:57PM -0700, Sagi Grimberg wrote: > Another question, from what I understand from the code, the client > always rdma_writes data on writes (with imm) from a remote pool of > server buffers dedicated to it. Essentially all writes are immediate (no > rdma reads ever). How is that different than using send wrs to a set of > pre-posted recv buffers (like all others are doing)? Is it faster? RDMA WRITE only is generally a bit faster, and if you use a buffer pool in a smart way it is possible to get very good data packing. With SEND the number of recvq entries dictates how big the rx buffer can be, or you waste even more memory by using partial send buffers.. A scheme like this seems like a high performance idea, but on the other side, I have no idea how you could possibly manage invalidations efficiently with a shared RX buffer pool... The RXer has to push out an invalidation for the shared buffer pool MR, but we don't have protocols for partial MR invalidation. Which is back to my earlier thought that the main reason this perfoms better is because it doesn't have synchronous MR invalidation. Maybe this is fine, but it needs to be made very clear that it uses this insecure operating model to get higher performance.. Jason
