Another question, from what I understand from the code, the client
always rdma_writes data on writes (with imm) from a remote pool of
server buffers dedicated to it. Essentially all writes are immediate (no
rdma reads ever). How is that different than using send wrs to a set of
pre-posted recv buffers (like all others are doing)? Is it faster?
RDMA WRITE only is generally a bit faster, and if you use a buffer
pool in a smart way it is possible to get very good data packing.
There is no packing, its used exactly as send/recv, but with a remote
buffer pool (pool of 512K buffers) and the client selects one and rdma
write with imm to it.
With
SEND the number of recvq entries dictates how big the rx buffer can
be, or you waste even more memory by using partial send buffers..
This is exactly how it used here.
A scheme like this seems like a high performance idea, but on the
other side, I have no idea how you could possibly manage invalidations
efficiently with a shared RX buffer pool...
There are no invalidations, this remote server pool is registered once
and long lived with the session.
The RXer has to push out an invalidation for the shared buffer pool
MR, but we don't have protocols for partial MR invalidation.
Which is back to my earlier thought that the main reason this perfoms
better is because it doesn't have synchronous MR invalidation.
This issue only exists on the client side. The server never
invalidates any of its buffers.
Maybe this is fine, but it needs to be made very clear that it uses
this insecure operating model to get higher performance..
I still do not understand why this should give any notice-able
performance advantage.
