On 3/12/20 1:58 AM, zerons wrote:
[...]
MR won't stay in the rbtree with force flag. If the MR is used or
use_once is set in both cases its removed from the tree.
See "if (mr->r_use_once || force)"
Sorry, I may misunderstand. Did you mean that if the MR is *used*,
it is removed from the tree with or without the force flag in
rds_rdma_unuse(), even when r_use_once is not set?
Once the MR is being used with use_once semantics it gets removed with or without remote side indicating it via extended header. use_once
optimization was added later. The base behavior is once the MR is
used by remote and same information is sent via extended header,
it gets cleaned up with force flag. Force flag ignores whether
its marked as used_once or not.
Sorry, I am still confused.
I check the code again. The rds_rdma_unuse() is called in two functions,
rds_recv_incoming_exthdrs() and rds_sendmsg().
In rds_sendmsg(), it calls rds_rdma_unuse() *with* force flag only when
the user included a RDMA_MAP cmsg *and* sendmsg() is failed.
correct.
In rds_recv_incoming_exthdrs(), the force is *false*. So we can consider
the rds_rdma_unuse() called *without* force flag.
Then I go check where r_use_once can be set.
__rds_rdma_map()
rds_get_mr()
rds_setsockopt()
rds_get_mr_for_dest()
rds_setsockopt()
rds_cmsg_rdma_map()
rds_cmsg_send()
rds_sendmsg()
It seems to me that r_use_once is controlled by user applications.
yes it is and its being set in the application using this in
production. But You do have point that if application don't set it
then even after MR being used and remote node indicated it being
used, the MR still remains in the RB tree.
Sorry to keep bothering you with my questions. I wish I had such a device
that I can test it on.
Not at all. You mostly found a race condition when use_once is not used
but need to verify it. We will look into it more. Thanks for your
patience.
Regards,
Santosh
