> On Jun 7, 2016, at 4:49 PM, Jason Gunthorpe <jgunthorpe@xxxxxxxxxxxxxxxxxxxx> wrote: > > On Tue, Jun 07, 2016 at 03:47:32PM -0400, Chuck Lever wrote: >> The fixed part of an R_key is 24 bits, but the variant part is only >> 8 bits, allowing 256 unique R_keys concurrenly in flight on one QP. >> >> Thus an rpcrdma_xprt cannot use more than 256 rpcrdma_mws at a time. >> >> Capping the number of rpcrdma_mws should prevent ro_map from >> re-using an R_key, which could change the MR co-ordinates or access >> settings while an RPC is still in progress. It also reduces the >> number of pre-allocated FRMRs per transport, allowing more >> transports per device. >> >> To get more R_keys in flight at once, additional QPs will be >> necessary. > > ?? > > The 24 bit / 8 bit thing doesn't create new MRs, it just disambiguates > the life cycle of a single MR. Right. > Further, the MR is connected to a PD, > not a QP, there is no such thing as '256 unique R_keys concurrenly in > flight on one QP' ??? OK, fair enough, but... > This is why no ULP has ever needed this constant before: > >> +enum { >> + RPCRDMA_RKEYS_PER_QP = 256 >> +}; > > Because you never need to care when working with FRMRs. > > Either the number in queue is limited by the invalidation > synchronization point to 2, or in the case of iwarp read, it doesn't > matter since everything executes serially in order and the value can > safely wrap. I don't understand this. How does invalidation prevent the ULP from registering more R_keys? And, I'd like to limit the number of pre-allocated MRs to no more than each transport endpoint can use. For xprtrdma, a transport is one QP and PD. What is that number? -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html