> On Aug 8, 2017, at 2:28 PM, Chuck Lever <chuck.lever@xxxxxxxxxx> wrote: > > >> On Aug 2, 2017, at 3:20 AM, Leon Romanovsky <leon@xxxxxxxxxx> wrote: >> >> On Tue, Aug 01, 2017 at 05:06:51PM -0400, Chuck Lever wrote: >>> Since I converted NFSD to use the new rdma_rw API, I've been >>> struggling with how NFSD determines the size of its send CQ, and how >>> it provisions its send queue limit (ie, when it waits for more send >>> queue entries to become available). >>> >>> The problem arises because rdma_rw hides the exact number of send >>> WQEs it has provisioned. It determines this number based on the MR >>> registration mode (FRWR for iWARP devices, local DMA rkey for >>> others), but the mode itself is no longer exposed to ULPs. >>> >>> Thus when FRWR is in use, rdma_rw adds more WQEs, but NFSD is no >>> longer aware of this, does not provision a larger CQ, and does not >>> raise its send queue accounting limit. That means the extra WQEs are >>> never really used. >>> >>> At LSF this year, Christoph and Sagi proposed a simple API that >>> expose information about how many CQ entries are needed so that ULPs >>> can provision their CQs more accurately. >> >> Indeed clean API. Does it make sense to get rid of sc_max_requests and >> use this exported value directly? > > I'm going to consider this clean up for a later patch. > > So, how shall I submit this new API for upstream? I propose that > I submit all three of these via Bruce's tree, with an Acked-by > from Doug on the "Add rdma_rw_mr_payload()" patch, if Doug > approves. Doug? >>> The first patch here is a pre-requisite for the third. The second >>> patch adds the new API. The third patch changes NFSD to use the new >>> API to provision its CQ and send queue accounting accurately. >>> >>> This approach was tested using the (fixed) force_mr core module >>> parameter. >> >> Good, I liked the word "tested" in the sentence :). >> >>> >>> >>> --- >>> >>> Chuck Lever (3): >>> svcrdma: Limit RQ depth >>> rdma core: Add rdma_rw_mr_payload() >>> svcrdma: Estimate Send Queue depth properly >>> >>> >>> drivers/infiniband/core/rw.c | 24 ++++++++++++++++++++ >>> include/rdma/rw.h | 2 ++ >>> net/sunrpc/xprtrdma/svc_rdma_transport.c | 36 +++++++++++++++++++++--------- >>> 3 files changed, 51 insertions(+), 11 deletions(-) >>> >>> -- >>> Chuck Lever >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > Chuck Lever > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
