This series overhauls the "reply send" side of the RPC-over-RDMA transport to use the new rdma_rw API. No significant performance changes were noticed with this overhaul. Benefits include: <> Better scalability -- fewer ib_post_send calls per RPC Reply <> Better code modularity -- less code duplication with other ULPs <> Ability for svcrdma to use any registration mode for RDMA Writes <> Correct handling of RPCs that have both a Write and a Reply chunk <> Correct handling of Write chunk overrun This revision makes two scalability improvements that Christoph and Sagi requested in previous review comments: 1. Use sg_alloc_table_chained to allocate the scatterlist for each rdma_rw_ctx. A permanent part of the scatterlist is used for smaller I/Os, and a chained part is allocated on demand when performing large I/Os. This reduces the amount of memory tied down per client. 2. So far each rdma_rw_ctx has been posted individually. Now all rdma_rw_ctx's for one chunk are chained together and posted with a single ib_post_send. This reduces the number of ib_post_send operations for complex RPC Replies. Bruce, IMO these are ready for your review. Available in the "nfsd-rdma-for-4.12" topic branch of this git repo: git://git.linux-nfs.org/projects/cel/cel-2.6.git Or for browsing: http://git.linux-nfs.org/?p=cel/cel-2.6.git;a=log;h=refs/heads/nfsd-rdma-for-4.12 Changes since v2: - Rebased on v4.11-rc5 - Use chained sg_tables - Always one ib_post_send per chunk Changes since v1: - Rebased on v4.11-rc4 - Dropped clean up of svc_rdma_post_recv - Postponed addition of code related to recvfrom - Made svc_rdma_{get,put}_rw_ctxt behave symmetrically - Converted scatterlist to sg_table - Moved transport header construction into svc_rdma_sendto.c --- Chuck Lever (14): svcrdma: Move send_wr to svc_rdma_op_ctxt svcrdma: Add svc_rdma_map_reply_hdr() svcrdma: Eliminate RPCRDMA_SQ_DEPTH_MULT svcrdma: Add helper to save pages under I/O svcrdma: Clean up svc_rdma_get_inv_rkey() svcrdma: Introduce local rdma_rw API helpers svcrdma: Use rdma_rw API in RPC reply path svcrdma: Clean up RDMA_ERROR path svcrdma: Report Write/Reply chunk overruns svcrdma: Clean up RPC-over-RDMA backchannel reply processing svcrdma: Reduce size of sge array in struct svc_rdma_op_ctxt svcrdma: Remove unused RDMA Write completion handler svcrdma: Remove the req_map cache svcrdma: Clean out old XDR encoders include/linux/sunrpc/rpc_rdma.h | 3 include/linux/sunrpc/svc_rdma.h | 75 +- net/sunrpc/Kconfig | 1 net/sunrpc/xprtrdma/Makefile | 2 net/sunrpc/xprtrdma/svc_rdma.c | 8 net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 71 +- net/sunrpc/xprtrdma/svc_rdma_marshal.c | 89 --- net/sunrpc/xprtrdma/svc_rdma_recvfrom.c | 79 ++ net/sunrpc/xprtrdma/svc_rdma_rw.c | 512 +++++++++++++++ net/sunrpc/xprtrdma/svc_rdma_sendto.c | 978 ++++++++++++++-------------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 110 --- 11 files changed, 1131 insertions(+), 797 deletions(-) create mode 100644 net/sunrpc/xprtrdma/svc_rdma_rw.c -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
