> On Nov 17, 2016, at 7:46 AM, Christoph Hellwig <hch@xxxxxx> wrote: > > On Wed, Nov 16, 2016 at 02:45:33PM -0500, Chuck Lever wrote: >> Out of curiosity, I hacked up my NFS client to limit the size of RDMA >> segments to 30 pages (the server HCA's max_sge). >> >> A 1MB NFS READ now takes 9 segments. That forces the after-conversion >> server to build single-Write chains and use 9 post_send calls to >> transmit the READ payload, just like the before-conversion server. >> >> Performance of before- and after-conversion servers is now equivalent. >> >> kB reclen write rewrite read reread >> 2097152 1024 1061237 1141614 1961410 2000223 > > What HCA is this, btw? ConnectX-3 Pro, f/w 2.31.5050 > Also did you try to always register for > max_sge > calls? The code can already register all segments with the > rdma_rw_force_mr module option, so it would only need a small tweak for > that behavior. For various reasons I decided the design should build one WR chain for each RDMA segment provided by the client. Good clients expose just one RDMA segment for the whole NFS READ payload. Does force_mr make the generic API use FRWR with RDMA Write? I had assumed it changed only the behavior with RDMA Read. I'll try that too, if RDMA Write can easily be made to use FRWR. But I'd like a better explanation for this result. Could be a bug in my implementation, my design, or in the driver. Continuing to investigate. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
