After seeing Trond’s patches for NFS multipathing on NFSv4.1, we decided to try using the same concept for NFSv3/4. The primary issue we identified was XID collision in the duplicate request cache (replay cache) for NFSv3/4. In NFSv3/4, entries are hashed based on XID instead of the slot ID and sequence ID that NFSv4.1 uses. Since the XIDs are generated by the RPC transports, and Trond’s patches create multiple transports for multipathing, different transports can end up using an overlapping set of XIDs. To fix this, we apply a mask to XIDs. Each transport is constrained to its own segment of the total XID range, and they can never overlap. In terms of loss of entropy, by masking out just enough bits from the XID, we are convinced that the probability of XID wraparound or collision on NFS client restart has not increased to a problematic level (so long as the RPCs are distributed round-robin, as in Trond’s patches). We tested multipathing out and discovered that it enables NFS to get more bandwidth on a bonded interface (instead of using only one physical link, it can use multiple). Specifically, we tested on a setup where the client was connected to the server via 4 bonded 10Gb/s links. Without multipathing, the client could only achieve 10Gb/s (using one physical link). With multipathing, the client was able to achieve a maximum of close to 40Gb/s. However, although the maximum performance was close to 40Gb/s, achieving an average throughput of even 30Gb/s required many connections. The performance of individual trials had a high variance. We traced this uneven performance to colliding network paths. With round-robin distribution of RPCs, no single TCP connection can exceed the performance of the slowest one. If the connections are distributed unevenly across network paths, some connections can bottleneck others. To solve this problem, we are currently working on patches to provide load-balancing as an alternative to round-robin for distributing RPCs. To use these patches, you first have to apply Trond's 5 patches (Available at https://www.spinics.net/lists/linux-nfs/msg63368.html). Let us know what you think or if you have any ideas for improving this. Jui-Yu Chang (1): NFS: Allow multiple connections to NFSv3 and NFSv4.0 servers Bennett Amodio (1): SUNRPC: Mask XIDs to prevent replay cache collision fs/nfs/client.c | 3 +++ fs/nfs/nfs4client.c | 2 +- include/linux/sunrpc/xprt.h | 5 +++++ net/sunrpc/clnt.c | 8 ++++++++ net/sunrpc/xprt.c | 14 ++++++-------- 5 files changed, 23 insertions(+), 9 deletions(-) -- 1.9.1 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
