On Fri, 21 Nov 2014 14:19:27 -0500 Jeff Layton <jlayton@xxxxxxxxxxxxxxx> wrote: > Hi Bruce! > > Here are the patches that I had mentioned earlier that reduce the > contention for the pool->sp_lock when the server is heavily loaded. > > The basic problem is that whenever a svc_xprt needs to be queued up for > servicing, we have to take the pool->sp_lock to try and find an idle > thread to service it. On a busy server, that lock becomes highly > contended and that limits the throughput. > > This patchset fixes this by changing how we search for an idle thread. > First, we convert svc_rqst and the sp_all_threads list to be > RCU-managed. Then we change the search for an idle thread to use the > sp_all_threads list, which now can be done under the rcu_read_lock. > When there is an available thread, queueing an xprt to it can now be > done without any spinlocking. > > With this, we see a pretty substantial increase in performance on a > larger-scale server that is heavily loaded. Chris has some preliminary > numbers, but they need to be cleaned up a bit before we can present > them. I'm hoping to have those by early next week. > > Jeff Layton (4): > sunrpc: add a rcu_head to svc_rqst and use kfree_rcu to free it > sunrpc: fix potential races in pool_stats collection > sunrpc: convert to lockless lookup of queued server threads > sunrpc: add some tracepoints around enqueue and dequeue of svc_xprt > > include/linux/sunrpc/svc.h | 12 +- > include/trace/events/sunrpc.h | 98 +++++++++++++++- > net/sunrpc/svc.c | 17 +-- > net/sunrpc/svc_xprt.c | 252 ++++++++++++++++++++++++------------------ > 4 files changed, 258 insertions(+), 121 deletions(-) > Here's what I've got so far. This is just a chart that shows the % increase in the number of iops in a distributed test on a NFSv3 server with this patchset vs. without. The numbers along the bottom show the number of total job threads running. Chris says: "There were 64 nfsd threads running on the server. There were 7 hypervisors running 2 VMs each running 2 and 4 threads per VM. Thus, 56 and 112 threads total." Cheers! -- Jeff Layton <jlayton@xxxxxxxxxxxxxxx>
Attachment:
percent_increase_in_iops.jpg
Description: JPEG image
