Hey Bruce & Jim & Olga, On Fri, Jan 25, 2013 at 02:16:20PM -0500, Jim Rees wrote: > J. Bruce Fields wrote: > > On Thu, Jan 24, 2013 at 06:59:30PM -0600, Ben Myers wrote: > > At 1020 threads the send buffer size wraps and becomes negative causing > > the nfs server to grind to a halt. Rather than setting bufsize based > > upon the number of nfsd threads, make the buffer sizes tuneable via > > module parameters. > > > > Set the buffer sizes in terms of the number of rpcs you want to fit into > > the buffer. > > From private communication, my understanding is that the original > problem here was due to memory pressure forcing the tcp send buffer size > below the size required to hold a single rpc. Years ago I did see wrapping of the buffer size when tcp was used with many threads. Today's problem is timeouts on a cluster with a heavy read workload... and I seem to remember seeing that the send buffer size was too small. > In which case the important variable here is lock_bufsize, as that's > what prevents the buffer size from going too low. I tested removing the lock of bufsize and did hit the timeouts, so the overflow is starting to look less relevant. I will test your minimal overflow fix to see if this is the case. > Cc'ing Jim Rees in case he remembers: I seem to recall discussing this > possibility, wondering whether we needed a special interface to the > network layer allowing us to set a minimum, and deciding it wasn't > really necessary at the time as we didn't think the network layer would > actually do this. Is that right? In which case either we were wrong, > or something changed. > > I do remember discussing this. My memory is that we needed it but no one > wanted to implement it and it never happened. But I could be > mis-remembering. Maybe ask Olga, she's the one who put the bufsize > autotuning patch in, commit 96604398. > > I'll go back through my old mail and see if I can figure out what happened > with this. Thanks, Ben -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
