On Tue, Apr 05, 2011 at 09:32:38AM -0700, Andrew Klaassen wrote: > I've done some more benchmarking, and in my case writes appear to *not* be the culprit. > > Having the HPC farm only reading files (with noatime set everywhere, of course) actually makes "ls -l" over NFS slightly ~slower~. Do you know which part of 'ls -l' is taking longer? The readdir, or the stats? (Would an strace determine it?) Probably the latter, I guess. (Apologies if you already said.) > Having the HPC farm only reading files that all fit in the server's cache makes "ls -l" over NFS yet slower. (I watched iostat while this was running to make sure that nothing was being written to or read from disk.) > > So I've eliminated the disk as a bottleneck, and (as per my earlier emails) I've eliminated the filesystem and VM system. > > It really does look at this point like nfsd is the choke point. So I suppose the replies are likely waiting to be sent back over the tcp connection to the client, and waiting behind big read replies? As a check I wonder if there's any cheesy way we could prioritize threads waiting to send back getattr requests over threads waiting to send back reads. Such threads are waiting on the xpt_mutex. Is there some mutex or semaphore variant that gives us control over the order things are woken up in, or would we need our own? --b. -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
