David Rees wrote: > On Fri, Mar 13, 2009 at 3:29 PM, Trond Myklebust > <trond.myklebust@xxxxxxxxxx> wrote: > >> On Fri, 2009-03-13 at 14:59 -0700, David Rees wrote: >> >>> And the activity around the time I am reproducing the slowdown: >>> >>> Server nfs v3: >>> null getattr setattr lookup access readlink >>> 0 0% 3503 75% 7 0% 31 0% 1027 22% 0 0% >>> read write create mkdir symlink mknod >>> 9 0% 50 1% 6 0% 0 0% 0 0% 0 0% >>> remove rmdir rename link readdir readdirplus >>> 2 0% 0 0% 2 0% 0 0% 0 0% 0 0% >>> fsstat fsinfo pathconf commit >>> 0 0% 0 0% 0 0% 13 0% >>> >> Is this the result of only doing 2 'dd' copies? That's a lot of GETATTR >> calls for that kind of workload... >> > > No - the client that I have been duplicating this from is also my > desktop and the NFS server hosts my home directory and it was active > during the test. > > That's where the extreme slowdown in NFS performance affects me the > most. When then heavy IO on the server is going on (even just a > single process writing as fast as it can), my applications (Firefox, > Thunderbird, Gnome Terminals, just about anything that accesses the > NFS mount) will basically lock up and go totally unresponsive while > they wait for the NFS server to respond. They will sit unresponsive > for minutes at a time and are unusable until the heavy IO stops on the > server. > > I do software development from this machine and I have timed one of my > project builds with and without the heavy IO on the NFS server - a > build that normally takes about 20 seconds will take 5 minutes to > complete (it does read/write a lot of small files). > David, could you try your test case with this command running on the server? tethereal -i eth0 -f 'port 2049' -z rpc,rtt,100003,3 -w /dev/null and ^C it when you've done. You should see a table by RPC call with minimum maximum and average "roundtrip times" (actually as they're measured on the server, they should be server response times). This should tell you which calls are slow and whether it's all those calls or only a few outliers. Another experiment worth trying is a local filesystem load fairness test on the server (no NFS client involved). This will tell you whether the nfsd's IO priority is an issue, or if you're just seeing an IO scheduling issue. 1. echo 3 > /proc/sys/vm/drop_caches 2. time find /your/home/some/dir -ls > /dev/null (choose a directory tree with a few hundred files) 3. echo 3 > /proc/sys/vm/drop_caches 4. start your large writes 5. time find /your/home/some/dir -ls > /dev/null -- Greg Banks, P.Engineer, SGI Australian Software Group. the brightly coloured sporks of revolution. I don't speak for SGI. -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
