[Linux-cluster] GFS is *very* slow when NFS exported

[Stephen Willey <stephen.willey@xxxxxxxxxxxxxxxxxx>]

We are looking at using GFS for load balanced NFS.  Going with GNBD 
exported GFS isn't really an option since we're talking about over 1000 
machines needing to access the storage.  For this reason we were looking 
at providing a relatively small cluster of GFS machines serving the 
storage via NFS.  The clients would then use round-robin DNS to load 
balance across these servers.

The results we've got back from our tests are shown below:

-= Setup ========================================-

Two machines with GFS filesystem mounted from a
dual-port RAID, connected to ethernet via GigE and
serving NFS as follows:

/mnt/gfs *(no_root_squash,rw,insecure,async)

-= End of Setup =================================-


-= Local GFS Filesystem Access ==================-

The tests show Mb/s computed by using 10Gb dd

1 machine, write (gfstest1):
 166
1 machine, read (gfstest2 reading file gfstest1 created):
 139.5
2 machines, sim writes (different files):
 113.7 (gfstest1) 108 (gfstest2) - 221.7aggr
2 machines, sim reads (different files):
 101.5 (gfstest1) 101.6 (gfstest2) - 203aggr
2 machines, sim reads (same file):
 130 (gfstest1) 134 (gfstest2) - 264aggr

-= End of Local GFS Filesystem Access ===========-


-= NFS/GFS Access ===============================-

1 write: (client1 to gfstest1)
 42.3
1 read: (client1 from gfstest1)
 Varies enormously between 10-35Mb/s)

Simultaneous writes and reads done as follows:
 client1 NFS mounting gfstest1
 clients2 NFS mounting gfstest2

2 sim writes (different files):
 39.6 (client1) 42.4 (client2) - 82aggr
2 sim reads (different files):
 11.3 (client1) 11.2 (client2) - 22.5aggr

-= End of NFS/GFS Access ========================-


-= NFS/XFS Access ===============================-

Done for comparison of XFS & GFS export speeds

1 write: 65.2
1 read: 73.5

-= End of NFS/XFS Access ========================-


We know NFS isn't the highest performing thing in the world, but it's a 
concern that the NFS performance of a GFS mounted filesystem is so much 
lower than that of an XFS system.  There are of course, the clustering 
overheads, but this would affect local performance as well.  Anyone got 
any ideas as to why this might be and how to get more performance?

Thanks,

Stephen