On 01/14/2011 06:26 PM, Marcus Bointon wrote: >> Our own experience has been generally that you are IOP constrained >> because of the stack you have to traverse. If you add more latency >> into this stack, you have more to traverse, and therefore, you have >> more you need to wait. Which will have a magnification effect upon >> times for small IO ops which are seeky (stat, small writes, random >> ops). > > Sure, and all that applies equally to both NFS and gluster, yet in > Max's example NFS was ~50x faster than gluster for an identical > small-file workload. So what's gluster doing over and above what NFS > is doing that's taking so long, given that network and disk factors > are equal? I'd buy a factor of 2 for replication, but not 50. If the NFS was doing attribute caching and the GlusterFS implementation had stat prefetch and other caching turned off, this could explain it. > In case you missed what I'm on about, it was these stats that Max > posted: > >> Here is the results per command: dd if=/dev/zero of=M/tmp bs=1M >> count=16384 69.2 MB/se (Native) 69.2 MB/sec(FUSE) 52 MB/sec (NFS) >> dd if=/dev/zero of=M/tmp bs=1K count=163840000 88.1 MB/sec >> (Native) 1.1MB/sec (FUSE) 52.4 MB/sec (NFS) time tar cf - M | pv> >> /dev/null 15.8 MB/sec (native) 3.48MB/sec (FUSE) 254 Kb/sec (NFS) Ok, I am not sure if I saw the numbers before. Thanks. > > In my case I'm running 30kiops SSDs over gigabit. At the moment my > problem (running 3.0.6) isn't performance but reliability - files are > occasionally reported as 'vanished' by front-end apps (like rsync) > even though they are present on both backing stores; no errors in > gluster logs, self-heal doesn't help. Check your stat-prefetch settings, and your time base. We've had some strange issues that seem to be correlated with time bases drifting. Including files disappearing. We have a few open tickets on this. The way we've worked around this problem is to abandon the NFS client and use the glusterfs client. Not our preferred option, but it provides a workaround for the moment. The NFS translator does appear to have a few issues. I am hoping we get more tuning knobs for it soon so we can see if we can work around this. Regards, Joe > > Marcus _______________________________________________ Gluster-users > mailing list Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics, Inc. email: landman at scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615
