We recently purchased a new cluster and I have been advocating the use of glusterfs. I have a lot of experience on the user side with nfs and my feeling is that it has poor performance. I suggested gluster which got relayed to our vendor and this is what they relayed back to me: - Suggests we stay away from glusterfs and gfs - glusterfs is not mature - We need a stable parallel file system. nfs is stable, glusterfs is not. - Using gluster would be adding complexity and removing reliability. It's important to note that I am advocating glusterfs in the presence of issues with our nfs file system practically every time I submit a job. The cluster admin characterizes the problem thusly: NFS cannot handle getting hit with many simultaneous requests. Most processes wait for io and this drives up the load. The result is that we get locked out of the cluster when the queue builds up. Here are the stats of our cluster: - 26 nodes + head node - Infiniband - Dual quad core xeons @ 2.53 Ghz - 24 GB memory per node - 20 TB hardware RAID 6 available over NFS - 26 nodes, each of which has a 207GB hard drive. These are presently all at 13GB utilization, meaning there is 5 terabytes of free space across the nodes that is essentially unavailable for use right now. My question to you is, how would you respond to these points made by our vendor, and given this setup how would you configure GlusterFS to 1. Replace NFS 2. Maximize write speed and throughput 3. Maximize read speed and throughput 4. Maximize reliability I'm looking for the stable middle of the ground solution that gets rid of our performance problems and takes advantage of what appears to be kickass hardware for a file system, all while making the case for gluster. I have a positive opinion of this project, I don't know why others do not.. Many thanks, Brian Mingus Professional Research Assistant Computational Cognitive Neuroscience Lab University of Colorado at Boulder
