Yup, pretty common for us. Once we hit ~90% on either of our two production clusters (107 TB usable each), performance takes a beating. I don't consider this a problem, per se. Most file systems (clustered or otherwise) are the same. I consider a high water mark for any production file system to be 80% (and I consider that vendor agnostic), at which time action should be taken to begin clean up. That's good sysadminning 101. -Dan ---------------- Dan Mons Unbreaker of broken things Cutting Edge http://cuttingedge.com.au On 7 October 2014 08:36, Ellison, Bob <bob.ellison@xxxxxxxx> wrote: > Hello, > > My glusterfs-3.4.2-1.el6 is having a performance issue. It was working fine > until the 100TB file system hit ~90% full. I was seeing around 90Mb/s for > the last 10 months. This then dropped to 40Mb/s. Since nothing changed on > the system, I focused on the transition to the 90% full file system. I also > found that the 6 undelaying XFS files systems were pretty fragmented (~56%). > > > > We are using gluster to achieve a large flat file system. This is a single > server/node configuration, so no network issues are involved. > > > > As the problem is on a production system, I setup a smaller test system. I > monitored performance and was able to duplicate the problem (90MB/s up to > 90% full, then a drop off in performance thereafter). The closer to 100% > full, the lower the throughput. > > > > I then started deleting content from the test server. I was surprised to > find that the performance did NOT increase – it stayed the same. I took the > test system to file system 50% full but still saw 40Mb/s! > > > > The test I set up was designed to fragment the XFS partitions (to mimic the > production system state). I defragmented online successfully, however > performance did not increase. > > > > I am currently trying a rebalance across the 6 XFS partitions to see if that > helps. > > > > I was wondering if anyone remembers a problem like this? Is there a chance > that the rebalance will get me back to the normally seen performance? Would > upgrading gluster fix this? > > > > Thanks, > > Bob > > > _______________________________________________ > Gluster-users mailing list > Gluster-users@xxxxxxxxxxx > http://supercolony.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://supercolony.gluster.org/mailman/listinfo/gluster-users
