Perfomance issue on a 90+% full file system

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

My glusterfs-3.4.2-1.el6 is having a performance issue. It was working fine until the 100TB file system hit ~90% full. I was seeing around 90Mb/s for the last 10 months. This then dropped to 40Mb/s. Since nothing changed on the system, I focused on the transition to the 90% full file system.  I also found that the 6 undelaying XFS files systems were pretty fragmented (~56%).

 

We are using gluster to achieve a large flat file system. This is a single server/node configuration, so no network issues are involved.

 

As the problem is on a production system, I setup a smaller test system. I monitored performance and was able to duplicate the problem (90MB/s up to 90% full, then a drop off in performance thereafter). The closer to 100% full, the lower the throughput.

 

I then started deleting content from the test server. I was surprised to find that the performance did NOT increase – it stayed the same. I took the test system to file system 50% full but still saw 40Mb/s!

 

The test I set up was designed to fragment the XFS partitions (to mimic the production system state). I defragmented online successfully, however performance did not increase.

 

I am currently trying a rebalance across the 6 XFS partitions to see if that helps.

 

I was wondering if anyone remembers a problem like this? Is there a chance that the rebalance will get me back to the normally seen performance? Would upgrading gluster fix this?

 

Thanks,

Bob

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux