Hi Joe, We've been seeing the same thing here with several GlusterFS setups. It is really bad with GigE under load. From looking at our setups, it appears to be a network latency/contention issue. It is definitely not a disk contention issue as there is not a lot of load on the disks at the time of the metadata performance issues. Like you, we've tried a number of different combinations to help improve the performance. statprefetch will help with subsequent accesses to the same metadata, but it doesn't help with the initial access. We have not tried Infiniband yet to see if it improves the performance, but I am concerned that in a high IOps environment (like we run here), that even IB will not be able to sustain performance. - Mike Michael Moore Staff Engineer - Network SOLiD Advanced Research and Collaborations 500 Cummings Center Suite 2400 Beverly, MA 01915 T: 978-232-7886 Michael.Moore at lifetech.com Mike.Moore at appliedbiosystems.com -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Joe Landman Sent: Tuesday, April 27, 2010 7:19 PM To: gluster-users at gluster.org Subject: heavy metadata issues? Hi folks Seeing some metadata slowdowns in some cases on GlusterFS 3.0.3. These metadata slowdowns manifest in terms of long ls, long stat, long rm times for directories with a few thousand files. We tried statprefetch at the end of the stack, as well as right after the protocol/client, tried increasing read-ahead, etc. Is there anything in particular we should be looking to do, in order to get reasonable metadata performance under load? Thanks! Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: landman at scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/jackrabbit phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
