This is a widely perceived feature/bug of gluster. It also affects other distributed filesystems, tho generally not as much. We've done 2 things to address this. One is a distributed 'du' that is clusterfork'ed out to the storage nodes and compiles the results. This is realtime and will provide data to that point. If you're interested in it, let me know and I can provide the code to do this. However, it requires clusterfork, some per-site config, and is specific to 'du', altho it could be modified to support other shell commands. Here's the difference in performance on a fairly busy gluster system (4 storage nodes, 8 volumes, 340TB, 60% used) ===================== 14:54:09 root at hpc-s:/som 1226 $ time du -sh abusch/* 694M abusch/MATS-gtf ^C real 3m58.098s <--- killed after ~4m user 0m0.033s sys 0m0.351s 14:58:24 root at hpc-s:/som 1227 $ gfdu abusch/\* INFO: Corrected gluster starting path: [/som/abusch/*] About to execute [/root/bin/cf --script --tar=GLSRV "du -s /raid1/som/abusch/* ; du -s /raid2/som/abusch/*; "] Go? [yN]y INFO: For raw results [cd /root/cf/CF-du--s--raid1-som- abu-14.58.38_2013-10-08] Size: File|Dir 693.8203 M /som/abusch/MATS-gtf 1.5292 G /som/abusch/MISO-gffs 764.5117 M /som/abusch/MISO-gffs-v2 23.8720 G /som/abusch/deepSeq 25.2845 G /som/abusch/genomes 5.4239 G /som/abusch/index 16.8011 G /som/abusch/index2 ----------------------------------- 74.3348 G Total time was ~4s ===================== The other approach is with the RobinHood Policy Engine <http://sourceforge.net/apps/trac/robinhood> which runs on a cron and recurses thru your FS, taking X hours, but compiles that info into a MySQL DB that is instantly responsive (but could be slightly out of date). NTL, it's a very helpful tool to detect hotspots and ZOTfiles (Zillions Of Tiny files) We are using it to monitor NFS volumes, Gluster, and Fraunhofer FSs. It is a very slick system and a student (Adam Brenner) is modifying it to generate better stats via the web interface. See his github and the robinhood trac: https://github.com/abrenner/robinhood-multifs-web http://sourceforge.net/apps/trac/robinhood On Tuesday, October 08, 2013 09:07:52 AM Anders Salling Andersen wrote: > Hi all i have a 50tb glusterfs replicated setup, with Many small files. My > metadata is very slow ex. Du -sh takes over 24 hours. Is there a Way to > make faster metadata ? > > Regards Anders. > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users --- Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps) --- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131008/fea421e6/attachment.html>
