On Tue, Jan 24, 2012 at 09:11:01AM -0600, Greg_Swift at aotx.uscourts.gov wrote: > We have to have large numbers of volumes (~200). Quick run down to give > context. > > Our nodes would have around 128TB of local storage from several 32TB raid > sets. We started with ext4, so had a 16TB maximum. Aside: http://blog.ronnyegner-consulting.de/2011/08/18/ext4-and-the-16-tb-limit-now-solved/ > So we broke it down > into nice even chunks of 16TB, thus 8 file systems. Our first attempt was > ~200 volumes all using the 8 bricks per node (thus 1600 process/ports) ... > We had issues, and Gluster recommended > reducing our process/port count. So just checking I understand, the original configuration was: /data1/vol1 .. /data1/vol200 ... /data8/vol1 .. /data8/vol200 Terminology issue: isn't each serverN:/dirM considered a separate 'brick' to Gluster? I would have thought that configuration would count as 1600 bricks per node (but with groups of 200 bricks sharing 1 underlying filesystem) > First we dropped down to only using 1 brick per volume per node, but this > left us in a scenario of managing growth Like this? /data1/vol1 .. /data1/vol25 /data2/vol26 .. /data2/vol50 ... /data8/vol175 .. /data8/vol200 I see, so you have to assign the right subset of volumes to each filesystem. I guess you could shuffle them around using replace-brick, but it would be a pain. > So we determined to move to XFS to reduce from 8 partitions > down to 2 LVs. Each would be 64TB each /data1/vol1 .. /data1/vol200 /data2/vol1 .. /data2/vol200 i.e. 400 ports/processes/(bricks?) per server. > We then ran into some performance > issues and found we had not tuned the XFS enough, which also deterred us > from pushing forward with the move. I don't have any experience with XFS, but the Gluster docs do recommend it as the one most heavily tested. I saw an old note here about tuning XFS to include extended attributes in the inode: http://www.gluster.org/community/documentation/index.php/Guide_to_Optimizing_GlusterFS (although the values shown seem to be defaults to mkfs.xfs nowadays) Did you find any other tuning was required? This is all extremely helpful - many thanks for sharing your experiences. BTW I am just in the process of setting up two test systems here. Somewhat smaller than yours, but they are based on this chassis: http://www.xcase.co.uk/24-bay-Hotswap-rackmount-chassis-norco-RPC-4224-p/case-xcase-rm424.htm with Hitachi low-power 3TB drives. Regards, Brian.
