Thanks for all of the responses, as Anthony said, we're just now building this out, so we look forward to doing benchmarking to find what's right for our environment, and of course share it to help adoption of GlusterFS. Also, we gave a talk in November at a conference in France covering our plans and reasons for moving to Gluster, you can see the slides for the talk here: http://www.slideshare.net/phil.cryer/building-a-scalable-open-source-storage-solution-2482448 All questions, comments are welcome! I'm working now on some detailed documentation for Gluster install using the latest Debian (Squeeze) testing branch to take advantage of of ext4. The storage platform looks very promising, I hope that you can break the web UI out of if so we can install it on our 'hand rolled' boxes. Thanks again for the support! P On Wed, Dec 30, 2009 at 12:07 PM, Tejas N. Bhise <tejas at gluster.com> wrote: > Thanks, Raghvendra. > > Anthony, > > Its a "lazy" self-heal mechanism, if you will. If one wants it all done right away, an ls -alR will access each file and hence cause the rebuild of the whole glusterfs volume which _may_, like you mentioned, be spread across disk partitions, LVM/RAID luns or even server nodes. > > Even after all that, only the files impacted in the volume would need to be rebuilt - although there might be some difference in overheads for different sized and configured Glusterfs volumes. It might be interesting to check - we have not done numbers on this. > > Let me check with the person who is more familiar with this area of code than me and he may be able to suggest ?some ballpark numbers till we run some real numbers. Meanwhile, if you do some tests, please share the numbers with the community. > > Regards, > Tejas. > > > ----- Original Message ----- > From: "Raghavendra G" <raghavendra at gluster.com> > To: "Anthony Goddard" <agoddard at mbl.edu> > Cc: "Tejas N. Bhise" <tejas at gluster.com>, "gluster-users" <gluster-users at gluster.org> > Sent: Wednesday, December 30, 2009 9:10:23 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi > Subject: Re: volume sizes > > Hi Anthony, > > > On Wed, Dec 30, 2009 at 6:30 PM, Anthony Goddard < agoddard at mbl.edu > wrote: > > > Hi Tejas, > Thanks for the advice. I will be using RAID as well as gluster replication I think.. as we'll only need to sacrifice 1 drive per raid set to add a bit of extra redundancy. > > The rebuild happens at the first access of a file, does this mean that the entire brick/node is rebuilt upon an initial file access? > > No, only the file which is accessed is rebuilt. That is the reason we recursively access all the files using 'ls -laR' on mount point. > > > > I think this is what I've seen from using gluster previously. If this is the case, it would rebuild the entire volume which could span many raid volumes or even machines, is this correct? If this is the case, then the underlying disk wouldn't have any effect at all, but if it's spanned over multiple machines and it only needs to rebuild one machine (or multiple volumes on one machine) it only needs to rebuild one volume. > I don't know if that made any sense.. haha.. but if it did, any insights into whether the size of the volumes (aside from RAID rebuilds) will have a positive effect on glusters rebuild operations? > > > Cheers, > Ant. > > > > > > On Dec 30, 2009, at 2:56 AM, Tejas N. Bhise wrote: > >> Anthony, >> >> Gluster can take the smaller ( 6TB ) volumes and aggregate them into a large Gluster volume ( as seen from the clients ). So that takes care of managebility on the client side of things. On the server side, once you make those smaller 6 TB volumes, you will depend on RAID to rebuild the disk behind it, so its good to have a smaller partition. Since you are using RAID and not Gluster replication, it might just make sense to have smaller RAID partitions. >> >> If instead you were using Gluster replication and resulting recovery, it would happen at first access of the file and the size of the Gluster volume or the backend native FS volume or the RAID ( or raw ) partition behind it would not be much of a consideration. >> >> Regards, >> Tejas. >> >> ----- Original Message ----- >> From: "Anthony Goddard" < agoddard at mbl.edu > >> To: gluster-users at gluster.org >> Sent: Wednesday, December 30, 2009 3:24:35 AM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi >> Subject: volume sizes >> >> First post! >> We're looking at setting up 6x 24 bay storage servers (36TB of JBOD storage per node) and running glusterFS over this cluster. >> We have RAID cards on these boxes and are trying to decide what the best size of each volume should be, for example if we present the OS's (and gluster) with six 36TB volumes, I imagine rebuilding one node would take a long time, and there may be other performance implications of this. On the other hand, if we present gluster / the OS's with 6x 6TB volumes on each node, we might have more trouble in managing a larger number of volumes. >> >> My gut tells me a lot of small (if you can call 6TB small) volumes will be lower risk and offer faster rebuilds from a failure, though I don't know what the pros and cons of these two approaches might be. >> >> Any advice would be much appreciated! >> >> >> Cheers, >> Anthony >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > > > > -- > Raghavendra G > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > -- http://philcryer.com
