On Wed, Jun 30, 2010 at 7:25 PM, Jeff Darcy <jdarcy at redhat.com> wrote: > Another option, since you do have a fast interconnect, would be > to place all of the permanent storage on the data nodes and use storage > on the app nodes only for caching (as we had discussed). Replicate > pair-wise or diagonally between data nodes, distribute across the > replica sets, and you'd have a pretty good solution to handle future > expansion. I think I'll probably go with this since you mention the replicate over distribute doesn't work that well and I like to keep the app and storage separate. But might change my mind if testing indicates the performance level is not acceptable. As for fast interconnect, does that imply 10GbE/FC kind of speeds or would normal GbE work? On 6/30/10, Jeff Darcy <jdarcy at redhat.com> wrote: > On 06/29/2010 11:31 PM, Emmanuel Noobadmin wrote: >> With the nufa volumes, a file is only written to one of the volumes >> listed in its definition. >> If the volume is a replicate volume, then the file is replicated on >> each of the volumes listed in its definition. >> >> e.g in this case >> volume my_nufa >> type cluster/nufa >> option local-volume-name rep1 >> subvolumes rep0 rep1 rep2 >> end-volume >> >> A file is only found in one of rep0 rep1 or rep2. If it was on rep2, >> then it would be inaccessible if rep2 fails such as network failure >> cutting rep2 off. > > Yes, but rep2 as a whole could only fail if all of its component volumes > - one on an app node and one on a data node - failed simultaneously. > That's about as good protection as you're going to get without > increasing your replication level (therefore decreasing both performance > and effective storage utilization). > >> Then when I add a rep3, gluster should automatically start putting new >> files onto it. >> >> At this point though, it seems that if I use nufa, I would have an >> issue if I add a purely storage only rep3 instead of an app+storage >> node. None of the servers will use it until their local volume reaches >> max capacity right? :D >> >> So if I preferred to have the load spread out more evenly, I should >> then be using cluster/distribute? > > If you want even distribution across different or variable numbers of > app/data nodes, then cluster/distribute would be the way to go. For > example, you could create a distribute set across the storage nodes and > a nufa set across the app nodes, and then replicate between the two > (each app node preferring the local member of the nufa set). You'd lose > the ability to suppress app-node-to-app-node communication with > different read-subvolume assignments, though, and in my experience > replicate over distribute doesn't work quite as well as the other way > around. Another option, since you do have a fast interconnect, would be > to place all of the permanent storage on the data nodes and use storage > on the app nodes only for caching (as we had discussed). Replicate > pair-wise or diagonally between data nodes, distribute across the > replica sets, and you'd have a pretty good solution to handle future > expansion. > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >
