Thanks a lot for your insightful reply; it really clarifies a lot of things. I think the DHT-AFR-DHT configuration makes a lot of sense. - Wei Dong Harald St?rzebecher wrote: > Hi, > > 2009/7/28 Wei Dong <wdong.pku at gmail.com>: > >> Hi All, >> >> We've been using GlusterFS 2.0.1 on our lab cluster to host a large number >> of small images for distributed processing with Hadoop and it has been >> working fine without human intervention for a couple of months. Thanks for >> the wonderful project -- it's the only freely available cluster filesystem >> that fits our needs. >> >> What keeps bothering me is the extremely high flexibility of ClusterFS. >> There's simply so many ways to achieve the same goal that I don't know >> which is the best. So I'm writing to ask if there are some general >> guidelines of configuration to improve both data safety and performance. >> > > AFAIK, there are some general guidelines in the GlusterFS documentation. > IMHO, sometimes it takes careful reading or some experimentation to find them. > Some examples have been discussed on the mailing list. > > >> Specifically, we have 66 machines (in two racks) with 4 x 1.5TB disks / >> machine. We want to aggregate all the available disk space into a single >> shared directory with 3 replications.. Following are some of the potential >> configurations. >> >> * Each node exports 4 directories, so there are 66x4 = 264 directories to >> the client. We then first group those directories into threes with AFR, >> making 88 replicated directories, and then aggregate them with DHT. When >> configuring AFR, we can either make the three replicates on different >> machines, or two on the same machine and the third on another machine. >> > > I'd put the three replicates on three different machines - three > machines are less likely to fail than just two. > > One setup on my list of setups to evaluate would be a DHT - AFR - DHT > configuration. > - aggregate the four disks on each server to a single volume, export > only that volume > - on the clients, group those 66 volumes into threes with AFR and > aggregate with DHT > That would reduce the client config file from 264 imported volumes to > 66, reducing complexity of the configuration and the number of open > connections > > >> * Each node first aggregates three disks (forget about the 4th for >> simplicity) and exports a replicated directory. The client side then >> aggregates the 66 single replicated directory into one. >> > > That might mean that access to some of the data is lost if one node > fails - not what I'd accept from a replicated setup. > > >> * When grouping the aggregated directories on the client side, we can use >> some kind of hierarchy. For example the 66 directories are first aggregated >> into groups of N each with DHT, and then the 66/N groups are again >> aggregated with DHT. >> > > Doesn't that just make the setup more complicated? > > >> * We don't do the grouping on the client side. Rather, we use some >> intermediate server to first aggregate small groups of directories with DHT >> and export them as a single directory. >> > > The network connection of the intermediate server might become a > bottleneck, limiting performance. > The intermediate server might become a single point of failure. > > >> * We can also put AFR after DHT >> ...... >> >> To make things more complicated, the 66 machines are separated into two >> racks with only 4-gigabit inter-rack connection, so all the directories >> exported by the servers are not equal to a particular client. >> > > A workaround might be to create two intermediate volumes that each > perform better when accessed from on one of the racks and use NUFA to > create the single volume. > > Keeping replicated data local to one rack would improve performance, > but the failure of one complete rack (e.g. power line failure, > inter-rack networking) would block access to half of your data. > > Getting a third rack and much faster inter-rack connection would > improve performance and protect better against failures - just place > the three copies of a file on different racks. ;-) > > >> I'm wondering if someone on the mailing list could provide me with some >> advice. >> > > Plan, build, test ... repeat until satisfied :-) > > Optional: share your solution, with benchmarks > > > IMHO, there won't be a single "best" solution. > > > Harald St?rzebecher >
