At 11:02 PM 2/20/2009, Jordan Mendler wrote: >I am prototyping GlusterFS with ~50-60TB of raw disk space across >non-raided disks in ~30 compute nodes. I initially separated the >nodes into groups of two, and did a replicate across each set of >single drives in a pair of servers. Next I did a stripe across the >33 resulting AFR groups, with a block size of 1MB and later with the >default block size. With these configurations I am only seeing >throughput of about 15-25 MB/s, despite a full Gig-E network. > >What is generally the recommended configuration in a large striped >environment? I am wondering if the number of nodes in the stripe is >causing too much overhead, or if the bottleneck is likely somewhere >else. In addition, I saw a thread on the list that indicates it is >better to replicate across stripes rather than stripe across >replicates. Does anyone have any comments or opinion regarding this? I think that's all guesswork, I'm not sure anyones done a thorough test with gluster 2.0 on those choices. Personally, from a data management perspective, I'd rather replicate then stripe, so that I know that each node in a replica has exactly the same data. With striping then replicating, I imagine there is the possibility to have some data that's on one node in one stripe set on 2 nodes in another stripe set and this causes a problem if you have to take it apart or deal with it later. However, if you have the time, it'd be great to see results of you testing with a 15 node stripe and a 10 node stripe to see how those numbers rate vs. the 30 node stripe you have now. then, flip the replication and do the same tests again. Keith
