On 06/30/2010 07:53 AM, Emmanuel Noobadmin wrote: > On Wed, Jun 30, 2010 at 7:25 PM, Jeff Darcy <jdarcy at redhat.com> wrote: > >> Another option, since you do have a fast interconnect, would be >> to place all of the permanent storage on the data nodes and use storage >> on the app nodes only for caching (as we had discussed). Replicate >> pair-wise or diagonally between data nodes, distribute across the >> replica sets, and you'd have a pretty good solution to handle future >> expansion. > > I think I'll probably go with this since you mention the replicate > over distribute doesn't work that well and I like to keep the app and > storage separate. But might change my mind if testing indicates the > performance level is not acceptable. > > As for fast interconnect, does that imply 10GbE/FC kind of speeds or > would normal GbE work? Hm, it appears I was confusing this thread with another one where the person had mentioned using DDR IB. By "fast interconnect" (having worked with interconnects up to 48Gb/s/node) I usually mean at least 10GbE and preferably some form of IB. Accessing all storage over a GbE network can work, but often requires more careful tuning and selection of equipment to get adequate performance. A lot depends on how much you can benefit from things like read-ahead and io-cache, or how much data you're willing to leave in write-behind buffers. It might well be the case that replicate over nufa/distribute will work better for your environment after all despite the issues with app-node "crosstalk" or the "inversion" of replicate vs. distribute. I think it's time to experiment with some of the options and see how they do.
