On Tue, May 19, 2015 at 7:35 PM, John Peebles <johnpeeb at gmail.com> wrote: > Hi, > > I'm hoping for advice on whether Ceph could be used in an atypical use case. > Specifically, I have about ~20TB of files that need replicated to 2 > different sites. Each site has its own internal gigabit ethernet network. > However, the connection between the sites is only ~320kbits. I'm trying to > find a solution where I set up one server at each site which has its own > full copy of the data, and when changes are made they are synced between the > sites. > > At first, this might seem hopeless because of the low bandwidth. However, > the long-term average rate of writes to the files is actually substantially > smaller than the available bandwidth, so this might not actually be a > problem. > > Off hand, does it seem like Ceph could yield decent performance in this use > case? In particular, I had a few questions: > > (1) Will clients at each site automatically prefer connecting to a > site-local Ceph node for reading files or will they try and pull files over > the slow site-to-site connection even when they are available site-locally? > If preferring a site-local node doesn't happen automatically, can it be > forced manually? > (2) When doing blocking IO to things backed by Ceph, will it block until the > data has been replicated? In other words, will my write speeds be > effectively be limited to 320kbits even if I am writing to a site-local > node? What kind of storage system are you looking for? In the raw RADOS sense, this is pretty unsuitable. There's a read-from-replica feature you can enable under specific circumstances, but that's basically only for snapshotted RBD "parent" images. And everything is replicated synchronously. But if you're looking at using RGW for the S3 object store, it has geo-replication features that *might* fit in here. Somebody else should discuss those. > (3) Is there anything else I should keep in mind when considering Ceph's > suitability for this sort of use? I think those first two pretty much rule it out for non-S3-like cases. > (4) If Ceph seems unsuitable for this task, is there a different piece of > software you would recommend? I haven't used it at all, but XtreemFS is the filesystem I know of that seems to fit best within this use case; and Swift is the open-source object store with "native" geo-replication that's designed for active-active sites. -Greg
 |