On Tue, Jan 15, 2013 at 9:38 AM, Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx> wrote: > Hi everyone, > > quick question: can I get ceph to replicate a bunch of files to every > host in compute cluster and then have those hosts read those files from > local disk? > > TFM looks like a custom crush map should get the files to [osd on] every > host, but I'm not clear on the read step: do I need an mds on every host > and mount the fs off localhost's mds? > > (We've $APP running on the cluster, normally one instance/cpu core, that > mmap's (read only) ~30GB of binary files. I/O over NFS kills the cluster > even with a few hosts. Currently the files are rsync'ed to every host at > the start of the batch; that'll only scale to a few dozen hosts at best.) There's a "read from replicas" operation flag that allows reading data off the local node, although I don't think there's a way to turn it on in the standard filesystem clients right now. It wouldn't be hard for somebody to add. I'm not sure you actually need it though; Ceph unlike NFS distributes the data over all the OSDs in the cluster so you could scale the number of suppliers as you scale the number of consumers. You would definitely not need to run an MDS on each node; just one should be fine. Rather more importantly however, Inktank still doesn't consider the CephFS filesystem to be production-ready, so you will want to tread with care in checking it out. :) -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
