Christian, Thanks for all the info. I’ve been looking over the mailing lists. There is so much info there and from the looks of it, setting up a cache tier is much more complex than I had originally thought. Moving the journals to OSDs was much simpler for me because you can just use ceph-deploy and point the journal to the device you want. I do understand the difference between the cache tier and journaling. As per your comment about the monitor nodes, the extra monitor nodes are for the purpose of resiliency. We are trying to build our storage and compute clusters with lots of failure in mind. Our NVME drives are only the 800GB 3600 series. As to our networking setup: The OSD nodes have 4 x 10G nics, a bonded pair for front end traffic and a bonded pair for cluster traffic. The monitor nodes have a bonded pair of 1Gig nics. Our clients have 4 x 10G nics as well with a bonded pair dedicated to storage front end traffic connected to the ceph cluster. The single NVMe for journaling was a concern but as you mentioned before, a host is our failure domain at this point. I did find your comments to another user about having to add multiple roots per node because their NVMe drives were on different nodes. That is the case for our gear as well. Also, my gear is already in house so I’ve got what I’ve got to work with at this point, for good for ill. Tim Gipson On 6/16/16, 7:47 PM, "Christian Balzer" <chibi@xxxxxxx> wrote: Hello, On Thu, 16 Jun 2016 15:31:13 +0000 Tim Gipson wrote: > A few questions. > > First, is there a good step by step to setting up a caching tier with > NVMe SSDs that are on separate hosts? Is that even possible? > Yes. And with a cluster of your size that's the way I'd do it. Larger cluster (dozen plus nodes) are likely to be better suited with storage nodes that have shared HDD OSDs for slow storage and SSD OSDs for cache pools. It would behoove you to scour this ML for the dozens of threads covering this and other aspects, like: "journal or cache tier on SSDs ?" "Steps for Adding Cache Tier" and even yesterdays: "Is Dynamic Cache tiering supported in Jewel" > Second, what sort of performance are people seeing from caching > tiers/journaling on SSDs in Jewel? > Not using Jewel, but it's bound to be better than Hammer. Performance will depend on a myriad of things, including CPU, SSD/NVMe models, networking, tuning, etc. It would be better if you had a performance target and a budget to see if they can be matched up. Cache tiering and journaling are very different things, don't mix them up. > Right now I am working on trying to find best practice for a CEPH > cluster with 3 monitor nodes, and 3 OSDs with 1 800GB NVMe drive and 12 > 6TB drives. > No need for dedicated monitor notes (definitely not 3 and with cluster of that size) if your storage nodes are designed correctly, see for example: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-April/008879.html > My goal is reliable/somewhat fast performance. > Well, for starters this cluster will give you the space of one of these nodes and worse performance than a single node due to the 3x replication. What NVMe did you have in mind, a DC P3600 will give you 1GB/s writes (and 3DWPD endurance), a P3700 2GB/s (and 10DWPD endurance). What about your network? Since the default failure domain in Ceph is the host, a single NVMe as journal for all HDD OSDs isn't particular risky, but it's something to keep in mind. Christian > Any help would be greatly appreciated! > > Tim Gipson > Systems Engineer > > [http://www.ena.com/signature/enaemaillogo.gif]<http://www.ena.com/> > > > 618 Grassmere Park Drive, Suite 12 > Nashville, TN 37211 > > > > website<http://www.ena.com/> | blog<http://www.ena.com/blog> | > support<http://support.ena.com/> > > > [http://www.ena.com/signature/facebook.png]<http://www.facebook.com/ENAConnects> > [http://www.ena.com/signature/twitter.png] > <http://twitter.com/#!/ENAConnects/> > [http://www.ena.com/signature/linkedin.png] > <http://www.linkedin.com/company/15330> > [http://www.ena.com/signature/youtube.png] > <https://www.youtube.com/user/EducationNetworks> > > > > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |