Hello, On Thu, 16 Jun 2016 15:31:13 +0000 Tim Gipson wrote: > A few questions. > > First, is there a good step by step to setting up a caching tier with > NVMe SSDs that are on separate hosts? Is that even possible? > Yes. And with a cluster of your size that's the way I'd do it. Larger cluster (dozen plus nodes) are likely to be better suited with storage nodes that have shared HDD OSDs for slow storage and SSD OSDs for cache pools. It would behoove you to scour this ML for the dozens of threads covering this and other aspects, like: "journal or cache tier on SSDs ?" "Steps for Adding Cache Tier" and even yesterdays: "Is Dynamic Cache tiering supported in Jewel" > Second, what sort of performance are people seeing from caching > tiers/journaling on SSDs in Jewel? > Not using Jewel, but it's bound to be better than Hammer. Performance will depend on a myriad of things, including CPU, SSD/NVMe models, networking, tuning, etc. It would be better if you had a performance target and a budget to see if they can be matched up. Cache tiering and journaling are very different things, don't mix them up. > Right now I am working on trying to find best practice for a CEPH > cluster with 3 monitor nodes, and 3 OSDs with 1 800GB NVMe drive and 12 > 6TB drives. > No need for dedicated monitor notes (definitely not 3 and with cluster of that size) if your storage nodes are designed correctly, see for example: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-April/008879.html > My goal is reliable/somewhat fast performance. > Well, for starters this cluster will give you the space of one of these nodes and worse performance than a single node due to the 3x replication. What NVMe did you have in mind, a DC P3600 will give you 1GB/s writes (and 3DWPD endurance), a P3700 2GB/s (and 10DWPD endurance). What about your network? Since the default failure domain in Ceph is the host, a single NVMe as journal for all HDD OSDs isn't particular risky, but it's something to keep in mind. Christian > Any help would be greatly appreciated! > > Tim Gipson > Systems Engineer > > [http://www.ena.com/signature/enaemaillogo.gif]<http://www.ena.com/> > > > 618 Grassmere Park Drive, Suite 12 > Nashville, TN 37211 > > > > website<http://www.ena.com/> | blog<http://www.ena.com/blog> | > support<http://support.ena.com/> > > > [http://www.ena.com/signature/facebook.png]<http://www.facebook.com/ENAConnects> > [http://www.ena.com/signature/twitter.png] > <http://twitter.com/#!/ENAConnects/> > [http://www.ena.com/signature/linkedin.png] > <http://www.linkedin.com/company/15330> > [http://www.ena.com/signature/youtube.png] > <https://www.youtube.com/user/EducationNetworks> > > > > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |