On Wed, 21 Jun 2017 19:44:08 -0500 Brady Deetz wrote: > Hello, > I'm expanding my 288 OSD, primarily cephfs, cluster by about 16%. I have 12 > osd nodes with 24 osds each. Each osd node has 2 P3700 400GB NVMe PCIe > drives providing 10GB journals for groups of 12 6TB spinning rust drives > and 2x lacp 40gbps ethernet. > > Our hardware provider is recommending that we start deploying P4600 drives > in place of our P3700s due to availability. > Welcome to the club and make sure to express your displeasure about Intel's "strategy" to your vendor. The P4600s are a poor replacement for P3700s and also still just "announced" according to ARK. Are you happy with your current NVMes? Firstly as in, what is their wearout, are you expecting them to easily survive 5 years at the current rate? Secondly, how about speed? with 12 HDDs and 1GB/s write capacity of the NVMe I'd expect them to not be a bottleneck in nearly all real life situations. Keep in mind that 1.6TB P4600 is going to last about as long as your 400GB P3700, so if wear-out is a concern, don't put more stress on them. Also the P4600 is only slightly faster in writes than the P3700, so that's where putting more workload onto them is going to be a notable issue. > I've seen some talk on here regarding this, but wanted to throw an idea > around. I was okay throwing away 280GB of fast capacity for the purpose of > providing reliable journals. But with as much free capacity as we'd have > with a 4600, maybe I could use that extra capacity as a cache tier for > writes on an rbd ec pool. If I wanted to go that route, I'd probably > replace several existing 3700s with 4600s to get additional cache capacity. > But, that sounds risky... > Risky as in high failure domain concentration and as mentioned above a cache-tier with obvious inline journals and thus twice the bandwidth needs will likely eat into the write speed capacity of the journals. If (and seems to be a big IF) you can find them, the Samsung PM1725a 1.6TB seems to be a) cheaper and b) at 2GB/s write speed more likely to be suitable for double duty. Similar (slightly better on paper) endurance than then P4600, so keep that in mind, too. Christian -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Rakuten Communications _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |