Adam; Earlier this week, another thread presented 3 white papers in support of running 2x on NVMe for Ceph. I searched each to find the section where 2x was discussed. What I found was interesting. First, there are really only 2 positions here: Micron's and Red Hat's. Supermicro copies Micron's positon paragraph word for word. Not surprising considering that they are advertising a Supermicro / Micron solution. This is Micron's statement: " NVMe SSDs have high reliability with high MTBR and low bit error rate. 2x replication is recommended in production when deploying OSDs on NVMe versus the 3x replication common with legacy storage." This is Red Hat's statement: " Given the better MTBF and MTTR of flash-based media, many Ceph customers have chosen to run 2x replications in production when deploying OSDs on flash. This differs from magnetic media deployments, which typically use 3x replication." Looking at these statements, these acronyms pop out at me: MTBR and MTTR. MTBR is Mean Time Between Replacements, while MTTR is Mean Time Till Replacement. Essentially; this is saying that most companies replaces these drives before they have to worry about large numbers failing. Regarding MTBF; I can't find any data to support Red Hat's assertion that MTBF is better for flash. I looked at both Western Digital Gold, and Seagate Exos 12 TB drives, and found they both list a MTBF of 2.5 million hours. I was unable to find any information on the MTBF of Micron drives, but the MTBF of Kingston's DC1000B 240GB drive is 2 million hours. Personally, this looks like marketing BS to me. SSD shops want to sell SSDs, but because of the cost difference they have to convince buyers that their products are competitive. Pitch is thus: Our products cost twice as much, but LOOK you only need 2/3 as many, and you get all these other benefits (performance). Plus, if you replace everything in 2 or 3 years anyway, then you won't have to worry about them failing. I'll address general concerns of 2x replication in another email. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos@xxxxxxxxxxxxxx www.PerformAir.com -----Original Message----- From: Adam Boyhan [mailto:adamb@xxxxxxxxxx] Sent: Thursday, February 4, 2021 4:38 AM To: ceph-users Subject: NVMe and 2x Replica I know there is already a few threads about 2x replication but I wanted to start one dedicated to discussion on NVMe. There are some older threads, but nothing recent that addresses how the vendors are now pushing the idea of 2x. We are in the process of considering Ceph to replace our Nimble setup. We will have two completely separate clusters at two different sites that we are using rbd-mirror snapshot replication. The plan would be to run 2x replication on each cluster. 3x is still an option, but for obvious reasons 2x is enticing. Both clusters will be spot on to the super micro example in the white paper below. It seems all the big vendors feel 2x is safe with NVMe but I get the feeling this community feels otherwise. Trying to wrap my head around were the disconnect is between the big players and the community. I could be missing something, but even our Supermicro contact that we worked the config out with was in agreement with 2x on NVMe. Appreciate the input! [ https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf | https://www.supermicro.com/white_paper/white_paper_Ceph-Ultra.pdf ] [ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-reference-architecture-f17294-201904-en.pdf ] [ https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-reference-architecture-f17294-201904-en.pdf | https://www.redhat.com/cms/managed-files/st-micron-ceph-performance-reference-architecture-f17294-201904-en.pdf ] [ https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/redhat-ceph-whitepaper-0521.pdf | https://www.samsung.com/semiconductor/global.semi/file/resource/2020/05/redhat-ceph-whitepaper-0521.pdf ] _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |