On 02-05-17 23:53, David Turner wrote: > I was only interjecting on the comment "So that is 5 . Which is real > easy to obtain" and commenting on what the sustained writes into a > cluster of 2,000 OSDs would require to actually sustain that 5 MBps on > each SSD journal. Reading your calculation below I understand where the 2000 comes from. I meant that hardware of the previous millennium could easily write 5 Mbyte/sec sustained. :) This does NOT otherwise invalidate your interesting math below. And your conclusion is an important one. Go back to 200 OSDs and 25 SSDs and you end up with 40MB/s sustained writes to wear out your SSDs exactly at then end of the warranty. Higher sustained writes will linearly shorten your SSD lifetime. --WjW > My calculation was off because I forgot replica size, but my corrected > math is this... > > 5 MBps per journal device > 8 OSDs per journal (overestimated number as most do 4) > 2,000 OSDs based on what you said "Which is real easy to obtain, even > with hardware 0f 2000." > 3 replicas > > 2,000 OSDs / 8 OSDs per journal = 250 journal SSDs > 250 SSDs * 5 MBps = 1,250 MBps / 3 replicas = 416.67 MBps required > sustained cluster write speed to cause each SSD to average 5 MBps on > each journal device. > > Swap out any variable you want to match your environment. For example, > if you only have 4 OSDs per journal device, that number would be double > for a cluster this size to require a cluster write speed of > 833.33 MBps to average 5 MBps on each journal. Also if you have less > than 2,000 OSDs, then everything shrinks fast. > > > On Tue, May 2, 2017 at 5:39 PM Willem Jan Withagen <wjw@xxxxxxxxxxx > <mailto:wjw@xxxxxxxxxxx>> wrote: > > On 02-05-17 19:54, David Turner wrote: > > Are you guys talking about 5Mbytes/sec to each journal device? > Even if > > you had 8 OSDs per journal and had 2000 osds... you would need a > > sustained 1.25 Gbytes/sec to average 5Mbytes/sec per journal device. > > I'm not sure I'm following this... > But I'm rather curious. > Are you saying that the required journal bandwidth versus OSD write > bandwidth has an approx 1:200 ratio?? > > Note that I took it the other way. > Given the Intel specs > - What sustained bandwidth is allowed to have the device last its > lifetime. > - How much more usage would a 3710 give in regards to a 3520 SSD per > dollar spent. > > --WjW > > > On Tue, May 2, 2017 at 1:47 PM Willem Jan Withagen > <wjw@xxxxxxxxxxx <mailto:wjw@xxxxxxxxxxx> > > <mailto:wjw@xxxxxxxxxxx <mailto:wjw@xxxxxxxxxxx>>> wrote: > > > > On 02-05-17 19:16, Дробышевский, Владимир wrote: > > > Willem, > > > > > > please note that you use 1.6TB Intel S3520 endurance > rating in your > > > calculations but then compare prices with 480GB model, which > has only > > > 945TBW or 1.1DWPD ( > > > > > > https://ark.intel.com/products/93026/Intel-SSD-DC-S3520-Series-480GB-2_5in-SATA-6Gbs-3D1-MLC > > > ). It also worth to notice that S3710 has tremendously > higher write > > > speed\IOPS and especially SYNC writes. Haven't seen S3520 > real sync > > > write tests yet but don't think they differ much from S3510 > ones. > > > > Arrgh, you are right. I guess I had too many pages open, and > copied the > > wrong one. > > > > But the good news is that the stats were already in favour of > the 3710 > > so this only increases that conclusion. > > > > The bad news is that the sustained write speed goes down with a > > factor 4. > > So that is 5Mbyte/sec. Which is real easy to obtain, even with > hardware > > 0f 2000. > > > > --WjW > > > > > > > Best regards, > > > Vladimir > > > > > > 2017-05-02 21:05 GMT+05:00 Willem Jan Withagen > <wjw@xxxxxxxxxxx <mailto:wjw@xxxxxxxxxxx> > > <mailto:wjw@xxxxxxxxxxx <mailto:wjw@xxxxxxxxxxx>> > > > <mailto:wjw@xxxxxxxxxxx <mailto:wjw@xxxxxxxxxxx> > <mailto:wjw@xxxxxxxxxxx <mailto:wjw@xxxxxxxxxxx>>>>: > > > > > > On 27-4-2017 20:46, Alexandre DERUMIER wrote: > > > > Hi, > > > > > > > >>> What I'm trying to get from the list is /why/ the > > "enterprise" drives > > > >>> are important. Performance? Reliability? Something else? > > > > > > > > performance, for sure (for SYNC write, > > > https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ > > > > > > <https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/>) > > > > > > > > Reliabity : yes, enteprise drive have supercapacitor > in case > > of powerfailure, and endurance (1 DWPD for 3520, 3 DWPD for 3610) > > > > > > > > > > > >>> Also, 4 x Intel DC S3520 costs as much as 1 x Intel DC > > S3610. Obviously > > > >>> the single drive leaves more bays free for OSD > disks, but > > is there any > > > >>> other reason a single S3610 is preferable to 4 S3520s? > > Wouldn't 4xS3520s > > > >>> mean: > > > > > > > > where do you see this price difference ? > > > > > > > > for me , S3520 are around 25-30% cheaper than S3610 > > > > > > I just checked for the DCS3520 on > > > > > > https://ark.intel.com/nl/products/93005/Intel-SSD-DC-S3520-Series-1_6TB-2_5in-SATA-6Gbs-3D1-MLC > > > > > > <https://ark.intel.com/nl/products/93005/Intel-SSD-DC-S3520-Series-1_6TB-2_5in-SATA-6Gbs-3D1-MLC> > > > > > > And is has a TBW of 2925 (Terrabytes Write over life time) = > > 2,9 PB > > > the warranty is 5 years. > > > > > > Now if I do the math: > > > 2925 * 104 /5 /365 /24 /60 = 1,14 Gbyte/min to be written. > > > which is approx 20Mbyte /sec > > > or approx 10Gbit/min = 0,15 Gbit/sec > > > > > > And that is only 20% of the capacity of that SATA link. > > > Also writing 20Mbyte/sec sustained is not really that > hard for > > modern > > > systems. > > > > > > Now a 400Gb 3710 takes 8.3 PB, which is ruffly 3 times > as much. > > > so it will last 3 times longer. > > > > > > Checking Amazone, I get > > > $520 for the DC S3710-400G > > > $300 for the DC S3520-480G > > > > > > So that is less than a factor of 2 for using the S3710's > and a > > 3 times > > > longer lifetime. To be exact (8.3/520) / (2,9/300) = > 1.65 more > > bang for > > > your buck. > > > > > > But still do not expect your SSDs to last very long if the > > write rate is > > > much over that 20Mbyte/sec > > > > > > --WjW > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@xxxxxxxxxxxxxx > <mailto:ceph-users@xxxxxxxxxxxxxx> <mailto:ceph-users@xxxxxxxxxxxxxx > <mailto:ceph-users@xxxxxxxxxxxxxx>> > > <mailto:ceph-users@xxxxxxxxxxxxxx > <mailto:ceph-users@xxxxxxxxxxxxxx> <mailto:ceph-users@xxxxxxxxxxxxxx > <mailto:ceph-users@xxxxxxxxxxxxxx>>> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > > > > > > > > > > > > > > -- > > > > > > С уважением, > > > Дробышевский Владимир > > > Компания "АйТи Город" > > > +7 343 2222192 <tel:+7%20343%20222-21-92> > <tel:+7%20343%20222-21-92> > > > > > > ИТ-консалтинг > > > Поставка проектов "под ключ" > > > Аутсорсинг ИТ-услуг > > > Аутсорсинг ИТ-инфраструктуры > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> > <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |