I was only interjecting on the comment "So that is 5 . Which is real easy to obtain" and commenting on what the sustained writes into a cluster of 2,000 OSDs would require to actually sustain that 5 MBps on each SSD journal.
5 MBps per journal device
8 OSDs per journal (overestimated number as most do 4)
2,000 OSDs based on what you said "Which is real easy to obtain, even with hardware 0f 2000."
3 replicas
2,000 OSDs / 8 OSDs per journal = 250 journal SSDs
250 SSDs * 5 MBps = 1,250 MBps / 3 replicas = 416.67 MBps required sustained cluster write speed to cause each SSD to average 5 MBps on each journal device.
Swap out any variable you want to match your environment. For example, if you only have 4 OSDs per journal device, that number would be double for a cluster this size to require a cluster write speed of 833.33 MBps to average 5 MBps on each journal. Also if you have less than 2,000 OSDs, then everything shrinks fast.
On Tue, May 2, 2017 at 5:39 PM Willem Jan Withagen <wjw@xxxxxxxxxxx> wrote:
On 02-05-17 19:54, David Turner wrote:
> Are you guys talking about 5Mbytes/sec to each journal device? Even if
> you had 8 OSDs per journal and had 2000 osds... you would need a
> sustained 1.25 Gbytes/sec to average 5Mbytes/sec per journal device.
I'm not sure I'm following this...
But I'm rather curious.
Are you saying that the required journal bandwidth versus OSD write
bandwidth has an approx 1:200 ratio??
Note that I took it the other way.
Given the Intel specs
- What sustained bandwidth is allowed to have the device last its lifetime.
- How much more usage would a 3710 give in regards to a 3520 SSD per
dollar spent.
--WjW
> On Tue, May 2, 2017 at 1:47 PM Willem Jan Withagen <wjw@xxxxxxxxxxx
> <mailto:wjw@xxxxxxxxxxx>> wrote:
>
> On 02-05-17 19:16, Дробышевский, Владимир wrote:
> > Willem,
> >
> > please note that you use 1.6TB Intel S3520 endurance rating in your
> > calculations but then compare prices with 480GB model, which has only
> > 945TBW or 1.1DWPD (
> >
> https://ark.intel.com/products/93026/Intel-SSD-DC-S3520-Series-480GB-2_5in-SATA-6Gbs-3D1-MLC
> > ). It also worth to notice that S3710 has tremendously higher write
> > speed\IOPS and especially SYNC writes. Haven't seen S3520 real sync
> > write tests yet but don't think they differ much from S3510 ones.
>
> Arrgh, you are right. I guess I had too many pages open, and copied the
> wrong one.
>
> But the good news is that the stats were already in favour of the 3710
> so this only increases that conclusion.
>
> The bad news is that the sustained write speed goes down with a
> factor 4.
> So that is 5Mbyte/sec. Which is real easy to obtain, even with hardware
> 0f 2000.
>
> --WjW
>
>
> > Best regards,
> > Vladimir
> >
> > 2017-05-02 21:05 GMT+05:00 Willem Jan Withagen <wjw@xxxxxxxxxxx
> <mailto:wjw@xxxxxxxxxxx>
> > <mailto:wjw@xxxxxxxxxxx <mailto:wjw@xxxxxxxxxxx>>>:
> >
> > On 27-4-2017 20:46, Alexandre DERUMIER wrote:
> > > Hi,
> > >
> > >>> What I'm trying to get from the list is /why/ the
> "enterprise" drives
> > >>> are important. Performance? Reliability? Something else?
> > >
> > > performance, for sure (for SYNC write,
> https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
> >
> <https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/>)
> > >
> > > Reliabity : yes, enteprise drive have supercapacitor in case
> of powerfailure, and endurance (1 DWPD for 3520, 3 DWPD for 3610)
> > >
> > >
> > >>> Also, 4 x Intel DC S3520 costs as much as 1 x Intel DC
> S3610. Obviously
> > >>> the single drive leaves more bays free for OSD disks, but
> is there any
> > >>> other reason a single S3610 is preferable to 4 S3520s?
> Wouldn't 4xS3520s
> > >>> mean:
> > >
> > > where do you see this price difference ?
> > >
> > > for me , S3520 are around 25-30% cheaper than S3610
> >
> > I just checked for the DCS3520 on
> >
> https://ark.intel.com/nl/products/93005/Intel-SSD-DC-S3520-Series-1_6TB-2_5in-SATA-6Gbs-3D1-MLC
> >
> <https://ark.intel.com/nl/products/93005/Intel-SSD-DC-S3520-Series-1_6TB-2_5in-SATA-6Gbs-3D1-MLC>
> >
> > And is has a TBW of 2925 (Terrabytes Write over life time) =
> 2,9 PB
> > the warranty is 5 years.
> >
> > Now if I do the math:
> > 2925 * 104 /5 /365 /24 /60 = 1,14 Gbyte/min to be written.
> > which is approx 20Mbyte /sec
> > or approx 10Gbit/min = 0,15 Gbit/sec
> >
> > And that is only 20% of the capacity of that SATA link.
> > Also writing 20Mbyte/sec sustained is not really that hard for
> modern
> > systems.
> >
> > Now a 400Gb 3710 takes 8.3 PB, which is ruffly 3 times as much.
> > so it will last 3 times longer.
> >
> > Checking Amazone, I get
> > $520 for the DC S3710-400G
> > $300 for the DC S3520-480G
> >
> > So that is less than a factor of 2 for using the S3710's and a
> 3 times
> > longer lifetime. To be exact (8.3/520) / (2,9/300) = 1.65 more
> bang for
> > your buck.
> >
> > But still do not expect your SSDs to last very long if the
> write rate is
> > much over that 20Mbyte/sec
> >
> > --WjW
> >
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> >
> >
> >
> >
> > --
> >
> > С уважением,
> > Дробышевский Владимир
> > Компания "АйТи Город"
> > +7 343 2222192 <tel:+7%20343%20222-21-92>
> >
> > ИТ-консалтинг
> > Поставка проектов "под ключ"
> > Аутсорсинг ИТ-услуг
> > Аутсорсинг ИТ-инфраструктуры
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
