Re: SSD MTBF

[Martin B Nielsen <martin@xxxxxxxxxxx>]

A bit late getting back on this one.

On Wed, Oct 1, 2014 at 5:05 PM, Christian Balzer <chibi@xxxxxxx> wrote:

> smartctl states something like
> Wear = 092%, Hours = 12883, Datawritten = 15321.83 TB avg on those. I
> think that is ~30TB/day if I'm doing the calc right.
>
Something very much does not add up there.
Either you've written 15321.83 GB on those drives, making it about
30GB/day and well withing the Samsung specs, or you've written 10-20 times
the expected TBW level of those drives...

My bad, I forgot to say the Wear indicator here (92%) is sorta backwards - so it means it still has 92% to go before reaching expected TBW limit.

I agree with what Massimiliano Cuttini wrote later as well - if your io boundaries are well within the expected TBW of the lifetime I see no reason to go for more expensive disks. Just monitor for wear and have a few in stock ready for replacement.

Regarding the table of ssd and vendors:
Brand   Model         TBW   €      €/TB
Intel   S3500 120Go   70    122    1,74
Intel   S3500 240Go   140   225    1,60
Intel   S3700 100Go   1873  220    0,11
Intel   S3700 200Go   3737  400    0,10
Samsung 840 pro 120Go 70    120    1,71

I don't disagree with the above - but the table assumes you'll wear out your SSD. Adjust the wear level and the price will change proportionally - if you're only writing 50-100TB/year pr ssd then the value will heavily swing in the cheaper consumer grade ssd favor. It is all about your estimated usage pattern and whether they're 'good enough' for your scenario or not (and/or you trust that vendor).

In my experience ceph seldom (ever) maxes out io of a ssd - it is much more likely to be cpu or network before coming to that.

Cheers,
Martin

 

In the article I mentioned previously:
http://www.anandtech.com/show/8239/update-on-samsung-850-pro-endurance-vnand-die-size

The author clearly comes with a relationship of durability versus SSD
size, as one would expect. But the Samsung homepage just stated 150TBW,
for all those models...

Christian

> Not to advertise or say every samsung 840 ssd is like this:
> http://www.vojcik.net/samsung-ssd-840-endurance-destruct-test/
>
Seen it before, but I have a feeling that this test doesn't quite put the
same strain on the poor NANDs as Emmanuel's environment.

Christian

> Cheers,
> Martin
>
>
> On Wed, Oct 1, 2014 at 10:18 AM, Christian Balzer <chibi@xxxxxxx> wrote:
>
> > On Wed, 1 Oct 2014 09:28:12 +0200 Kasper Dieter wrote:
> >
> > > On Tue, Sep 30, 2014 at 04:38:41PM +0200, Mark Nelson wrote:
> > > > On 09/29/2014 03:58 AM, Dan Van Der Ster wrote:
> > > > > Hi Emmanuel,
> > > > > This is interesting, because we?ve had sales guys telling us that
> > > > > those Samsung drives are definitely the best for a Ceph journal
> > > > > O_o !
> > > >
> > > > Our sales guys or Samsung sales guys?  :)  If it was ours, let me
> > > > know.
> > > >
> > > > > The conventional wisdom has been to use the Intel DC S3700
> > > > > because of its massive durability.
> > > >
> > > > The S3700 is definitely one of the better drives on the market for
> > > > Ceph journals.  Some of the higher end PCIE SSDs have pretty high
> > > > durability (and performance) as well, but cost more (though you can
> > > > save SAS bay space, so it's a trade-off).
> > > Intel P3700 could be an alternative with 10 Drive-Writes/Day for 5
> > > years (see attachment)
> > >
> > They're certainly nice and competitively priced (TBW/$ wise at least).
> > However as I said in another thread, once your SSDs start to outlive
> > your planned server deployment time (in our case 5 years) that's
> > probably good enough.
> >
> > It's all about finding the balance between cost, speed (BW and IOPS),
> > durability and space.
> >
> > For example I'm currently building a cluster based on 2U, 12 hotswap
> > bays servers (because I already had 2 floating around) and am using 4
> > 100GB DC S3700 (at US$200 each) and 8 HDDS in them.
> > Putting in a 400GB DC P3700 (US$1200( instead and 4 more HDDs would
> > have pushed me over the budget and left me with a less than 30% "used"
> > SSD 5 years later, at a time when we clearly can expect these things
> > to be massively faster and cheaper.
> >
> > Now if you're actually having a cluster that would wear out a P3700 in
> > 5 years (or you're planning to run your machines until they burst into
> > flames), then that's another story. ^.^
> >
> > Christian
> >
> > > -Dieter
> > >
> > > >
> > > > >
> > > > > Anyway, I?m curious what do the SMART counters say on your SSDs??
> > > > > are they really failing due to worn out P/E cycles or is it
> > > > > something else?
> > > > >
> > > > > Cheers, Dan
> > > > >
> > > > >
> > > > >> On 29 Sep 2014, at 10:31, Emmanuel Lacour
> > > > >> <elacour@xxxxxxxxxxxxxxx> wrote:
> > > > >>
> > > > >>
> > > > >> Dear ceph users,
> > > > >>
> > > > >>
> > > > >> we are managing ceph clusters since 1 year now. Our setup is
> > > > >> typically made of Supermicro servers with OSD sata drives and
> > > > >> journal on SSD.
> > > > >>
> > > > >> Those SSD are all failing one after the other after one year :(
> > > > >>
> > > > >> We used Samsung 850 pro (120Go) with two setup (small nodes
> > > > >> with 2 ssd, 2 HD in 1U):
> > > > >>
> > > > >> 1) raid 1 :( (bad idea, each SSD support all the OSDs journals
> > > > >> writes :() 2) raid 1 for OS (nearly no writes) and dedicated
> > > > >> partition for journals (one per OSD)
> > > > >>
> > > > >>
> > > > >> I'm convinced that the second setup is better and we migrate old
> > > > >> setup to this one.
> > > > >>
> > > > >> Thought, statistics gives 60GB (option 2) to 100 GB (option 1)
> > > > >> writes per day on SSD on a not really over loaded cluster.
> > > > >> Samsung claims to give 5 years warranty if under 40GB/day.
> > > > >> Those numbers seems very low to me.
> > > > >>
> > > > >> What are your experiences on this? What write volumes do you
> > > > >> encounter, on wich SSD models, which setup and what MTBF?
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Easter-eggs                              Spécialiste GNU/Linux
> > > > >> 44-46 rue de l'Ouest  -  75014 Paris  -  France -  Métro Gaité
> > > > >> Phone: +33 (0) 1 43 35 00 37    -   Fax: +33 (0) 1 43 35 00 76
> > > > >> mailto:elacour@xxxxxxxxxxxxxxx  -   http://www.easter-eggs.com
> > > > >> _______________________________________________
> > > > >> ceph-users mailing list
> > > > >> ceph-users@xxxxxxxxxxxxxx
> > > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > >
> > > > > _______________________________________________
> > > > > ceph-users mailing list
> > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > >
> > > >
> > > > _______________________________________________
> > > > ceph-users mailing list
> > > > ceph-users@xxxxxxxxxxxxxx
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> > --
> > Christian Balzer        Network/Systems Engineer
> > chibi@xxxxxxx           Global OnLine Japan/Fusion Communications
> > http://www.gol.com/
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >


--
Christian Balzer        Network/Systems Engineer
chibi@xxxxxxx           Global OnLine Japan/Fusion Communications
http://www.gol.com/
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com