Hello, Another thing that crossed my mind aside from failure probabilities caused by actual HDDs dying is of course the little detail that most Ceph installations will have have WAL/DB (journal) on SSDs, the most typical ratio being 1:4. And given the current thread about compaction killing pure HDD OSDs, something you may _have_ to do. So if you get unlucky and a SSD dies 4 OSDs are irrecoverably lost, unlike a dead node that can be recovered. Combine that with the background noise of HDDs failing, things got just quite a bit scarier. And if you have a "crap firmware of the week" situation like experienced with several people here, you're even more like to wind up in trouble very fast. This is of course all something people do (or should know), I'm more wondering how to model it to correctly asses risks. Christian On Wed, 3 Apr 2019 10:28:09 +0900 Christian Balzer wrote: > On Tue, 2 Apr 2019 19:04:28 +0900 Hector Martin wrote: > > > On 02/04/2019 18.27, Christian Balzer wrote: > > > I did a quick peek at my test cluster (20 OSDs, 5 hosts) and a replica 2 > > > pool with 1024 PGs. > > > > (20 choose 2) is 190, so you're never going to have more than that many > > unique sets of OSDs. > > > And this is why one shouldn't send mails when in a rush, w/o fully groking > the math one was just given. > Thanks for setting me straight. > > > I just looked at the OSD distribution for a replica 3 pool across 48 > > OSDs with 4096 PGs that I have and the result is reasonable. There are > > 3782 unique OSD tuples, out of (48 choose 3) = 17296 options. Since this > > is a random process, due to the birthday paradox, some duplicates are > > expected after only the order of 17296^0.5 = ~131 PGs; at 4096 PGs > > having 3782 unique choices seems to pass the gut feeling test. Too lazy > > to do the math closed form, but here's a quick simulation: > > > > >>> len(set(random.randrange(17296) for i in range(4096))) > > 3671 > > > > So I'm actually slightly ahead. > > > > At the numbers in my previous example (1500 OSDs, 50k pool PGs), > > statistically you should get something like ~3 collisions on average, so > > negligible. > > > Sounds promising. > > > > Another thing to look at here is of course critical period and disk > > > failure probabilities, these guys explain the logic behind their > > > calculator, would be delighted if you could have a peek and comment. > > > > > > https://www.memset.com/support/resources/raid-calculator/ > > > > I'll take a look tonight :) > > > Thanks, a look at the Backblaze disk failure rates (picking the worst > ones) gives a good insight into real life probabilities, too. > https://www.backblaze.com/blog/hard-drive-stats-for-2018/ > If we go with 2%/year, that's an average failure ever 12 days. > > Aside from how likely the actual failure rate is, another concern of > course is extended periods of the cluster being unhealthy, with certain > versions there was that "mon map will grow indefinitely" issue, other more > subtle ones might lurk still. > > Christian > > -- > > Hector Martin (hector@xxxxxxxxxxxxxx) > > Public Key: https://mrcn.st/pub > > > > > -- > Christian Balzer Network/Systems Engineer > chibi@xxxxxxx Rakuten Communications > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Rakuten Communications _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |