Oliver,
Here's the commit info:
Caspar
2018-02-27 14:28 GMT+01:00 Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx>:
Am 27.02.2018 um 14:16 schrieb Caspar Smit:
> Oliver,
>
> Be aware that for k=4,m=2 the min_size will be 5 (k+1), so after a node failure the min_size is already reached.
> Any OSD failure beyond the node failure will probably result in some PG's to be become incomplete (I/O Freeze) until the incomplete PG's data is recovered to another OSD in that node.
>
> So please reconsider your statement "one host + x safety" as the x safety (with I/O freeze) is probably not what you want.
>
> Forcing to run with min_size=4 could also be dangerous for other reasons. (there's a reason why min_size = k+1)
Thanks for pointing this out!
Yes, indeed, in case we need to take down a host for a longer period (we would hope this never has to happen for > 24 hours... but you never know),
and in case disks start to fail, we would indeed have to degrade to min_size=4 to keep running.
What exactly are the implications?
It should still be possible to ensure the data is not corrupt (with the checksums), and recovery to k+1 copies should start automatically once a disk fails -
so what's the actual implication?
Of course pg repair can not work in that case (if a PG for which the additional disk failed is corrupted),
but in general, when there's the need to reinstall a host, we'd try to bring it back with OSD data intact -
which should then allow to postpone the repair until that point.
Is there a danger I miss in my reasoning?
Cheers and many thanks!
Oliver
>
> Caspar
>
> 2018-02-27 0:17 GMT+01:00 Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx <mailto:freyermuth@xxxxxxxxxx-bonn.de >>:
>
> Am 27.02.2018 um 00:10 schrieb Gregory Farnum:
> > On Mon, Feb 26, 2018 at 2:59 PM Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx <mailto:freyermuth@xxxxxxxxxx-bonn.de > <mailto:freyermuth@xxxxxxxxxx-bonn.de <mailto:freyermuth@xxxxxxxxxx-bonn.de >>> wrote:
> >
> >
> > > Does this match expectations?
> > >
> > >
> > > Can you get the output of eg "ceph pg 2.7cd query"? Want to make sure the backfilling versus acting sets and things are correct.
> >
> > You'll find attached:
> > query_allwell) Output of "ceph pg 2.7cd query" when all OSDs are up and everything is healthy.
> > query_one_host_out) Output of "ceph pg 2.7cd query" when OSDs 164-195 (one host) are down and out.
> >
> >
> > Yep, that's what we want to see. So when everything's well, we have OSDs 91, 63, 33, 163, 192, 103. That corresponds to chassis 3, 2, 1, 5, 6, 4.
> >
> > When marking out a host, we have OSDs 91, 63, 33, 163, 123, UNMAPPED. That corresponds to chassis 3, 2, 1, 5, 4, UNMAPPED.
> >
> > So what's happened is that with the new map, when choosing the home for shard 4, we selected host 4 instead of host 6 (which is gone). And now shard 5 can't map properly. But of course we still have shard 5 available on host 4, so host 4 is going to end up properly owning shard 4, but also just carrying that shard 5 around as a remapped location.
> >
> > So this is as we expect. Whew.
> > -Greg
>
> Understood. Thanks for explaining step by step :-).
> It's of course a bit weird that this happens, since in the end, this really means data is moved (or rather, a shard is recreated) and taking up space without increasing redundancy
> (well, it might, if it lands on a different OSD than shard 5, but that's not really ensured). I'm unsure if this can be solved "better" in any way.
>
> Anyways, it seems this would be another reason why running with k+m=number of hosts should not be a general recommendation. For us, it's fine for now,
> especially since we want to keep the cluster open for later extension with more OSDs, and we do now know the gotchas - and I don't see a better EC configuration at the moment
> which would accomodate our wishes (one host + x safety, don't reduce space too much).
>
> So thanks again!
>
> Cheers,
> Oliver
>
>
> _______________________________________________ > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxx.
> ceph-users mailing list
com >
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. <http://lists.ceph.com/com listinfo.cgi/ceph-users-ceph. >com
>
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
