Am 27.02.2018 um 00:10 schrieb Gregory Farnum: > On Mon, Feb 26, 2018 at 2:59 PM Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx <mailto:freyermuth@xxxxxxxxxxxxxxxxxx>> wrote: > > > > Does this match expectations? > > > > > > Can you get the output of eg "ceph pg 2.7cd query"? Want to make sure the backfilling versus acting sets and things are correct. > > You'll find attached: > query_allwell) Output of "ceph pg 2.7cd query" when all OSDs are up and everything is healthy. > query_one_host_out) Output of "ceph pg 2.7cd query" when OSDs 164-195 (one host) are down and out. > > > Yep, that's what we want to see. So when everything's well, we have OSDs 91, 63, 33, 163, 192, 103. That corresponds to chassis 3, 2, 1, 5, 6, 4. > > When marking out a host, we have OSDs 91, 63, 33, 163, 123, UNMAPPED. That corresponds to chassis 3, 2, 1, 5, 4, UNMAPPED. > > So what's happened is that with the new map, when choosing the home for shard 4, we selected host 4 instead of host 6 (which is gone). And now shard 5 can't map properly. But of course we still have shard 5 available on host 4, so host 4 is going to end up properly owning shard 4, but also just carrying that shard 5 around as a remapped location. > > So this is as we expect. Whew. > -Greg Understood. Thanks for explaining step by step :-). It's of course a bit weird that this happens, since in the end, this really means data is moved (or rather, a shard is recreated) and taking up space without increasing redundancy (well, it might, if it lands on a different OSD than shard 5, but that's not really ensured). I'm unsure if this can be solved "better" in any way. Anyways, it seems this would be another reason why running with k+m=number of hosts should not be a general recommendation. For us, it's fine for now, especially since we want to keep the cluster open for later extension with more OSDs, and we do now know the gotchas - and I don't see a better EC configuration at the moment which would accomodate our wishes (one host + x safety, don't reduce space too much). So thanks again! Cheers, Oliver
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
