Re: Understanding incomplete PGs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Friday, July 5, 2019 11:50:44 AM CDT Paul Emmerich wrote:
> * There are virtually no use cases for ec pools with m=1, this is a bad
> configuration as you can't have both availability and durability

I'll have to look into this more. The cluster only has 4 hosts, so it might be 
worth switching to osd failure domain for the EC pools and using k=5,m=2.

> 
> * Due to weird internal restrictions ec pools below their min size can't
> recover, you'll probably have to reduce min_size temporarily to recover it

Lowering min_size to 2 did allow it to recover.

> 
> * Depending on your version it might be necessary to restart some of the
> OSDs due to a bug (fixed by now) that caused it to mark some objects as
> degraded if you remove or restart an OSD while you have remapped objects
> 
> * run "ceph osd safe-to-destroy X" to check if it's safe to destroy a given
> OSD

Excellent, thanks!

> 
> > Hello,
> > 
> > I'm working with a small ceph cluster (about 10TB, 7-9 OSDs, all Bluestore
> > on
> > lvm) and recently ran into a problem with 17 pgs marked as incomplete
> > after
> > adding/removing OSDs.
> > 
> > Here's the sequence of events:
> > 1. 7 osds in the cluster, health is OK, all pgs are active+clean
> > 2. 3 new osds on a new host are added, lots of backfilling in progress
> > 3. osd 6 needs to be removed, so we do "ceph osd crush reweight osd.6 0"
> > 4. after a few hours we see "min osd.6 with 0 pgs" from "ceph osd
> > utilization"
> > 5. ceph osd out 6
> > 6. systemctl stop ceph-osd@6
> > 7. the drive backing osd 6 is pulled and wiped
> > 8. backfilling has now finished all pgs are active+clean except for 17
> > incomplete pgs
> > 
> > From reading the docs, it sounds like there has been unrecoverable data
> > loss
> > in those 17 pgs. That raises some questions for me:
> > 
> > Was "ceph osd utilization" only showing a goal of 0 pgs allocated instead
> > of
> > the current actual allocation?
> > 
> > Why is there data loss from a single osd being removed? Shouldn't that be
> > recoverable?
> > All pools in the cluster are either replicated 3 or erasure-coded k=2,m=1
> > with
> > default "host" failure domain. They shouldn't suffer data loss with a
> > single
> > osd being removed even if there were no reweighting beforehand. Does the
> > backfilling temporarily reduce data durability in some way?
> > 
> > Is there a way to see which pgs actually have data on a given osd?
> > 
> > I attached an example of one of the incomplete pgs.
> > 
> > Thanks for any help,
> > 
> > Kyle_______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux