Re: Ceph PG Incomplete = Cluster unusable

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Dec 29, 2014, Christian Eichelmann <christian.eichelmann@xxxxxxxx> wrote:

> After we got everything up and running again, we still have 3 PGs in the
> state incomplete. I was checking one of them directly on the systems
> (replication factor is 3).

I have run into this myself at least twice before.  I had not lost or
replaced the OSDs altogether, though; I had just rolled too many of them
back to an earlier snapshots, which required them to be backfilled to
catch up.  It looks like an OSD won't get out of incomplete state, even
to backfill others, if this would keep the PG active size under the min
size for the pool.

In my case, I brought the current-ish snapshot of the OSD back up to
enable backfilling of enough replicas, so that I could then roll the
remaining OSDs back again and have them backfilled too.

However, I suspect that temporarily setting min size to a lower number
could be enough for the PGs to recover.  If "ceph osd pool <pool> set
min_size 1" doesn't get the PGs going, I suppose restarting at least one
of the OSDs involved in the recovery, so that they PG undergoes peering
again, would get you going again.

Once backfilling completes for all formerly-incomplete PGs, or maybe
even as soon as backfilling begins, bringing the pool min_size back up
to (presumably) 2 is advisable.  You don't want to be running too long
with a too-low min size :-)

I hope this helps,

Happy GNU Year,

-- 
Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
You must be the change you wish to see in the world. -- Gandhi
Be Free! -- http://FSFLA.org/   FSF Latin America board member
Free Software Evangelist|Red Hat Brasil GNU Toolchain Engineer
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux