On Wed, Oct 9, 2013 at 10:19 PM, Kees Bos <k.bos@xxxxxxxxxxx> wrote:
> Hi,
>
>
> I've managed to get cepth in a unhealthy state, from which it will not
> recover automatically. I've done some 'ceph osd out X' and stopped
> ceph-osd processes before the rebalancing was completed. (All in a test
> environment :-) )
>
> Now I see:
>
> # ceph -w
> cluster 7fac9ad3-455e-4570-ae24-5c4311763bf9
> health HEALTH_WARN 12 pgs degraded; 9 pgs stale; 9 pgs stuck stale; 964 pgs stuck unclean; recovery 617/50262 degraded (1.228%)
> monmap e4: 3 mons at {n2=192.168.5.12:6789/0,node01=192.168.5.10:6789/0,node03=192.168.5.11:6789/0}, election epoch 126, quorum 0,1,2 n2,node01,node03
> osdmap e1462: 17 osds: 17 up, 10 in
> pgmap v198793: 4416 pgs: 3452 active+clean, 2 stale+active, 943 active+remapped, 12 active+degraded, 7 stale+active+remapped; 95639 MB data, 192 GB used, 15628 GB / 15821 GB avail; 0B/s rd, 110KB/s wr, 9op/s; 617/50262 degraded (1.228%)
> mdsmap e1: 0/0/1 up
>
>
> 2013-10-10 07:02:57.741031 mon.0 [INF] pgmap v198792: 4416 pgs: 3452 active+clean, 2 stale+active, 943 active+remapped, 12 active+degraded, 7 stale+active+remapped; 95639 MB data, 192 GB used, 15628 GB / 15821 GB avail; 0B/s rd, 17492B/s wr, 2op/s; 617/50262 degraded (1.228%)
>
> I've seen some documentation at
> http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/
>
> * inactive - The placement group has not been active for too long
> (i.e., it hasn’t been able to service read/write requests).
> * unclean - The placement group has not been clean for too long
> (i.e., it hasn’t been able to completely recover from a previous
> failure).
> * stale - The placement group status has not been updated by a
> ceph-osd, indicating that all nodes storing this placement group
> may be down.
>
> Which leaves 'remapped' and 'degraded' unexplained (though I can imagine
> what they mean).
>
> I presume I've lost some data. Alas. How to get to a clean state again?
> I mean, if you're stuck with lost data, you don't want to have the
> cluster in a unhealthy state forever. I'd like to just cut my losses an
> get on.
Have you figured out what's going on with the stale PGs (and the OSDs
hosting them), following the instructions at that Troubleshooting
link? At some point it will probably become necessary to declare OSDs
lost, and then Ceph will give up on the data and move on. If you still
have any of those "down" OSDs, turning them back on should resolve
things, and any of them which you've actually decommissioned you need
to remove from the map. Your remapped PGs will probably get better if
you mark the up+out OSDs back in to the cluster.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
