CRUSH is a probabilistic algorithm. By having all those non-existent OSDs in the map, you made it so that 10/12 attempts at mapping would fail and need to be retried. CRUSH handles a lot of retries, but not enough for that to work out well. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Mon, Jul 7, 2014 at 4:09 AM, hua peng <huapeng at arcor.de> wrote: > Hi, > > I have resolved it by run these: > root at ceph2:~# ceph osd crush rm osd.0 > removed item id 0 name 'osd.0' from crush map > root at ceph2:~# ceph osd crush rm osd.1 > removed item id 1 name 'osd.1' from crush map > root at ceph2:~# ceph osd crush rm osd.2 > removed item id 2 name 'osd.2' from crush map > root at ceph2:~# ceph osd crush rm osd.3 > removed item id 3 name 'osd.3' from crush map > root at ceph2:~# ceph osd crush rm osd.4 > removed item id 4 name 'osd.4' from crush map > root at ceph2:~# ceph osd crush rm osd.5 > removed item id 5 name 'osd.5' from crush map > root at ceph2:~# ceph osd crush rm osd.6 > removed item id 6 name 'osd.6' from crush map > root at ceph2:~# ceph osd crush rm osd.7 > removed item id 7 name 'osd.7' from crush map > root at ceph2:~# ceph osd crush rm osd.8 > removed item id 8 name 'osd.8' from crush map > root at ceph2:~# ceph osd crush rm osd.9 > removed item id 9 name 'osd.9' from crush map > > Though I am not sure why it resolved my problem... > Now it's oK: > > root at ceph2:~# ceph health > HEALTH_OK > > Thanks. > > >> Hi, >> >> I have 135 pgs degraded in the system. How can I remove them? >> They are in test environment, all data are not important. >> >> Thanks for the kind helps. >> >> root at ceph2:~# ceph osd tree >> >> # id weight type name up/down reweight >> -1 0.8398 root default >> -2 0.8398 host ceph2 >> 0 0.01999 osd.0 DNE >> 1 0.01999 osd.1 DNE >> 2 0.07999 osd.2 DNE >> 3 0.07999 osd.3 DNE >> 4 0.07999 osd.4 DNE >> 5 0.07999 osd.5 DNE >> 6 0.07999 osd.6 DNE >> 7 0.07999 osd.7 DNE >> 8 0.07999 osd.8 DNE >> 9 0.07999 osd.9 DNE >> 10 0.07999 osd.10 up 1 >> 11 0.07999 osd.11 up 1 >> >> root at ceph2:~# ceph -s >> health HEALTH_WARN 135 pgs degraded; 135 pgs stuck unclean >> monmap e1: 1 mons at {ceph2=172.17.6.176:6789/0}, election epoch 1, >> quorum 0 ceph2 >> osdmap e95: 2 osds: 2 up, 2 in >> pgmap v2351: 192 pgs: 57 active+clean, 135 active+degraded; 0 bytes >> data, 33243 MB used, 114 GB / 154 GB avail >> mdsmap e1: 0/0/1 up >> >> root at ceph2:~# >> root at ceph2:~# ceph health >> HEALTH_WARN 135 pgs degraded; 135 pgs stuck unclean >> root at ceph2:~# >> root at ceph2:~# ceph health detail >> HEALTH_WARN 135 pgs degraded; 135 pgs stuck unclean >> pg 2.3d is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.3e is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.3f is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.3f is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.3f is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.3e is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.39 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.3a is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.3b is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.38 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.3a is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.38 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.39 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.39 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.34 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.36 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.37 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.34 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.35 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.35 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.34 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.31 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.32 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.33 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.30 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.33 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.32 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.33 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.30 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.31 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.32 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.31 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.2f is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.2c is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.2d is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.29 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.2a is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.2b is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.2b is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.28 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.25 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.26 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.27 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.24 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.27 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.26 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.24 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.25 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.26 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.25 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.24 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.21 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.22 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.23 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.20 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.23 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.22 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.23 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.20 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.21 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.22 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.21 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.20 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.1d is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.1e is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.1f is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.1c is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.1f is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.1e is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.1f is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.1c is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.1d is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.1e is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.1d is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.1c is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.18 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.1b is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.1a is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.1b is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.18 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.19 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.1a is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.19 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.15 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.16 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.17 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.14 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.16 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.17 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.14 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.15 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.15 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.14 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.11 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.12 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.13 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.10 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.13 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.12 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.13 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.12 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.11 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.d is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.e is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.f is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.c is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.d is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.c is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.9 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.a is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.b is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.b is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.b is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.8 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.9 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.a is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.8 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 2.5 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.6 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.7 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.4 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.7 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 0.6 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.7 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.4 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.5 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.6 is stuck unclean since forever, current state active+degraded, >> last acting [10] >> pg 1.5 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.0 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.2 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 2.3 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.0 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.1 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 1.1 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.0 is stuck unclean since forever, current state active+degraded, >> last acting [11] >> pg 0.3f is active+degraded, acting [11] >> pg 1.3e is active+degraded, acting [11] >> pg 2.3d is active+degraded, acting [11] >> pg 1.3f is active+degraded, acting [11] >> pg 2.3f is active+degraded, acting [10] >> pg 2.3e is active+degraded, acting [11] >> pg 0.3b is active+degraded, acting [10] >> pg 1.3a is active+degraded, acting [10] >> pg 2.39 is active+degraded, acting [10] >> pg 0.3a is active+degraded, acting [11] >> pg 2.38 is active+degraded, acting [11] >> pg 0.39 is active+degraded, acting [11] >> pg 1.38 is active+degraded, acting [11] >> pg 1.39 is active+degraded, acting [11] >> pg 0.36 is active+degraded, acting [10] >> pg 2.34 is active+degraded, acting [10] >> pg 0.35 is active+degraded, acting [11] >> pg 1.34 is active+degraded, acting [11] >> pg 2.37 is active+degraded, acting [11] >> pg 0.34 is active+degraded, acting [10] >> pg 1.35 is active+degraded, acting [10] >> pg 0.33 is active+degraded, acting [11] >> pg 1.32 is active+degraded, acting [11] >> pg 2.31 is active+degraded, acting [11] >> pg 0.32 is active+degraded, acting [11] >> pg 1.33 is active+degraded, acting [10] >> pg 2.30 is active+degraded, acting [11] >> pg 0.31 is active+degraded, acting [10] >> pg 1.30 is active+degraded, acting [10] >> pg 2.33 is active+degraded, acting [11] >> pg 1.31 is active+degraded, acting [11] >> pg 2.32 is active+degraded, acting [10] >> pg 0.2d is active+degraded, acting [10] >> pg 1.2c is active+degraded, acting [10] >> pg 2.2f is active+degraded, acting [10] >> pg 0.2b is active+degraded, acting [10] >> pg 1.2a is active+degraded, acting [10] >> pg 2.29 is active+degraded, acting [10] >> pg 2.2b is active+degraded, acting [10] >> pg 0.28 is active+degraded, acting [10] >> pg 0.27 is active+degraded, acting [10] >> pg 1.26 is active+degraded, acting [10] >> pg 2.25 is active+degraded, acting [10] >> pg 0.26 is active+degraded, acting [11] >> pg 1.27 is active+degraded, acting [10] >> pg 2.24 is active+degraded, acting [11] >> pg 0.25 is active+degraded, acting [11] >> pg 1.24 is active+degraded, acting [11] >> pg 0.24 is active+degraded, acting [10] >> pg 1.25 is active+degraded, acting [11] >> pg 2.26 is active+degraded, acting [10] >> pg 0.23 is active+degraded, acting [11] >> pg 1.22 is active+degraded, acting [11] >> pg 2.21 is active+degraded, acting [11] >> pg 0.22 is active+degraded, acting [11] >> pg 1.23 is active+degraded, acting [10] >> pg 2.20 is active+degraded, acting [11] >> pg 0.21 is active+degraded, acting [10] >> pg 1.20 is active+degraded, acting [10] >> pg 2.23 is active+degraded, acting [11] >> pg 0.20 is active+degraded, acting [11] >> pg 1.21 is active+degraded, acting [11] >> pg 2.22 is active+degraded, acting [10] >> pg 0.1f is active+degraded, acting [11] >> pg 1.1e is active+degraded, acting [11] >> pg 2.1d is active+degraded, acting [11] >> pg 0.1e is active+degraded, acting [11] >> pg 1.1f is active+degraded, acting [11] >> pg 2.1c is active+degraded, acting [11] >> pg 0.1d is active+degraded, acting [10] >> pg 1.1c is active+degraded, acting [10] >> pg 2.1f is active+degraded, acting [10] >> pg 0.1c is active+degraded, acting [10] >> pg 1.1d is active+degraded, acting [11] >> pg 2.1e is active+degraded, acting [11] >> pg 0.1a is active+degraded, acting [11] >> pg 1.1b is active+degraded, acting [10] >> pg 2.18 is active+degraded, acting [11] >> pg 0.19 is active+degraded, acting [10] >> pg 1.18 is active+degraded, acting [10] >> pg 2.1b is active+degraded, acting [10] >> pg 1.19 is active+degraded, acting [11] >> pg 2.1a is active+degraded, acting [10] >> pg 0.17 is active+degraded, acting [10] >> pg 1.16 is active+degraded, acting [10] >> pg 2.15 is active+degraded, acting [10] >> pg 0.16 is active+degraded, acting [11] >> pg 2.14 is active+degraded, acting [11] >> pg 0.15 is active+degraded, acting [11] >> pg 1.14 is active+degraded, acting [11] >> pg 2.17 is active+degraded, acting [10] >> pg 0.14 is active+degraded, acting [10] >> pg 1.15 is active+degraded, acting [11] >> pg 0.13 is active+degraded, acting [10] >> pg 1.12 is active+degraded, acting [10] >> pg 2.11 is active+degraded, acting [10] >> pg 0.12 is active+degraded, acting [11] >> pg 1.13 is active+degraded, acting [10] >> pg 2.10 is active+degraded, acting [11] >> pg 2.13 is active+degraded, acting [11] >> pg 1.11 is active+degraded, acting [11] >> pg 2.12 is active+degraded, acting [10] >> pg 0.f is active+degraded, acting [11] >> pg 1.e is active+degraded, acting [11] >> pg 2.d is active+degraded, acting [11] >> pg 0.d is active+degraded, acting [11] >> pg 1.c is active+degraded, acting [11] >> pg 0.c is active+degraded, acting [11] >> pg 0.b is active+degraded, acting [10] >> pg 1.a is active+degraded, acting [10] >> pg 2.9 is active+degraded, acting [10] >> pg 1.b is active+degraded, acting [11] >> pg 0.9 is active+degraded, acting [11] >> pg 1.8 is active+degraded, acting [11] >> pg 2.b is active+degraded, acting [11] >> pg 0.8 is active+degraded, acting [10] >> pg 2.a is active+degraded, acting [11] >> pg 0.7 is active+degraded, acting [11] >> pg 1.6 is active+degraded, acting [11] >> pg 2.5 is active+degraded, acting [11] >> pg 0.6 is active+degraded, acting [11] >> pg 1.7 is active+degraded, acting [10] >> pg 2.4 is active+degraded, acting [11] >> pg 0.5 is active+degraded, acting [11] >> pg 1.4 is active+degraded, acting [11] >> pg 2.7 is active+degraded, acting [11] >> pg 1.5 is active+degraded, acting [11] >> pg 2.6 is active+degraded, acting [10] >> pg 0.2 is active+degraded, acting [11] >> pg 2.0 is active+degraded, acting [11] >> pg 0.1 is active+degraded, acting [11] >> pg 1.0 is active+degraded, acting [11] >> pg 2.3 is active+degraded, acting [11] >> pg 0.0 is active+degraded, acting [11] >> pg 1.1 is active+degraded, acting [11] >> _______________________________________________ >> ceph-users mailing list >> ceph-users at lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |