It took a while. It appears to have cleaned up quite a bit... but still has issues. I've been seeing below message for more than a day and cpu utilization and io utilization is low... looks like something is stuck... I rebooted OSDs several times when it looked like it was stuck earlier and it would work on something else, but now it is not changing much. What can I try now?
Regards,
Hong
# ceph health detail
HEALTH_ERR 22 pgs are stuck inactive for more than 300 seconds; 22 pgs degraded; 6 pgs down; 11 pgs inconsistent; 6 pgs peering; 6 pgs recovering; 16 pgs stale; 22 pgs stuck degraded; 6 pgs stuck inactive; 16 pgs stuck stale; 28 pgs stuck unclean; 16 pgs stuck undersized; 16 pgs undersized; 1 requests are blocked > 32 sec; 1 osds have slow requests; recovery 221990/4503980 objects degraded (4.929%); recovery 147/2251990 unfound (0.007%); 95 scrub errors; mds cluster is degraded; no legacy OSD present but 'sortbitwise' flag is not set
pg 0.e is stuck inactive since forever, current state down+peering, last acting [11,2]
pg 1.d is stuck inactive since forever, current state down+peering, last acting [11,2]
pg 1.28 is stuck inactive since forever, current state down+peering, last acting [11,6]
pg 0.29 is stuck inactive since forever, current state down+peering, last acting [11,6]
pg 1.2b is stuck inactive since forever, current state down+peering, last acting [1,11]
pg 0.2c is stuck inactive since forever, current state down+peering, last acting [1,11]
pg 0.e is stuck unclean since forever, current state down+peering, last acting [11,2]
pg 0.a is stuck unclean for 1233182.248198, current state stale+active+undersized+degraded+inconsistent, last acting [0]
pg 2.8 is stuck unclean for 1238044.714421, current state stale+active+undersized+degraded, last acting [0]
pg 2.1a is stuck unclean for 1238933.203920, current state active+recovering+degraded, last acting [2,11]
pg 2.3 is stuck unclean for 1238882.443876, current state stale+active+undersized+degraded, last acting [0]
pg 2.27 is stuck unclean for 1295260.765981, current state active+recovering+degraded, last acting [11,6]
pg 0.d is stuck unclean for 1230831.504001, current state stale+active+undersized+degraded, last acting [0]
pg 1.c is stuck unclean for 1238044.715698, current state stale+active+undersized+degraded, last acting [0]
pg 1.3d is stuck unclean for 1232066.572856, current state stale+active+undersized+degraded, last acting [0]
pg 1.28 is stuck unclean since forever, current state down+peering, last acting [11,6]
pg 0.29 is stuck unclean since forever, current state down+peering, last acting [11,6]
pg 1.2b is stuck unclean since forever, current state down+peering, last acting [1,11]
pg 2.2f is stuck unclean for 1238127.474088, current state active+recovering+degraded+remapped, last acting [9,10]
pg 0.0 is stuck unclean for 1233182.247776, current state stale+active+undersized+degraded, last acting [0]
pg 0.2c is stuck unclean since forever, current state down+peering, last acting [1,11]
pg 2.b is stuck unclean for 1238044.640982, current state stale+active+undersized+degraded, last acting [0]
pg 1.1b is stuck unclean for 1234021.660986, current state stale+active+undersized+degraded, last acting [0]
pg 0.1c is stuck unclean for 1232574.189549, current state stale+active+undersized+degraded, last acting [0]
pg 1.4 is stuck unclean for 1293624.075753, current state stale+active+undersized+degraded, last acting [0]
pg 0.5 is stuck unclean for 1237356.776788, current state stale+active+undersized+degraded+inconsistent, last acting [0]
pg 2.1f is stuck unclean for 8825246.729513, current state active+recovering+degraded, last acting [10,2]
pg 1.d is stuck unclean since forever, current state down+peering, last acting [11,2]
pg 2.39 is stuck unclean for 1238933.214406, current state stale+active+undersized+degraded, last acting [0]
pg 1.3a is stuck unclean for 2125299.164204, current state stale+active+undersized+degraded, last acting [0]
pg 0.3b is stuck unclean for 1233432.895409, current state stale+active+undersized+degraded, last acting [0]
pg 2.3c is stuck unclean for 1238933.208648, current state active+recovering+degraded, last acting [10,2]
pg 2.35 is stuck unclean for 1295260.753354, current state active+recovering+degraded, last acting [11,6]
pg 1.9 is stuck unclean for 1238044.722811, current state stale+active+undersized+degraded, last acting [0]
pg 0.a is stuck undersized for 1229917.081228, current state stale+active+undersized+degraded+inconsistent, last acting [0]
pg 2.8 is stuck undersized for 1229917.081016, current state stale+active+undersized+degraded, last acting [0]
pg 2.b is stuck undersized for 1229917.068181, current state stale+active+undersized+degraded, last acting [0]
pg 1.9 is stuck undersized for 1229917.075164, current state stale+active+undersized+degraded, last acting [0]
pg 0.5 is stuck undersized for 1229917.085330, current state stale+active+undersized+degraded+inconsistent, last acting [0]
pg 1.4 is stuck undersized for 1229917.085148, current state stale+active+undersized+degraded, last acting [0]
pg 0.d is stuck undersized for 1229917.080800, current state stale+active+undersized+degraded, last acting [0]
pg 1.c is stuck undersized for 1229917.080592, current state stale+active+undersized+degraded, last acting [0]
pg 1.3d is stuck undersized for 1229816.808393, current state stale+active+undersized+degraded, last acting [0]
pg 0.3b is stuck undersized for 1229917.074358, current state stale+active+undersized+degraded, last acting [0]
pg 1.3a is stuck undersized for 1229917.076592, current state stale+active+undersized+degraded, last acting [0]
pg 2.39 is stuck undersized for 1229917.077505, current state stale+active+undersized+degraded, last acting [0]
pg 0.1c is stuck undersized for 1229816.811773, current state stale+active+undersized+degraded, last acting [0]
pg 1.1b is stuck undersized for 1229816.812506, current state stale+active+undersized+degraded, last acting [0]
pg 2.3 is stuck undersized for 1229917.090143, current state stale+active+undersized+degraded, last acting [0]
pg 0.0 is stuck undersized for 1229917.073670, current state stale+active+undersized+degraded, last acting [0]
pg 0.a is stuck degraded for 1229917.081375, current state stale+active+undersized+degraded+inconsistent, last acting [0]
pg 2.8 is stuck degraded for 1229917.081162, current state stale+active+undersized+degraded, last acting [0]
pg 2.b is stuck degraded for 1229917.068328, current state stale+active+undersized+degraded, last acting [0]
pg 0.5 is stuck degraded for 1229917.085470, current state stale+active+undersized+degraded+inconsistent, last acting [0]
pg 1.4 is stuck degraded for 1229917.085288, current state stale+active+undersized+degraded, last acting [0]
pg 2.3c is stuck degraded for 2732.174512, current state active+recovering+degraded, last acting [10,2]
pg 0.d is stuck degraded for 1229917.080946, current state stale+active+undersized+degraded, last acting [0]
pg 1.c is stuck degraded for 1229917.080739, current state stale+active+undersized+degraded, last acting [0]
pg 1.3d is stuck degraded for 1229816.808539, current state stale+active+undersized+degraded, last acting [0]
pg 0.3b is stuck degraded for 1229917.074504, current state stale+active+undersized+degraded, last acting [0]
pg 1.3a is stuck degraded for 1229917.076739, current state stale+active+undersized+degraded, last acting [0]
pg 2.39 is stuck degraded for 1229917.077652, current state stale+active+undersized+degraded, last acting [0]
pg 2.1f is stuck degraded for 2732.122575, current state active+recovering+degraded, last acting [10,2]
pg 0.1c is stuck degraded for 1229816.811926, current state stale+active+undersized+degraded, last acting [0]
pg 1.1b is stuck degraded for 1229816.812659, current state stale+active+undersized+degraded, last acting [0]
pg 2.1a is stuck degraded for 2744.851402, current state active+recovering+degraded, last acting [2,11]
pg 2.3 is stuck degraded for 1229917.090302, current state stale+active+undersized+degraded, last acting [0]
pg 0.0 is stuck degraded for 1229917.073830, current state stale+active+undersized+degraded, last acting [0]
pg 2.27 is stuck degraded for 2744.828928, current state active+recovering+degraded, last acting [11,6]
pg 2.2f is stuck degraded for 2731.468651, current state active+recovering+degraded+remapped, last acting [9,10]
pg 1.9 is stuck degraded for 1229917.075428, current state stale+active+undersized+degraded, last acting [0]
pg 2.35 is stuck degraded for 2744.828894, current state active+recovering+degraded, last acting [11,6]
pg 0.a is stuck stale for 1227812.807624, current state stale+active+undersized+degraded+inconsistent, last acting [0]
pg 2.8 is stuck stale for 1227812.807638, current state stale+active+undersized+degraded, last acting [0]
pg 2.b is stuck stale for 1227812.807651, current state stale+active+undersized+degraded, last acting [0]
pg 1.9 is stuck stale for 1227812.807665, current state stale+active+undersized+degraded, last acting [0]
pg 0.5 is stuck stale for 1227812.807699, current state stale+active+undersized+degraded+inconsistent, last acting [0]
pg 1.4 is stuck stale for 1227812.807709, current state stale+active+undersized+degraded, last acting [0]
pg 0.d is stuck stale for 1227812.807624, current state stale+active+undersized+degraded, last acting [0]
pg 1.c is stuck stale for 1227812.807634, current state stale+active+undersized+degraded, last acting [0]
pg 1.3d is stuck stale for 1227812.807799, current state stale+active+undersized+degraded, last acting [0]
pg 0.3b is stuck stale for 1227812.807813, current state stale+active+undersized+degraded, last acting [0]
pg 1.3a is stuck stale for 1227812.807823, current state stale+active+undersized+degraded, last acting [0]
pg 2.39 is stuck stale for 1227812.807833, current state stale+active+undersized+degraded, last acting [0]
pg 0.1c is stuck stale for 1227812.807936, current state stale+active+undersized+degraded, last acting [0]
pg 1.1b is stuck stale for 1227812.807960, current state stale+active+undersized+degraded, last acting [0]
pg 2.3 is stuck stale for 1227812.808049, current state stale+active+undersized+degraded, last acting [0]
pg 0.0 is stuck stale for 1227812.808073, current state stale+active+undersized+degraded, last acting [0]
pg 0.38 is active+clean+inconsistent, acting [11,1]
pg 2.35 is active+recovering+degraded, acting [11,6], 29 unfound
pg 0.36 is active+clean+inconsistent, acting [10,1]
pg 2.2f is active+recovering+degraded+remapped, acting [9,10], 24 unfound
pg 0.2c is down+peering, acting [1,11]
pg 1.2b is down+peering, acting [1,11]
pg 0.29 is down+peering, acting [11,6]
pg 1.28 is down+peering, acting [11,6]
pg 0.26 is active+clean+inconsistent, acting [6,11]
pg 2.27 is active+recovering+degraded, acting [11,6], 19 unfound
pg 0.23 is active+clean+inconsistent, acting [6,10]
pg 2.1a is active+recovering+degraded, acting [2,11], 29 unfound
pg 0.18 is active+clean+inconsistent, acting [11,1]
pg 2.1f is active+recovering+degraded, acting [10,2], 20 unfound
pg 0.3c is active+clean+inconsistent, acting [1,11]
pg 0.3d is active+clean+inconsistent, acting [2,11]
pg 2.3c is active+recovering+degraded, acting [10,2], 26 unfound
pg 0.b is active+clean+inconsistent, acting [11,2]
pg 1.d is down+peering, acting [11,2]
pg 0.e is down+peering, acting [11,2]
pg 0.f is active+clean+inconsistent, acting [11,6]
1 ops are blocked > 4194.3 sec on osd.10
1 osds have slow requests
recovery 221990/4503980 objects degraded (4.929%)
recovery 147/2251990 unfound (0.007%)
95 scrub errors
mds cluster is degraded
mds.MDS1.1 at 192.168.1.20:6802/1862098088 rank 0 is replaying journal
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
