Could you share the 'ceph osd tree dump' and CRUSH map dump ? Thanks, Guang ---------------------------------------- > Date: Thu, 13 Aug 2015 08:16:09 -0700 > From: sdainard@xxxxxxxx > To: yangyongpeng@xxxxxxxxxxxxx; ceph-users@xxxxxxxxxxxxxx > Subject: Re: Cluster health_warn 1 active+undersized+degraded/1 active+remapped > > I decided to set OSD 76 out and let the cluster shuffle the data off > that disk and then brought the OSD back in. For the most part this > seemed to be working, but then I had 1 object degraded and 88xxx > objects misplaced: > > # ceph health detail > HEALTH_WARN 11 pgs stuck unclean; recovery 1/66089446 objects degraded > (0.000%); recovery 88844/66089446 objects misplaced (0.134%) > pg 2.e7f is stuck unclean for 88398.251351, current state > active+remapped, last acting [58,5] > pg 2.143 is stuck unclean for 13892.364101, current state > active+remapped, last acting [16,76] > pg 2.968 is stuck unclean for 13892.363521, current state > active+remapped, last acting [44,76] > pg 2.5f8 is stuck unclean for 13892.377245, current state > active+remapped, last acting [17,76] > pg 2.81c is stuck unclean for 13892.363443, current state > active+remapped, last acting [25,76] > pg 2.1a3 is stuck unclean for 13892.364400, current state > active+remapped, last acting [16,76] > pg 2.2cb is stuck unclean for 13892.374390, current state > active+remapped, last acting [14,76] > pg 2.d41 is stuck unclean for 13892.373636, current state > active+remapped, last acting [27,76] > pg 2.3f9 is stuck unclean for 13892.373147, current state > active+remapped, last acting [35,76] > pg 2.a62 is stuck unclean for 86283.741920, current state > active+remapped, last acting [2,38] > pg 2.1b0 is stuck unclean for 13892.363268, current state > active+remapped, last acting [3,76] > recovery 1/66089446 objects degraded (0.000%) > recovery 88844/66089446 objects misplaced (0.134%) > > I say apparently because with one object degraded, none of the pg's > are showing degraded: > # ceph pg dump_stuck degraded > ok > > # ceph pg dump_stuck unclean > ok > pg_stat state up up_primary acting acting_primary > 2.e7f active+remapped [58] 58 [58,5] 58 > 2.143 active+remapped [16] 16 [16,76] 16 > 2.968 active+remapped [44] 44 [44,76] 44 > 2.5f8 active+remapped [17] 17 [17,76] 17 > 2.81c active+remapped [25] 25 [25,76] 25 > 2.1a3 active+remapped [16] 16 [16,76] 16 > 2.2cb active+remapped [14] 14 [14,76] 14 > 2.d41 active+remapped [27] 27 [27,76] 27 > 2.3f9 active+remapped [35] 35 [35,76] 35 > 2.a62 active+remapped [2] 2 [2,38] 2 > 2.1b0 active+remapped [3] 3 [3,76] 3 > > All of the OSD filesystems are below 85% full. > > I then compared a 0.94.2 cluster that was new and had not been updated > (current cluster is 0.94.2 which had been updated a couple times) and > noticed the crush map had 'tunable straw_calc_version 1' so I added it > to the current cluster. > > After the data moved around for about 8 hours or so I'm left with this state: > > # ceph health detail > HEALTH_WARN 2 pgs stuck unclean; recovery 16357/66089446 objects > misplaced (0.025%) > pg 2.e7f is stuck unclean for 149422.331848, current state > active+remapped, last acting [58,5] > pg 2.782 is stuck unclean for 64878.002464, current state > active+remapped, last acting [76,31] > recovery 16357/66089446 objects misplaced (0.025%) > > I attempted a pg repair on both of the pg's listed above, but it > doesn't look like anything is happening. The doc's reference an > inconsistent state as a use case for the repair command so that's > likely why. > > These 2 pg's have been the issue throughout this process so how can I > dig deeper to figure out what the problem is? > > # ceph pg 2.e7f query: http://pastebin.com/jMMsbsjS > # ceph pg 2.e7f query: http://pastebin.com/0ntBfFK5 > > > On Wed, Aug 12, 2015 at 6:52 PM, yangyongpeng@xxxxxxxxxxxxx > <yangyongpeng@xxxxxxxxxxxxx> wrote: >> You can try "ceph pg repair pg_id"to repair the unhealth pg."ceph health >> detail" command is very useful to detect unhealth pgs. >> >> ________________________________ >> yangyongpeng@xxxxxxxxxxxxx >> >> >> From: Steve Dainard >> Date: 2015-08-12 23:48 >> To: ceph-users >> Subject: Cluster health_warn 1 active+undersized+degraded/1 >> active+remapped >> I ran a ceph osd reweight-by-utilization yesterday and partway through >> had a network interruption. After the network was restored the cluster >> continued to rebalance but this morning the cluster has stopped >> rebalance and status will not change from: >> >> # ceph status >> cluster af859ff1-c394-4c9a-95e2-0e0e4c87445c >> health HEALTH_WARN >> 1 pgs degraded >> 1 pgs stuck degraded >> 2 pgs stuck unclean >> 1 pgs stuck undersized >> 1 pgs undersized >> recovery 8163/66089054 objects degraded (0.012%) >> recovery 8194/66089054 objects misplaced (0.012%) >> monmap e24: 3 mons at >> {mon1=10.0.231.53:6789/0,mon2=10.0.231.54:6789/0,mon3=10.0.231.55:6789/0} >> election epoch 250, quorum 0,1,2 mon1,mon2,mon3 >> osdmap e184486: 100 osds: 100 up, 100 in; 1 remapped pgs >> pgmap v3010985: 4144 pgs, 7 pools, 125 TB data, 32270 kobjects >> 251 TB used, 111 TB / 363 TB avail >> 8163/66089054 objects degraded (0.012%) >> 8194/66089054 objects misplaced (0.012%) >> 4142 active+clean >> 1 active+undersized+degraded >> 1 active+remapped >> >> >> # ceph health detail >> HEALTH_WARN 1 pgs degraded; 1 pgs stuck degraded; 2 pgs stuck unclean; >> 1 pgs stuck undersized; 1 pgs undersized; recovery 8163/66089054 >> objects degraded (0.012%); recovery 8194/66089054 objects misplaced >> (0.012%) >> pg 2.e7f is stuck unclean for 65125.554509, current state >> active+remapped, last acting [58,5] >> pg 2.782 is stuck unclean for 65140.681540, current state >> active+undersized+degraded, last acting [76] >> pg 2.782 is stuck undersized for 60568.221461, current state >> active+undersized+degraded, last acting [76] >> pg 2.782 is stuck degraded for 60568.221549, current state >> active+undersized+degraded, last acting [76] >> pg 2.782 is active+undersized+degraded, acting [76] >> recovery 8163/66089054 objects degraded (0.012%) >> recovery 8194/66089054 objects misplaced (0.012%) >> >> # ceph pg 2.e7f query >> "recovery_state": [ >> { >> "name": "Started\/Primary\/Active", >> "enter_time": "2015-08-11 15:43:09.190269", >> "might_have_unfound": [], >> "recovery_progress": { >> "backfill_targets": [], >> "waiting_on_backfill": [], >> "last_backfill_started": "0\/\/0\/\/-1", >> "backfill_info": { >> "begin": "0\/\/0\/\/-1", >> "end": "0\/\/0\/\/-1", >> "objects": [] >> }, >> "peer_backfill_info": [], >> "backfills_in_flight": [], >> "recovering": [], >> "pg_backend": { >> "pull_from_peer": [], >> "pushing": [] >> } >> }, >> "scrub": { >> "scrubber.epoch_start": "0", >> "scrubber.active": 0, >> "scrubber.waiting_on": 0, >> "scrubber.waiting_on_whom": [] >> } >> }, >> { >> "name": "Started", >> "enter_time": "2015-08-11 15:43:04.955796" >> } >> ], >> >> >> # ceph pg 2.782 query >> "recovery_state": [ >> { >> "name": "Started\/Primary\/Active", >> "enter_time": "2015-08-11 15:42:42.178042", >> "might_have_unfound": [ >> { >> "osd": "5", >> "status": "not queried" >> } >> ], >> "recovery_progress": { >> "backfill_targets": [], >> "waiting_on_backfill": [], >> "last_backfill_started": "0\/\/0\/\/-1", >> "backfill_info": { >> "begin": "0\/\/0\/\/-1", >> "end": "0\/\/0\/\/-1", >> "objects": [] >> }, >> "peer_backfill_info": [], >> "backfills_in_flight": [], >> "recovering": [], >> "pg_backend": { >> "pull_from_peer": [], >> "pushing": [] >> } >> }, >> "scrub": { >> "scrubber.epoch_start": "0", >> "scrubber.active": 0, >> "scrubber.waiting_on": 0, >> "scrubber.waiting_on_whom": [] >> } >> }, >> { >> "name": "Started", >> "enter_time": "2015-08-11 15:42:41.139709" >> } >> ], >> "agent_state": {} >> >> I tried restarted osd.5/58/76 but no change. >> >> Any suggestions? >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |