Hello,
2 PG's were stuck in "inactive state". I'm trying to find out why this
happened. Here's what I found in logs (only what I think is important):
2022-10-27T22:39:29.707362+0200 mgr.skarb (mgr.40364478) 152930 :
cluster [DBG] pgmap v154892: 410 pgs: 1 active+remapped+backfilling, 2
active+clean+scrubbing+deep, 1 active+clean+snaptrim, 24
active+remapped+backfill_wait, 382 active+clean; 3.5 TiB data, 11 TiB
used, 19 TiB / 30 TiB avail; 11 KiB/s rd, 1.1 MiB/s wr, 168 op/s;
142367/2859735 objects misplaced (4.978%); 12 MiB/s, 3 objects/s recovering
2022-10-27T22:39:30.266560+0200 mon.skarb (mon.0) 4717148 : cluster
[DBG] osdmap e18492: 9 total, 9 up, 9 in
[...]
2022-10-27T22:39:31.317055+0200 osd.3 (osd.3) 106245 : cluster [DBG]
2.51 starting backfill to osd.7 from (0'0,0'0] MAX to 18489'135867589
2022-10-27T22:39:31.524742+0200 osd.3 (osd.3) 106246 : cluster [INF]
2.31 continuing backfill to osd.2 from (18485'135835568,18490'135837516]
MIN to 18490'135837516
2022-10-27T22:39:31.713775+0200 mgr.skarb (mgr.40364478) 152931 :
cluster [DBG] pgmap v154895: 409 pgs: 3 peering, 1
active+remapped+backfilling, 2 active+clean+scrubbing+deep, 24
active+remapped+backfil
l_wait, 379 active+clean; 3.5 TiB data, 11 TiB used, 19 TiB / 30 TiB
avail; 5.8 KiB/s rd, 874 KiB/s wr, 108 op/s; 142367/2859732 objects
misplaced (4.978%); 10 MiB/s, 3 objects/s recovering
[...]
2022-10-27T22:39:33.333564+0200 mon.skarb (mon.0) 4717153 : cluster
[DBG] osdmap e18495: 9 total, 9 up, 9 in
2022-10-27T22:39:33.337963+0200 osd.2 (osd.2) 62236 : cluster [DBG] 2.77
starting backfill to osd.1 from (0'0,0'0] MAX to 18489'149579766
2022-10-27T22:39:33.338240+0200 osd.2 (osd.2) 62237 : cluster [DBG] 2.37
starting backfill to osd.1 from (0'0,0'0] MAX to 18489'149493224
2022-10-27T22:39:33.718993+0200 mgr.skarb (mgr.40364478) 152932 :
cluster [DBG] pgmap v154898: 409 pgs: 13 activating+remapped, 3
remapped+peering, 3 peering, 2 active+remapped+backfilling, 2 active+clea
n+scrubbing+deep, 23 active+remapped+backfill_wait, 363 active+clean;
3.5 TiB data, 11 TiB used, 19 TiB / 30 TiB avail; 804 KiB/s wr, 98 op/s;
208558/2859732 objects misplaced (7.293%); 35 MiB/s, 8 objec
ts/s recovering
[...]
2022-10-27T22:39:51.784172+0200 mgr.skarb (mgr.40364478) 152941 :
cluster [DBG] pgmap v154913: 408 pgs: 2
remapped+premerge+backfill_wait+peered, 3 active+remapped+backfilling, 2
active+clean+scrubbing+d
eep, 87 active+remapped+backfill_wait, 314 active+clean; 3.5 TiB data,
11 TiB used, 19 TiB / 30 TiB avail; 451185/2858388 objects misplaced
(15.785%)
[...]
2022-10-27T22:40:00.000177+0200 mon.skarb (mon.0) 4717169 : cluster
[INF] overall HEALTH_OK
2022-10-27T22:40:01.821923+0200 mgr.skarb (mgr.40364478) 152946 :
cluster [DBG] pgmap v154918: 408 pgs: 2
remapped+premerge+backfill_wait+peered, 3 active+remapped+backfilling, 2
active+clean+scrubbing+d
eep, 87 active+remapped+backfill_wait, 314 active+clean; 3.5 TiB data,
11 TiB used, 19 TiB / 30 TiB avail; 117 MiB/s rd, 150 MiB/s wr, 5.92k
op/s; 450997/2859732 objects misplaced (15.771%); 264 MiB/s, 7
4 objects/s recovering
[...]
2022-10-27T22:40:32.115085+0200 mon.skarb (mon.0) 4717180 : cluster
[WRN] Health check failed: Reduced data availability: 2 pgs inactive
(PG_AVAILABILITY)
2022-10-27T22:40:34.038597+0200 mgr.skarb (mgr.40364478) 152962 :
cluster [DBG] pgmap v154934: 408 pgs: 2
remapped+premerge+backfill_wait+peered, 3 active+remapped+backfilling, 2
active+clean+scrubbing+deep, 87 active+remapped+backfill_wait, 314
active+clean; 3.5 TiB data, 11 TiB used, 19 TiB / 30 TiB avail; 6.0
KiB/s rd, 3.2 MiB/s wr, 209 op/s; 450255/2859732 objects misplaced
(15.745%); 44 MiB/s, 14 objects/s recovering
[...]
2022-10-27T22:49:59.062366+0200 mgr.skarb (mgr.40364478) 153243 :
cluster [DBG] pgmap v155219: 408 pgs: 1 active+clean+scrubbing+deep, 2
remapped+premerge+backfill_wait+peered, 4 active+remapped+backfill
ing, 84 active+remapped+backfill_wait, 317 active+clean; 3.5 TiB data,
11 TiB used, 19 TiB / 30 TiB avail; 4.6 KiB/s rd, 1.6 MiB/s wr, 108
op/s; 438424/2859736 objects misplaced (15.331%); 57 MiB/s, 16 o
bjects/s recovering
2022-10-27T22:50:00.000198+0200 mon.skarb (mon.0) 4717320 : cluster
[WRN] Health detail: HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-27T22:50:00.000245+0200 mon.skarb (mon.0) 4717321 : cluster
[WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive
2022-10-27T22:50:00.000263+0200 mon.skarb (mon.0) 4717322 : cluster
[WRN] pg 2.37 is stuck inactive for 10m, current state
remapped+premerge+backfill_wait+peered, last acting [2,6,3]
2022-10-27T22:50:00.000307+0200 mon.skarb (mon.0) 4717323 : cluster
[WRN] pg 2.77 is stuck inactive for 10m, current state
remapped+premerge+backfill_wait+peered, last acting [2,6,3]
2022-10-27T22:50:01.069589+0200 mgr.skarb (mgr.40364478) 153244 :
cluster [DBG] pgmap v155220: 408 pgs: 1 active+clean+scrubbing+deep, 2
remapped+premerge+backfill_wait+peered, 4 active+remapped+backfill
ing, 84 active+remapped+backfill_wait, 317 active+clean; 3.5 TiB data,
11 TiB used, 19 TiB / 30 TiB avail; 4.6 KiB/s rd, 1.3 MiB/s wr, 94 op/s;
438413/2859736 objects misplaced (15.331%); 50 MiB/s, 14 ob
jects/s recovering
It recovered a few hours later:
2022-10-28T03:43:22.268113+0200 mgr.skarb (mgr.40364478) 162018 :
cluster [DBG] pgmap v164234: 408 pgs: 1
remapped+premerge+backfilling+peered, 1 clean+premerge+peered, 1
active+clean+snaptrim, 42 active+remapped+backfill_wait, 363
active+clean; 3.5 TiB data, 11 TiB used, 19 TiB / 30 TiB avail; 1.3
KiB/s rd, 105 KiB/s wr, 22 op/s; 233157/2841886 objects misplaced
(8.204%); 28 MiB/s, 7 objects/s recovering
2022-10-28T03:43:24.269793+0200 mgr.skarb (mgr.40364478) 162019 :
cluster [DBG] pgmap v164235: 408 pgs: 1
remapped+premerge+backfilling+peered, 1 clean+premerge+peered, 1
active+clean+snaptrim, 42 active+remapped+backfill_wait, 363
active+clean; 3.5 TiB data, 11 TiB used, 19 TiB / 30 TiB avail; 1.3
KiB/s rd, 91 KiB/s wr, 19 op/s; 233157/2841886 objects misplaced
(8.204%); 19 MiB/s, 5 objects/s recovering
2022-10-28T03:43:26.271601+0200 mgr.skarb (mgr.40364478) 162020 :
cluster [DBG] pgmap v164236: 408 pgs: 1
remapped+premerge+backfilling+peered, 1 clean+premerge+peered, 1
active+clean+snaptrim, 42 active+remapped+backfill_wait, 363
active+clean; 3.5 TiB data, 11 TiB used, 19 TiB / 30 TiB avail; 1.3
KiB/s rd, 77 KiB/s wr, 17 op/s; 233157/2841877 objects misplaced
(8.204%); 28 MiB/s, 7 objects/s recovering
2022-10-28T03:43:28.273799+0200 mgr.skarb (mgr.40364478) 162021 :
cluster [DBG] pgmap v164237: 408 pgs: 1 active+remapped+backfilling, 1
remapped+premerge+backfilling+peered, 1 clean+premerge+peered, 1
active+clean+snaptrim, 41 active+remapped+backfill_wait, 363
active+clean; 3.5 TiB data, 11 TiB used, 19 TiB / 30 TiB avail; 1.3
KiB/s rd, 64 KiB/s wr, 14 op/s; 233157/2841877 objects misplaced
(8.204%); 18 MiB/s, 4 objects/s recovering
2022-10-28T03:43:28.379923+0200 mon.skarb (mon.0) 4719355 : cluster
[DBG] osdmap e18746: 9 total, 9 up, 9 in
2022-10-28T03:43:30.381806+0200 mon.skarb (mon.0) 4719356 : cluster
[WRN] Health check update: Reduced data availability: 1 pg inactive
(PG_AVAILABILITY)
2022-10-28T03:43:30.389645+0200 mon.skarb (mon.0) 4719357 : cluster
[DBG] osdmap e18747: 9 total, 9 up, 9 in
2022-10-28T03:43:30.275554+0200 mgr.skarb (mgr.40364478) 162022 :
cluster [DBG] pgmap v164239: 407 pgs: 1 active+remapped+backfilling, 1
clean+premerge+peered, 1 active+clean+snaptrim, 41
active+remapped+backfill_wait, 363 active+clean; 3.5 TiB data, 11 TiB
used, 19 TiB / 30 TiB avail; 233157/2837083 objects misplaced (8.218%)
And here is an overview of this situation:
root@skarb:/var/log/ceph# cat ceph.log.1.gz | gunzip | grep inactive
2022-10-27T22:40:32.115085+0200 mon.skarb (mon.0) 4717180 : cluster
[WRN] Health check failed: Reduced data availability: 2 pgs inactive
(PG_AVAILABILITY)
2022-10-27T22:50:00.000198+0200 mon.skarb (mon.0) 4717320 : cluster
[WRN] Health detail: HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-27T22:50:00.000245+0200 mon.skarb (mon.0) 4717321 : cluster
[WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive
2022-10-27T22:50:00.000263+0200 mon.skarb (mon.0) 4717322 : cluster
[WRN] pg 2.37 is stuck inactive for 10m, current state
remapped+premerge+backfill_wait+peered, last acting [2,6,3]
2022-10-27T22:50:00.000307+0200 mon.skarb (mon.0) 4717323 : cluster
[WRN] pg 2.77 is stuck inactive for 10m, current state
remapped+premerge+backfill_wait+peered, last acting [2,6,3]
2022-10-27T23:00:00.000173+0200 mon.skarb (mon.0) 4717460 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-27T23:10:00.000149+0200 mon.skarb (mon.0) 4717592 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-27T23:20:00.000140+0200 mon.skarb (mon.0) 4717718 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-27T23:30:00.000198+0200 mon.skarb (mon.0) 4717855 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-27T23:40:00.000133+0200 mon.skarb (mon.0) 4717988 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-27T23:50:00.000102+0200 mon.skarb (mon.0) 4718119 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T00:00:00.000129+0200 mon.skarb (mon.0) 4718259 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
root@skarb:/var/log/ceph# cat ceph.log | grep inactive
2022-10-28T00:10:00.000139+0200 mon.skarb (mon.0) 4718393 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T00:20:00.000265+0200 mon.skarb (mon.0) 4718525 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T00:30:00.004136+0200 mon.skarb (mon.0) 4718661 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T00:40:00.000131+0200 mon.skarb (mon.0) 4718741 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T00:50:00.000130+0200 mon.skarb (mon.0) 4718774 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T01:00:00.000094+0200 mon.skarb (mon.0) 4718805 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T01:10:00.000168+0200 mon.skarb (mon.0) 4718838 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T01:20:00.000146+0200 mon.skarb (mon.0) 4718873 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T01:30:00.000152+0200 mon.skarb (mon.0) 4718904 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T01:40:00.000151+0200 mon.skarb (mon.0) 4718935 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T01:50:00.000144+0200 mon.skarb (mon.0) 4718967 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T02:00:00.000170+0200 mon.skarb (mon.0) 4719004 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T02:10:00.000158+0200 mon.skarb (mon.0) 4719047 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive
2022-10-28T02:20:00.000109+0200 mon.skarb (mon.0) 4719078 : cluster
[WRN] Health detail: HEALTH_WARN Reduced data availability: 2 pgs
inactive; 3 pgs not deep-scrubbed in time
2022-10-28T02:20:00.000151+0200 mon.skarb (mon.0) 4719079 : cluster
[WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive
2022-10-28T02:20:00.000170+0200 mon.skarb (mon.0) 4719080 : cluster
[WRN] pg 2.37 is stuck inactive for 3h, current state
remapped+premerge+backfill_wait+peered, last acting [2,6,3]
2022-10-28T02:20:00.000185+0200 mon.skarb (mon.0) 4719081 : cluster
[WRN] pg 2.77 is stuck inactive for 3h, current state
remapped+premerge+backfill_wait+peered, last acting [2,6,3]
2022-10-28T02:30:00.000152+0200 mon.skarb (mon.0) 4719118 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive; 3
pgs not deep-scrubbed in time
2022-10-28T02:40:00.000141+0200 mon.skarb (mon.0) 4719150 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive; 3
pgs not deep-scrubbed in time
2022-10-28T02:50:00.000163+0200 mon.skarb (mon.0) 4719178 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive; 3
pgs not deep-scrubbed in time
2022-10-28T03:00:00.000148+0200 mon.skarb (mon.0) 4719211 : cluster
[WRN] Health detail: HEALTH_WARN Reduced data availability: 2 pgs
inactive; 4 pgs not deep-scrubbed in time
2022-10-28T03:00:00.000172+0200 mon.skarb (mon.0) 4719212 : cluster
[WRN] [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive
2022-10-28T03:00:00.000186+0200 mon.skarb (mon.0) 4719213 : cluster
[WRN] pg 2.37 is stuck inactive for 4h, current state
remapped+premerge+backfill_wait+peered, last acting [2,6,3]
2022-10-28T03:00:00.000196+0200 mon.skarb (mon.0) 4719214 : cluster
[WRN] pg 2.77 is stuck inactive for 4h, current state
remapped+premerge+backfill_wait+peered, last acting [2,6,3]
2022-10-28T03:10:00.000128+0200 mon.skarb (mon.0) 4719252 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive; 4
pgs not deep-scrubbed in time
2022-10-28T03:20:00.000143+0200 mon.skarb (mon.0) 4719280 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive; 4
pgs not deep-scrubbed in time
2022-10-28T03:30:00.000150+0200 mon.skarb (mon.0) 4719305 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive; 4
pgs not deep-scrubbed in time
2022-10-28T03:40:00.000151+0200 mon.skarb (mon.0) 4719341 : cluster
[WRN] overall HEALTH_WARN Reduced data availability: 2 pgs inactive; 4
pgs not deep-scrubbed in time
2022-10-28T03:43:30.381806+0200 mon.skarb (mon.0) 4719356 : cluster
[WRN] Health check update: Reduced data availability: 1 pg inactive
(PG_AVAILABILITY)
2022-10-28T03:43:32.444590+0200 mon.skarb (mon.0) 4719360 : cluster
[INF] Health check cleared: PG_AVAILABILITY (was: Reduced data
availability: 1 pg inactive)
There are 9 OSD's, all of them were up all the time. But the last 9-th
one was added just 2 days ago, and just before that 2 OSD's from one
host were destroyed and recreated. When the problem occurred there was
no PG marked as undersized (it managed to recreate 3rd replica by that
time). However, as you can see here, PG's are being merged due to
autoscaling which is turned on.
Also, since the new OSD was a nvme device, there were some minor changes
made to crush map (default rule was limited to use only hdd OSDs, and
new rule was created to pick nvme as the first OSD for new pool).
So, what happened here and how to prevent it from happening again?
Paweł
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
