Hi,
I follow and use your articles regularly to help with our Ceph environment, I am looking for urgent help with our infrastructure after a series of outages over the weekend has ground our ceph environment to its knees.
The system is .0.94.5 and deployed as part of open stack.
In the series of events the following has happened.. After a aircon failure in our hosting enviroment
We have 3 OSD Nodes and 3 MON's
1. One of the nodes failed due to the heat with the SSD journal dieing. This was left to do the usual ceph cluster repair between the two remaining nodes.
2. The second OSD node then failed with multiple rw errors on the backplane, this was left to cool down and a reboot recoved the node, apart from 2 OSD's that we not starting, they were then removed and recreated and left to repair.
3. We recreated the journals on the first failed node, symlink'd the journals and recreated with the OSD's coming online and the repair continuing.
The problem is it does not matter what we do at the moment we have zero client IO to our openstack environment, based on the health detail and blocked io I have tried OSD outing the ones that block, this moved the blocks to other OSD..
So hence reaching out for some professional if possible.
So hence reaching out for some professional if possible.
Below is the Health Detail
Croot@node31-a4:~ceph health detail
HEALTH_ERR 92 pgs backfill; 17 pgs backfill_toofull; 37 pgs backfilling; 26 pgs degraded; 31 pgs incomplete; 1 pgs inconsistent; 2 pgs recovery_wait; 1 pgs stale; 26 pgs stuck degraded; 31 pgs stuck inactive; 1 pgs stuck stale; 167 pgs stuck unclean; 22 pgs stuck undersized; 22 pgs undersized; 776 requests are blocked > 32 sec; 13 osds have slow requests; recovery 162084/4909838 objects degraded (3.301%); recovery 1129045/4909838 objects misplaced (22.996%); 1 scrub errors; noscrub,nodeep-scrub flag(s) set
pg 2.2a is stuck inactive for 106393.047125, current state incomplete, last acting [24,5]
pg 2.b0 is stuck inactive for 106764.257598, current state incomplete, last acting [24,7]
pg 9.42 is stuck inactive for 117599.672358, current state incomplete, last acting [31,12]
pg 9.de is stuck inactive since forever, current state incomplete, last acting [6,5]
pg 2.75 is stuck inactive since forever, current state incomplete, last acting [7,15]
pg 9.dc is stuck inactive for 122255.368826, current state incomplete, last acting [6,7]
pg 2.74 is stuck inactive for 106421.951578, current state incomplete, last acting [13,5]
pg 9.1e is stuck inactive since forever, current state incomplete, last acting [7,15]
pg 2.15 is stuck inactive since forever, current state incomplete, last acting [7,1]
pg 11.1c is stuck inactive since forever, current state incomplete, last acting [6,7]
pg 2.a1 is stuck inactive for 107549.457443, current state incomplete, last acting [14,12]
pg 9.d8 is stuck inactive for 123846.143723, current state incomplete, last acting [21,5]
pg 9.a8 is stuck inactive for 127338.603835, current state incomplete, last acting [14,7]
pg 9.78 is stuck inactive since forever, current state incomplete, last acting [5,24]
pg 2.a2 is stuck inactive since forever, current state incomplete, last acting [5,13]
pg 7.16 is stuck inactive since forever, current state incomplete, last acting [6,7]
pg 2.13 is stuck inactive since forever, current state incomplete, last acting [7,10]
pg 9.f5 is stuck inactive for 111773.007635, current state incomplete, last acting [18,5]
pg 2.d is stuck inactive since forever, current state incomplete, last acting [5,10]
pg 9.5 is stuck inactive since forever, current state incomplete, last acting [5,18]
pg 9.3 is stuck inactive since forever, current state incomplete, last acting [7,15]
pg 9.fc is stuck inactive for 210239.661539, current state incomplete, last acting [13,5]
pg 11.33 is stuck inactive since forever, current state incomplete, last acting [7,6]
pg 9.3f is stuck inactive since forever, current state incomplete, last acting [5,14]
pg 9.a is stuck inactive since forever, current state incomplete, last acting [2,35]
pg 2.63 is stuck inactive for 106428.745149, current state incomplete, last acting [31,7]
pg 2.3 is stuck inactive for 106418.848299, current state incomplete, last acting [14,5]
pg 2.32 is stuck inactive since forever, current state incomplete, last acting [5,13]
pg 2.bf is stuck inactive for 108677.444437, current state incomplete, last acting [15,7]
pg 9.26 is stuck inactive since forever, current state incomplete, last acting [5,24]
pg 9.22 is stuck inactive since forever, current state incomplete, last acting [31,7]
pg 9.24 is stuck unclean for 1526.160835, current state active+remapped+wait_backfill, last acting [36,10]
pg 2.f6 is stuck unclean for 1671.518273, current state active+remapped+backfilling, last acting [15,35]
pg 9.25 is stuck unclean for 28855.346557, current state active+remapped+wait_backfill+backfill_toofull, last acting [2,15]
pg 7.2b is stuck unclean for 107594.228813, current state stale+active+undersized+degraded, last acting [5]
pg 11.27 is stuck unclean for 1092.271193, current state active+remapped+wait_backfill+backfill_toofull, last acting [4,36]
pg 9.22 is stuck unclean since forever, current state incomplete, last acting [31,7]
pg 2.29 is stuck unclean for 14392.759144, current state active+remapped+backfilling, last acting [24,40]
pg 5.2d is stuck unclean for 1526.161831, current state active+remapped+wait_backfill, last acting [39,8]
pg 2.2a is stuck unclean for 106393.047543, current state incomplete, last acting [24,5]
pg 2.ed is stuck unclean for 1973.906519, current state active+remapped+wait_backfill, last acting [40,12]
pg 2.ec is stuck unclean for 1672.518631, current state active+remapped+backfilling, last acting [13,34]
pg 9.e7 is stuck unclean for 1091.816859, current state active+remapped+wait_backfill, last acting [36,11]
pg 2.24 is stuck unclean for 15155.442484, current state active+remapped+wait_backfill, last acting [13,40]
pg 11.2f is stuck unclean for 4380.625920, current state active+undersized+degraded+remapped+wait_backfill, last acting [39]
pg 2.e8 is stuck unclean for 20522.942384, current state active+remapped+wait_backfill, last acting [36,15]
pg 11.29 is stuck unclean for 15671.252669, current state active+remapped+backfilling, last acting [14,40]
pg 2.20 is stuck unclean for 4414.096950, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.eb is stuck unclean for 23238.520409, current state active+remapped+backfilling, last acting [31,0]
pg 2.ea is stuck unclean for 12358.965400, current state active+remapped+backfilling, last acting [34,4]
pg 9.29 is stuck unclean for 109100.740914, current state active+remapped+wait_backfill, last acting [21,1]
pg 9.16 is stuck unclean for 1675.249962, current state active+remapped+wait_backfill, last acting [38,4]
pg 11.14 is stuck unclean for 4881.295216, current state active+remapped+wait_backfill+backfill_toofull, last acting [8,14]
pg 2.1d is stuck unclean for 21009.460327, current state active+remapped+backfilling, last acting [3,6]
pg 9.ef is stuck unclean for 7721.829367, current state active+remapped+wait_backfill, last acting [1,4]
pg 11.15 is stuck unclean for 1092.265751, current state active+remapped+wait_backfill+backfill_toofull, last acting [34,9]
pg 2.1c is stuck unclean for 23447.324040, current state active+remapped+backfilling, last acting [14,11]
pg 11.16 is stuck unclean for 6387019.713095, current state active+remapped+backfilling, last acting [15,40]
pg 2.1f is stuck unclean for 20621.718171, current state active+remapped+wait_backfill, last acting [3,15]
pg 2.e6 is stuck unclean for 1526.160225, current state active+remapped+backfilling, last acting [0,2]
pg 2.1e is stuck unclean for 111294.887804, current state active+undersized+degraded+remapped+backfilling, last acting [13]
pg 9.12 is stuck unclean for 35515.025125, current state active+remapped+backfilling, last acting [10,12]
pg 9.eb is stuck unclean for 1986.656982, current state active+remapped+wait_backfill, last acting [3,39]
pg 5.1f is stuck unclean for 133010.215157, current state active+remapped+wait_backfill, last acting [1,11]
pg 2.e3 is stuck unclean for 2835.329234, current state active+remapped+backfilling, last acting [4,41]
pg 9.e8 is stuck unclean for 1671.512994, current state active+remapped+wait_backfill, last acting [39,14]
pg 2.1b is stuck unclean for 4418.999057, current state active+undersized+degraded+remapped+backfilling, last acting [41]
pg 11.13 is stuck unclean for 15671.251502, current state active+remapped+backfilling, last acting [14,8]
pg 9.1e is stuck unclean since forever, current state incomplete, last acting [7,15]
pg 11.1c is stuck unclean since forever, current state incomplete, last acting [6,7]
pg 2.15 is stuck unclean since forever, current state incomplete, last acting [7,1]
pg 2.dc is stuck unclean for 20473.343458, current state active+recovery_wait+degraded+remapped, last acting [40,4]
pg 2.df is stuck unclean for 13811.058314, current state active+remapped+wait_backfill, last acting [13,0]
pg 9.d4 is stuck unclean for 7767.022062, current state active+remapped+wait_backfill, last acting [18,39]
pg 11.1e is stuck unclean for 1977687.891447, current state active+remapped+wait_backfill, last acting [3,38]
pg 2.de is stuck unclean for 106385.186644, current state active+undersized+degraded+remapped+wait_backfill+backfill_toofull, last acting [3]
pg 9.1d is stuck unclean for 57113.387239, current state active+remapped+wait_backfill, last acting [0,4]
pg 9.d2 is stuck unclean for 1092.004695, current state active+remapped+backfilling, last acting [11,38]
pg 2.d8 is stuck unclean for 16182.152184, current state active+remapped+backfilling, last acting [21,41]
pg 11.19 is stuck unclean for 24418.152968, current state active+remapped+wait_backfill, last acting [38,15]
pg 9.18 is stuck unclean for 36263.679728, current state active+remapped+backfill_toofull, last acting [18,13]
pg 7.16 is stuck unclean since forever, current state incomplete, last acting [6,7]
pg 2.13 is stuck unclean since forever, current state incomplete, last acting [7,10]
pg 9.de is stuck unclean since forever, current state incomplete, last acting [6,5]
pg 9.6 is stuck unclean for 4214.590467, current state active+remapped+wait_backfill+backfill_toofull, last acting [41,2]
pg 2.d is stuck unclean since forever, current state incomplete, last acting [5,10]
pg 9.df is stuck unclean for 1092.378025, current state active+remapped+wait_backfill+backfill_toofull, last acting [2,35]
pg 2.d7 is stuck unclean for 92546.249349, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 9.dc is stuck unclean for 122255.369561, current state incomplete, last acting [6,7]
pg 9.5 is stuck unclean since forever, current state incomplete, last acting [5,18]
pg 2.d1 is stuck unclean for 92532.828413, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 9.2 is stuck unclean for 1526.160103, current state active+remapped+wait_backfill, last acting [39,12]
pg 9.3 is stuck unclean since forever, current state incomplete, last acting [7,15]
pg 2.8 is stuck unclean for 4340.900590, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 9.d8 is stuck unclean for 123846.144451, current state incomplete, last acting [21,5]
pg 2.b is stuck unclean for 16182.133217, current state active+remapped+backfilling, last acting [24,40]
pg 9.d9 is stuck unclean for 1021.081037, current state active+remapped+wait_backfill, last acting [39,4]
pg 11.d is stuck unclean for 23355.866613, current state active+remapped+wait_backfill, last acting [36,4]
pg 9.c4 is stuck unclean for 1092.259689, current state active+remapped+wait_backfill, last acting [3,36]
pg 9.c5 is stuck unclean for 20810.947695, current state active+remapped+wait_backfill, last acting [38,5]
pg 9.a is stuck unclean since forever, current state incomplete, last acting [2,35]
pg 11.9 is stuck unclean for 2015.295484, current state active+remapped+backfilling, last acting [21,39]
pg 2.0 is stuck unclean for 106637.057550, current state active+undersized+degraded+remapped+backfilling, last acting [1]
pg 2.cb is stuck unclean for 1969.572538, current state active+remapped+wait_backfill, last acting [1,4]
pg 2.3 is stuck unclean for 106418.849006, current state incomplete, last acting [14,5]
pg 2.2 is stuck unclean for 106498.061609, current state active+recovery_wait+degraded+remapped, last acting [13,9]
pg 2.c5 is stuck unclean for 4380.116033, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 9.cf is stuck unclean for 1981.041038, current state active+remapped+wait_backfill, last acting [38,11]
pg 2.c1 is stuck unclean for 4286.839271, current state active+undersized+degraded+remapped+wait_backfill, last acting [34]
pg 9.c9 is stuck unclean for 926.515906, current state active+remapped+wait_backfill, last acting [34,3]
pg 9.b6 is stuck unclean for 1526.914659, current state active+remapped+wait_backfill, last acting [38,4]
pg 9.b7 is stuck unclean for 1526.152259, current state active+remapped+wait_backfill, last acting [36,11]
pg 2.bf is stuck unclean for 108677.445165, current state incomplete, last acting [15,7]
pg 9.b2 is stuck unclean for 1111.324643, current state active+remapped+wait_backfill, last acting [38,18]
pg 2.b9 is stuck unclean for 2007.883548, current state active+remapped+backfill_toofull, last acting [40,31]
pg 9.b3 is stuck unclean for 966.059380, current state active+remapped+wait_backfill, last acting [3,35]
pg 2.ba is stuck unclean for 4422.180694, current state active+undersized+degraded+remapped+backfilling, last acting [40]
pg 2.b5 is stuck unclean for 36789.908854, current state active+remapped+wait_backfill+backfill_toofull, last acting [2,40]
pg 9.bd is stuck unclean for 870.689548, current state active+remapped+wait_backfill, last acting [1,10]
pg 2.b0 is stuck unclean for 106764.258450, current state incomplete, last acting [24,7]
pg 2.b3 is stuck unclean for 14392.751616, current state active+remapped+backfilling, last acting [3,0]
pg 2.af is stuck unclean for 1672.572137, current state active+remapped+wait_backfill, last acting [6,1]
pg 2.ae is stuck unclean for 20626.536114, current state active+degraded+remapped+backfilling, last acting [34,13]
pg 9.a2 is stuck unclean for 1091.734311, current state active+remapped+wait_backfill+backfill_toofull, last acting [8,38]
pg 9.a0 is stuck unclean for 1092.616760, current state active+remapped+wait_backfill, last acting [1,3]
pg 9.ae is stuck unclean for 869.945312, current state active+remapped+wait_backfill, last acting [8,35]
pg 2.a7 is stuck unclean for 870.479679, current state active+remapped+backfilling, last acting [2,1]
pg 2.a1 is stuck unclean for 107549.458293, current state incomplete, last acting [14,12]
pg 9.a8 is stuck unclean for 127338.604674, current state incomplete, last acting [14,7]
pg 2.a2 is stuck unclean since forever, current state incomplete, last acting [5,13]
pg 2.9d is stuck unclean for 20625.064995, current state active+remapped+wait_backfill, last acting [6,38]
pg 9.97 is stuck unclean for 1091.946506, current state active+remapped+backfilling, last acting [4,0]
pg 9.95 is stuck unclean for 1981.941139, current state active+remapped+wait_backfill, last acting [1,8]
pg 2.9b is stuck unclean for 14392.752743, current state active+remapped+wait_backfill, last acting [6,0]
pg 9.91 is stuck unclean for 1969.725543, current state active+remapped+wait_backfill, last acting [41,3]
pg 2.96 is stuck unclean for 2001.408581, current state active+remapped+wait_backfill+backfill_toofull, last acting [8,24]
pg 2.91 is stuck unclean for 37317.548536, current state active+remapped+backfilling, last acting [24,0]
pg 9.9a is stuck unclean for 1091.600521, current state active+remapped+wait_backfill, last acting [36,8]
pg 2.92 is stuck unclean for 33794.780329, current state active+undersized+degraded+remapped+backfilling, last acting [8]
pg 9.80 is stuck unclean for 122092.103275, current state active+remapped+wait_backfill, last acting [0,10]
pg 9.8f is stuck unclean for 12358.964551, current state active+remapped+wait_backfill, last acting [4,35]
pg 9.8d is stuck unclean for 1672.472951, current state active+remapped+wait_backfill, last acting [13,35]
pg 2.81 is stuck unclean for 92522.916550, current state active+undersized+degraded+remapped+wait_backfill, last acting [39]
pg 2.7e is stuck unclean for 2009.258005, current state active+remapped+wait_backfill+backfill_toofull, last acting [11,15]
pg 2.78 is stuck unclean for 20625.074358, current state active+remapped+wait_backfill, last acting [6,0]
pg 9.70 is stuck unclean for 1526.151570, current state active+remapped+wait_backfill, last acting [21,3]
pg 2.75 is stuck unclean since forever, current state incomplete, last acting [7,15]
pg 2.74 is stuck unclean for 106421.952507, current state incomplete, last acting [13,5]
pg 9.7c is stuck unclean for 1686.550215, current state active+remapped+backfill_toofull, last acting [5,39]
pg 2.71 is stuck unclean for 20622.912960, current state active+remapped+wait_backfill, last acting [21,3]
pg 2.73 is stuck unclean for 20622.986357, current state active+remapped+wait_backfill, last acting [39,15]
pg 9.78 is stuck unclean since forever, current state incomplete, last acting [5,24]
pg 9.79 is stuck unclean for 1092.619242, current state active+remapped+wait_backfill, last acting [39,3]
pg 2.6d is stuck unclean for 20626.395068, current state active+remapped+wait_backfill, last acting [13,4]
pg 9.66 is stuck unclean for 1985.730632, current state active+remapped+wait_backfill, last acting [39,4]
pg 2.6e is stuck unclean for 4345.796708, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 9.63 is stuck unclean for 1672.430038, current state active+remapped+wait_backfill, last acting [1,13]
pg 2.6a is stuck unclean for 29087.086323, current state active+remapped+backfilling, last acting [6,40]
pg 9.6c is stuck unclean for 1092.024607, current state active+remapped+backfill_toofull, last acting [41,2]
pg 2.61 is stuck unclean for 1275.192356, current state active+remapped+wait_backfill, last acting [36,3]
pg 9.6a is stuck unclean for 4193.627351, current state active+remapped+wait_backfill+backfill_toofull, last acting [4,36]
pg 2.63 is stuck unclean for 106428.746049, current state incomplete, last acting [31,7]
pg 2.5e is stuck unclean for 106499.633039, current state active+remapped+wait_backfill, last acting [36,35]
pg 2.5b is stuck unclean for 4319.281057, current state active+undersized+degraded+remapped+wait_backfill, last acting [38]
pg 2.54 is stuck unclean for 4288.707146, current state active+undersized+degraded+remapped+wait_backfill, last acting [38]
pg 3.50 is stuck unclean for 4295.462664, current state active+remapped+wait_backfill, last acting [24,1]
pg 9.59 is stuck unclean for 35632.554796, current state active+remapped+wait_backfill, last acting [31,34]
pg 9.44 is stuck unclean for 1526.158022, current state active+remapped+wait_backfill, last acting [0,8]
pg 2.4f is stuck unclean for 1055.424481, current state active+remapped+backfilling, last acting [11,40]
pg 2.4e is stuck unclean for 4460.329031, current state active+undersized+degraded+remapped+wait_backfill, last acting [38]
pg 9.42 is stuck unclean for 117599.673376, current state incomplete, last acting [31,12]
pg 2.45 is stuck unclean for 20048.149119, current state active+remapped+backfilling, last acting [24,2]
pg 9.4f is stuck unclean for 2015.296488, current state active+remapped+wait_backfill, last acting [0,21]
pg 2.44 is stuck unclean for 36387.008340, current state active+remapped+wait_backfill, last acting [6,11]
pg 9.49 is stuck unclean for 57106.178089, current state active+remapped+wait_backfill+backfill_toofull, last acting [2,31]
pg 9.37 is stuck unclean for 1627.140955, current state active+remapped+wait_backfill, last acting [36,2]
pg 11.35 is stuck unclean for 5881158.013723, current state active+remapped+wait_backfill, last acting [36,40]
pg 11.37 is stuck unclean for 1092.376476, current state active+remapped+backfilling, last acting [2,34]
pg 2.38 is stuck unclean for 13903.889671, current state active+remapped+wait_backfill, last acting [0,5]
pg 2.3b is stuck unclean for 23220.193403, current state active+remapped+wait_backfill, last acting [18,2]
pg 11.33 is stuck unclean since forever, current state incomplete, last acting [7,6]
pg 9.f6 is stuck unclean for 7673.274780, current state active+remapped+wait_backfill, last acting [14,11]
pg 9.3e is stuck unclean for 1627.658454, current state active+remapped+backfilling, last acting [36,12]
pg 9.3f is stuck unclean since forever, current state incomplete, last acting [5,14]
pg 9.f4 is stuck unclean for 4380.628457, current state active+undersized+degraded+remapped+wait_backfill, last acting [39]
pg 9.f5 is stuck unclean for 111773.008703, current state incomplete, last acting [18,5]
pg 7.34 is stuck unclean for 70436.168988, current state active+remapped+wait_backfill, last acting [13,39]
pg 2.31 is stuck unclean for 4380.635574, current state active+undersized+degraded+remapped+backfilling, last acting [39]
pg 2.fb is stuck unclean for 65937.554227, current state active+degraded+remapped+backfilling, last acting [6,8]
pg 9.38 is stuck unclean for 1526.162363, current state active+remapped+wait_backfill, last acting [39,8]
pg 2.fa is stuck unclean for 13117.872966, current state active+undersized+degraded+remapped+backfilling, last acting [38]
pg 2.32 is stuck unclean since forever, current state incomplete, last acting [5,13]
pg 9.fe is stuck unclean for 1092.638297, current state active+remapped+wait_backfill, last acting [3,41]
pg 9.26 is stuck unclean since forever, current state incomplete, last acting [5,24]
pg 2.2d is stuck unclean for 928.333643, current state active+remapped+backfilling, last acting [8,40]
pg 9.27 is stuck unclean for 2026.887994, current state active+remapped+wait_backfill, last acting [2,41]
pg 2.f7 is stuck unclean for 15679.101427, current state active+remapped+backfilling, last acting [4,15]
pg 9.fc is stuck unclean for 210239.662635, current state incomplete, last acting [13,5]
pg 7.2b is stuck undersized for 73045.738621, current state stale+active+undersized+degraded, last acting [5]
pg 11.2f is stuck undersized for 847.750939, current state active+undersized+degraded+remapped+wait_backfill, last acting [39]
pg 2.20 is stuck undersized for 847.756110, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.1e is stuck undersized for 849.081933, current state active+undersized+degraded+remapped+backfilling, last acting [13]
pg 2.1b is stuck undersized for 848.310321, current state active+undersized+degraded+remapped+backfilling, last acting [41]
pg 2.de is stuck undersized for 849.247989, current state active+undersized+degraded+remapped+wait_backfill+backfill_toofull, last acting [3]
pg 2.d7 is stuck undersized for 848.185110, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.d1 is stuck undersized for 849.300643, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.8 is stuck undersized for 849.172662, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.0 is stuck undersized for 849.299969, current state active+undersized+degraded+remapped+backfilling, last acting [1]
pg 2.c5 is stuck undersized for 848.293734, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 2.c1 is stuck undersized for 846.984387, current state active+undersized+degraded+remapped+wait_backfill, last acting [34]
pg 2.ba is stuck undersized for 847.725573, current state active+undersized+degraded+remapped+backfilling, last acting [40]
pg 2.92 is stuck undersized for 849.176102, current state active+undersized+degraded+remapped+backfilling, last acting [8]
pg 2.81 is stuck undersized for 849.309568, current state active+undersized+degraded+remapped+wait_backfill, last acting [39]
pg 2.6e is stuck undersized for 848.290328, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.5b is stuck undersized for 847.261619, current state active+undersized+degraded+remapped+wait_backfill, last acting [38]
pg 2.54 is stuck undersized for 847.264749, current state active+undersized+degraded+remapped+wait_backfill, last acting [38]
pg 2.4e is stuck undersized for 847.265787, current state active+undersized+degraded+remapped+wait_backfill, last acting [38]
pg 9.f4 is stuck undersized for 847.746939, current state active+undersized+degraded+remapped+wait_backfill, last acting [39]
pg 2.31 is stuck undersized for 1265.209720, current state active+undersized+degraded+remapped+backfilling, last acting [39]
pg 2.fa is stuck undersized for 847.848948, current state active+undersized+degraded+remapped+backfilling, last acting [38]
pg 7.2b is stuck degraded for 73045.738726, current state stale+active+undersized+degraded, last acting [5]
pg 11.2f is stuck degraded for 847.751044, current state active+undersized+degraded+remapped+wait_backfill, last acting [39]
pg 2.20 is stuck degraded for 847.756215, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.1e is stuck degraded for 849.082038, current state active+undersized+degraded+remapped+backfilling, last acting [13]
pg 2.1b is stuck degraded for 848.310426, current state active+undersized+degraded+remapped+backfilling, last acting [41]
pg 2.dc is stuck degraded for 1241.901373, current state active+recovery_wait+degraded+remapped, last acting [40,4]
pg 2.de is stuck degraded for 849.248097, current state active+undersized+degraded+remapped+wait_backfill+backfill_toofull, last acting [3]
pg 2.d7 is stuck degraded for 848.185218, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.d1 is stuck degraded for 849.300752, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.8 is stuck degraded for 849.172770, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.0 is stuck degraded for 849.300078, current state active+undersized+degraded+remapped+backfilling, last acting [1]
pg 2.2 is stuck degraded for 849.113582, current state active+recovery_wait+degraded+remapped, last acting [13,9]
pg 2.c5 is stuck degraded for 848.293845, current state active+undersized+degraded+remapped+wait_backfill, last acting [1]
pg 2.c1 is stuck degraded for 846.984499, current state active+undersized+degraded+remapped+wait_backfill, last acting [34]
pg 2.ba is stuck degraded for 847.725685, current state active+undersized+degraded+remapped+backfilling, last acting [40]
pg 2.ae is stuck degraded for 3632.366349, current state active+degraded+remapped+backfilling, last acting [34,13]
pg 2.92 is stuck degraded for 849.176217, current state active+undersized+degraded+remapped+backfilling, last acting [8]
pg 2.81 is stuck degraded for 849.309683, current state active+undersized+degraded+remapped+wait_backfill, last acting [39]
pg 2.6e is stuck degraded for 848.290443, current state active+undersized+degraded+remapped+wait_backfill, last acting [36]
pg 2.5b is stuck degraded for 847.261734, current state active+undersized+degraded+remapped+wait_backfill, last acting [38]
pg 2.54 is stuck degraded for 847.264863, current state active+undersized+degraded+remapped+wait_backfill, last acting [38]
pg 2.4e is stuck degraded for 847.265902, current state active+undersized+degraded+remapped+wait_backfill, last acting [38]
pg 9.f4 is stuck degraded for 847.747054, current state active+undersized+degraded+remapped+wait_backfill, last acting [39]
pg 2.31 is stuck degraded for 1265.209835, current state active+undersized+degraded+remapped+backfilling, last acting [39]
pg 2.fb is stuck degraded for 849.167040, current state active+degraded+remapped+backfilling, last acting [6,8]
pg 2.fa is stuck degraded for 847.849066, current state active+undersized+degraded+remapped+backfilling, last acting [38]
pg 7.2b is stuck stale for 72995.830862, current state stale+active+undersized+degraded, last acting [5]
pg 9.f6 is active+remapped+wait_backfill, acting [14,11]
pg 9.f4 is active+undersized+degraded+remapped+wait_backfill, acting [39]
pg 9.f5 is incomplete, acting [18,5]
pg 2.fb is active+degraded+remapped+backfilling, acting [6,8]
pg 2.fa is active+undersized+degraded+remapped+backfilling, acting [38]
pg 9.fe is active+remapped+wait_backfill, acting [3,41]
pg 9.fc is incomplete, acting [13,5]
pg 2.f7 is active+remapped+backfilling, acting [4,15]
pg 2.f6 is active+remapped+backfilling, acting [15,35]
pg 2.ed is active+remapped+wait_backfill, acting [40,12]
pg 9.e7 is active+remapped+wait_backfill, acting [36,11]
pg 2.ec is active+remapped+backfilling, acting [13,34]
pg 2.e8 is active+remapped+wait_backfill, acting [36,15]
pg 2.eb is active+remapped+backfilling, acting [31,0]
pg 2.ea is active+remapped+backfilling, acting [34,4]
pg 9.ef is active+remapped+wait_backfill, acting [1,4]
pg 2.e6 is active+remapped+backfilling, acting [0,2]
pg 9.eb is active+remapped+wait_backfill, acting [3,39]
pg 2.e0 is active+clean+inconsistent, acting [9,6]
pg 9.e8 is active+remapped+wait_backfill, acting [39,14]
pg 2.e3 is active+remapped+backfilling, acting [4,41]
pg 2.dc is active+recovery_wait+degraded+remapped, acting [40,4]
pg 9.d4 is active+remapped+wait_backfill, acting [18,39]
pg 2.df is active+remapped+wait_backfill, acting [13,0]
pg 2.de is active+undersized+degraded+remapped+wait_backfill+backfill_toofull, acting [3]
pg 9.d2 is active+remapped+backfilling, acting [11,38]
pg 2.d8 is active+remapped+backfilling, acting [21,41]
pg 9.de is incomplete, acting [6,5]
pg 9.df is active+remapped+wait_backfill+backfill_toofull, acting [2,35]
pg 9.dc is incomplete, acting [6,7]
pg 2.d7 is active+undersized+degraded+remapped+wait_backfill, acting [36]
pg 2.d1 is active+undersized+degraded+remapped+wait_backfill, acting [36]
pg 9.d8 is incomplete, acting [21,5]
pg 9.d9 is active+remapped+wait_backfill, acting [39,4]
pg 9.c4 is active+remapped+wait_backfill, acting [3,36]
pg 9.c5 is active+remapped+wait_backfill, acting [38,5]
pg 2.cb is active+remapped+wait_backfill, acting [1,4]
pg 2.c5 is active+undersized+degraded+remapped+wait_backfill, acting [1]
pg 9.cf is active+remapped+wait_backfill, acting [38,11]
pg 2.c1 is active+undersized+degraded+remapped+wait_backfill, acting [34]
pg 9.c9 is active+remapped+wait_backfill, acting [34,3]
pg 9.b6 is active+remapped+wait_backfill, acting [38,4]
pg 9.b7 is active+remapped+wait_backfill, acting [36,11]
pg 2.bf is incomplete, acting [15,7]
pg 2.b9 is active+remapped+backfill_toofull, acting [40,31]
pg 9.b2 is active+remapped+wait_backfill, acting [38,18]
pg 9.b3 is active+remapped+wait_backfill, acting [3,35]
pg 2.ba is active+undersized+degraded+remapped+backfilling, acting [40]
pg 2.b5 is active+remapped+wait_backfill+backfill_toofull, acting [2,40]
pg 9.bd is active+remapped+wait_backfill, acting [1,10]
pg 2.b0 is incomplete, acting [24,7]
pg 2.b3 is active+remapped+backfilling, acting [3,0]
pg 2.af is active+remapped+wait_backfill, acting [6,1]
pg 2.ae is active+degraded+remapped+backfilling, acting [34,13]
pg 9.a2 is active+remapped+wait_backfill+backfill_toofull, acting [8,38]
pg 9.a0 is active+remapped+wait_backfill, acting [1,3]
pg 9.ae is active+remapped+wait_backfill, acting [8,35]
pg 2.a7 is active+remapped+backfilling, acting [2,1]
pg 2.a1 is incomplete, acting [14,12]
pg 9.a8 is incomplete, acting [14,7]
pg 2.a2 is incomplete, acting [5,13]
pg 2.9d is active+remapped+wait_backfill, acting [6,38]
pg 9.97 is active+remapped+backfilling, acting [4,0]
pg 9.95 is active+remapped+wait_backfill, acting [1,8]
pg 2.9b is active+remapped+wait_backfill, acting [6,0]
pg 9.91 is active+remapped+wait_backfill, acting [41,3]
pg 2.96 is active+remapped+wait_backfill+backfill_toofull, acting [8,24]
pg 2.91 is active+remapped+backfilling, acting [24,0]
pg 9.9a is active+remapped+wait_backfill, acting [36,8]
pg 2.92 is active+undersized+degraded+remapped+backfilling, acting [8]
pg 9.80 is active+remapped+wait_backfill, acting [0,10]
pg 9.8f is active+remapped+wait_backfill, acting [4,35]
pg 9.8d is active+remapped+wait_backfill, acting [13,35]
pg 2.81 is active+undersized+degraded+remapped+wait_backfill, acting [39]
pg 2.7e is active+remapped+wait_backfill+backfill_toofull, acting [11,15]
pg 2.78 is active+remapped+wait_backfill, acting [6,0]
pg 9.70 is active+remapped+wait_backfill, acting [21,3]
pg 2.75 is incomplete, acting [7,15]
pg 2.74 is incomplete, acting [13,5]
pg 9.7c is active+remapped+backfill_toofull, acting [5,39]
pg 2.71 is active+remapped+wait_backfill, acting [21,3]
pg 2.73 is active+remapped+wait_backfill, acting [39,15]
pg 9.78 is incomplete, acting [5,24]
pg 9.79 is active+remapped+wait_backfill, acting [39,3]
pg 2.6d is active+remapped+wait_backfill, acting [13,4]
pg 9.66 is active+remapped+wait_backfill, acting [39,4]
pg 2.6e is active+undersized+degraded+remapped+wait_backfill, acting [36]
pg 9.63 is active+remapped+wait_backfill, acting [1,13]
pg 2.6a is active+remapped+backfilling, acting [6,40]
pg 9.6c is active+remapped+backfill_toofull, acting [41,2]
pg 2.61 is active+remapped+wait_backfill, acting [36,3]
pg 9.6a is active+remapped+wait_backfill+backfill_toofull, acting [4,36]
pg 2.63 is incomplete, acting [31,7]
pg 2.5e is active+remapped+wait_backfill, acting [36,35]
pg 2.5b is active+undersized+degraded+remapped+wait_backfill, acting [38]
pg 2.54 is active+undersized+degraded+remapped+wait_backfill, acting [38]
pg 3.50 is active+remapped+wait_backfill, acting [24,1]
pg 9.59 is active+remapped+wait_backfill, acting [31,34]
pg 2.4f is active+remapped+backfilling, acting [11,40]
pg 9.44 is active+remapped+wait_backfill, acting [0,8]
pg 2.4e is active+undersized+degraded+remapped+wait_backfill, acting [38]
pg 9.42 is incomplete, acting [31,12]
pg 2.45 is active+remapped+backfilling, acting [24,2]
pg 2.44 is active+remapped+wait_backfill, acting [6,11]
pg 9.4f is active+remapped+wait_backfill, acting [0,21]
pg 9.49 is active+remapped+wait_backfill+backfill_toofull, acting [2,31]
pg 11.35 is active+remapped+wait_backfill, acting [36,40]
pg 9.37 is active+remapped+wait_backfill, acting [36,2]
pg 11.37 is active+remapped+backfilling, acting [2,34]
pg 2.38 is active+remapped+wait_backfill, acting [0,5]
pg 2.3b is active+remapped+wait_backfill, acting [18,2]
pg 11.33 is incomplete, acting [7,6]
pg 9.3e is active+remapped+backfilling, acting [36,12]
pg 9.3f is incomplete, acting [5,14]
pg 2.31 is active+undersized+degraded+remapped+backfilling, acting [39]
pg 7.34 is active+remapped+wait_backfill, acting [13,39]
pg 9.38 is active+remapped+wait_backfill, acting [39,8]
pg 2.32 is incomplete, acting [5,13]
pg 2.2d is active+remapped+backfilling, acting [8,40]
pg 9.26 is incomplete, acting [5,24]
pg 9.27 is active+remapped+wait_backfill, acting [2,41]
pg 9.24 is active+remapped+wait_backfill, acting [36,10]
pg 11.27 is active+remapped+wait_backfill+backfill_toofull, acting [4,36]
pg 9.25 is active+remapped+wait_backfill+backfill_toofull, acting [2,15]
pg 2.29 is active+remapped+backfilling, acting [24,40]
pg 9.22 is incomplete, acting [31,7]
pg 2.2a is incomplete, acting [24,5]
pg 5.2d is active+remapped+wait_backfill, acting [39,8]
pg 2.24 is active+remapped+wait_backfill, acting [13,40]
pg 11.2f is active+undersized+degraded+remapped+wait_backfill, acting [39]
pg 2.20 is active+undersized+degraded+remapped+wait_backfill, acting [36]
pg 11.29 is active+remapped+backfilling, acting [14,40]
pg 9.29 is active+remapped+wait_backfill, acting [21,1]
pg 2.1d is active+remapped+backfilling, acting [3,6]
pg 11.14 is active+remapped+wait_backfill+backfill_toofull, acting [8,14]
pg 9.16 is active+remapped+wait_backfill, acting [38,4]
pg 2.1c is active+remapped+backfilling, acting [14,11]
pg 11.15 is active+remapped+wait_backfill+backfill_toofull, acting [34,9]
pg 2.1f is active+remapped+wait_backfill, acting [3,15]
pg 11.16 is active+remapped+backfilling, acting [15,40]
pg 2.1e is active+undersized+degraded+remapped+backfilling, acting [13]
pg 9.12 is active+remapped+backfilling, acting [10,12]
pg 5.1f is active+remapped+wait_backfill, acting [1,11]
pg 2.1b is active+undersized+degraded+remapped+backfilling, acting [41]
pg 11.13 is active+remapped+backfilling, acting [14,8]
pg 2.15 is incomplete, acting [7,1]
pg 11.1c is incomplete, acting [6,7]
pg 9.1e is incomplete, acting [7,15]
pg 11.1e is active+remapped+wait_backfill, acting [3,38]
pg 9.1d is active+remapped+wait_backfill, acting [0,4]
pg 11.19 is active+remapped+wait_backfill, acting [38,15]
pg 2.13 is incomplete, acting [7,10]
pg 7.16 is incomplete, acting [6,7]
pg 9.18 is active+remapped+backfill_toofull, acting [18,13]
pg 2.d is incomplete, acting [5,10]
pg 9.6 is active+remapped+wait_backfill+backfill_toofull, acting [41,2]
pg 9.5 is incomplete, acting [5,18]
pg 9.2 is active+remapped+wait_backfill, acting [39,12]
pg 2.8 is active+undersized+degraded+remapped+wait_backfill, acting [36]
pg 9.3 is incomplete, acting [7,15]
pg 2.b is active+remapped+backfilling, acting [24,40]
pg 11.d is active+remapped+wait_backfill, acting [36,4]
pg 9.a is incomplete, acting [2,35]
pg 2.0 is active+undersized+degraded+remapped+backfilling, acting [1]
pg 11.9 is active+remapped+backfilling, acting [21,39]
pg 2.3 is incomplete, acting [14,5]
pg 2.2 is active+recovery_wait+degraded+remapped, acting [13,9]
2 ops are blocked > 2097.15 sec
690 ops are blocked > 1048.58 sec
77 ops are blocked > 65.536 sec
7 ops are blocked > 32.768 sec
2 ops are blocked > 2097.15 sec on osd.2
100 ops are blocked > 1048.58 sec on osd.5
9 ops are blocked > 1048.58 sec on osd.6
77 ops are blocked > 65.536 sec on osd.6
80 ops are blocked > 1048.58 sec on osd.7
3 ops are blocked > 32.768 sec on osd.9
99 ops are blocked > 1048.58 sec on osd.13
100 ops are blocked > 1048.58 sec on osd.14
100 ops are blocked > 1048.58 sec on osd.15
1 ops are blocked > 32.768 sec on osd.18
2 ops are blocked > 1048.58 sec on osd.21
100 ops are blocked > 1048.58 sec on osd.24
100 ops are blocked > 1048.58 sec on osd.31
3 ops are blocked > 32.768 sec on osd.36
13 osds have slow requests
recovery 162084/4909838 objects degraded (3.301%)
recovery 1129045/4909838 objects misplaced (22.996%)
1 scrub errors
noscrub,nodeep-scrub flag(s) set
root@node31-a4:~#
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
