Hi,
Yes, we have added new osd. Previously we had only one type disk, hdd. now
we have added ssd disk separate them with replicated_rule and device class
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
0 hdd 5.57100 1.00000 5.6 TiB 1.8 TiB 3.8 TiB 31.61 1.04 850
1 hdd 5.57100 1.00000 5.6 TiB 1.6 TiB 4.0 TiB 29.07 0.96 830
2 hdd 5.57100 1.00000 5.6 TiB 1.6 TiB 4.0 TiB 27.98 0.92 820
3 hdd 5.57100 1.00000 5.6 TiB 1.3 TiB 4.2 TiB 23.74 0.78 696
4 hdd 5.57100 1.00000 5.6 TiB 1.8 TiB 3.8 TiB 32.15 1.06 866
5 hdd 5.57100 1.00000 5.6 TiB 1.9 TiB 3.7 TiB 34.38 1.13 835
6 hdd 5.57100 1.00000 5.6 TiB 1.4 TiB 4.2 TiB 25.10 0.83 702
7 hdd 5.57100 1.00000 5.6 TiB 1.5 TiB 4.1 TiB 26.34 0.87 680
8 hdd 5.57100 1.00000 5.6 TiB 1.6 TiB 3.9 TiB 29.25 0.96 811
9 hdd 5.57100 1.00000 5.6 TiB 2.0 TiB 3.6 TiB 36.01 1.18 819
10 hdd 5.57100 1.00000 5.6 TiB 1.5 TiB 4.0 TiB 27.66 0.91 848
11 hdd 5.57100 1.00000 5.6 TiB 1.5 TiB 4.0 TiB 27.63 0.91 844
12 hdd 5.57100 1.00000 5.6 TiB 1.5 TiB 4.0 TiB 27.68 0.91 719
13 hdd 5.57100 1.00000 5.6 TiB 2.0 TiB 3.6 TiB 35.08 1.15 832
14 hdd 5.57100 1.00000 5.6 TiB 1.8 TiB 3.8 TiB 32.09 1.06 840
15 hdd 5.57100 1.00000 5.6 TiB 1.8 TiB 3.8 TiB 32.53 1.07 825
16 hdd 5.57100 1.00000 5.6 TiB 1.4 TiB 4.2 TiB 24.85 0.82 855
17 hdd 5.57100 1.00000 5.6 TiB 2.1 TiB 3.5 TiB 37.46 1.23 899
18 hdd 5.57100 1.00000 5.6 TiB 1.7 TiB 3.9 TiB 30.03 0.99 856
19 hdd 5.57100 1.00000 5.6 TiB 1.6 TiB 4.0 TiB 28.20 0.93 850
20 hdd 5.57100 1.00000 5.6 TiB 1.7 TiB 3.8 TiB 31.38 1.03 776
21 hdd 5.57100 1.00000 5.6 TiB 1.5 TiB 4.0 TiB 27.56 0.91 868
22 hdd 5.57100 1.00000 5.6 TiB 1.9 TiB 3.6 TiB 34.61 1.14 836
23 hdd 5.57100 1.00000 5.6 TiB 1.8 TiB 3.8 TiB 32.30 1.06 849
24 hdd 5.57100 1.00000 5.6 TiB 1.7 TiB 3.9 TiB 30.11 0.99 847
25 hdd 5.57100 1.00000 5.6 TiB 1.9 TiB 3.6 TiB 34.90 1.15 887
26 hdd 5.57100 1.00000 5.6 TiB 1.6 TiB 3.9 TiB 29.53 0.97 792
27 hdd 5.57100 1.00000 5.6 TiB 2.2 TiB 3.4 TiB 38.98 1.28 878
28 hdd 5.57100 1.00000 5.6 TiB 1.8 TiB 3.8 TiB 32.53 1.07 845
29 hdd 5.57100 1.00000 5.6 TiB 1.8 TiB 3.7 TiB 32.95 1.08 853
30 hdd 5.57100 1.00000 5.6 TiB 1.7 TiB 3.9 TiB 30.42 1.00 838
31 hdd 5.57100 1.00000 5.6 TiB 1.5 TiB 4.0 TiB 27.39 0.90 823
32 hdd 5.57100 1.00000 5.6 TiB 1.7 TiB 3.9 TiB 30.57 1.01 827
33 hdd 5.57100 1.00000 5.6 TiB 1.8 TiB 3.8 TiB 32.18 1.06 860
34 hdd 5.57100 1.00000 5.6 TiB 1.7 TiB 3.9 TiB 30.70 1.01 821
35 hdd 5.57100 1.00000 5.6 TiB 1.7 TiB 3.9 TiB 29.99 0.99 897
36 hdd 5.57100 1.00000 5.6 TiB 1.4 TiB 4.1 TiB 25.54 0.84 828
37 hdd 5.57100 1.00000 5.6 TiB 1.7 TiB 3.9 TiB 30.09 0.99 784
38 hdd 5.57100 1.00000 5.6 TiB 1.8 TiB 3.8 TiB 31.92 1.05 834
39 hdd 5.57100 1.00000 5.6 TiB 2.2 TiB 3.4 TiB 39.44 1.30 887
40 ssd 1.81898 1.00000 1.8 TiB 141 GiB 1.7 TiB 7.55 0.25 1035
41 ssd 1.81898 1.00000 1.8 TiB 94 GiB 1.7 TiB 5.07 0.17 1043
TOTAL 226 TiB 69 TiB 158 TiB 30.40
MIN/MAX VAR: 0.17/1.30 STDDEV: 6.38
# ceph health detail
HEALTH_ERR 11759382/17719047 objects misplaced (66.366%); Degraded data
redundancy: 1127230/17719047 objects degraded (6.362%), 644 pgs degraded,
652 pgs undersized; Degraded data redundancy (low space): 161 pgs
backfill_toofull
OBJECT_MISPLACED 11759382/17719047 objects misplaced (66.366%)
PG_DEGRADED Degraded data redundancy: 1127230/17719047 objects degraded
(6.362%), 644 pgs degraded, 652 pgs undersized
pg 16.30c is stuck undersized for 40925.003419, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [8,24]
pg 17.2dd is active+undersized+degraded+remapped+backfill_wait, acting
[27,34]
pg 17.2df is stuck undersized for 29653.050654, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [20,31]
pg 17.2e1 is stuck undersized for 40852.659824, current state
active+undersized+degraded+remapped+backfill_wait, last acting [19,32]
pg 17.2e5 is stuck undersized for 40947.285931, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [2,14]
pg 17.2e6 is stuck undersized for 40853.656156, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [15,27]
pg 17.2eb is stuck undersized for 29653.065202, current state
active+undersized+degraded+remapped+backfill_wait, last acting [39,26]
pg 17.2f0 is stuck undersized for 40951.126668, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [13,27]
pg 17.2f2 is stuck undersized for 40931.707178, current state
active+undersized+degraded+remapped+backfill_wait, last acting [15,28]
pg 17.2f7 is stuck undersized for 29653.031067, current state
active+undersized+degraded+remapped+backfill_wait, last acting [23,28]
pg 17.2fb is stuck undersized for 40853.695152, current state
active+undersized+degraded+remapped+backfill_wait, last acting [36,22]
pg 17.2ff is stuck undersized for 40947.940588, current state
active+undersized+degraded+remapped+backfill_wait, last acting [28,2]
pg 17.302 is stuck undersized for 40949.177851, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [0,13]
pg 17.30b is stuck undersized for 29651.994227, current state
active+undersized+degraded+remapped+backfill_wait, last acting [24,7]
pg 17.30d is stuck undersized for 40852.662189, current state
active+undersized+degraded+remapped+backfill_wait, last acting [18,27]
pg 17.315 is stuck undersized for 40944.552556, current state
active+undersized+degraded+remapped+backfill_wait, last acting [39,30]
pg 17.318 is stuck undersized for 29653.039859, current state
active+undersized+degraded+remapped+backfill_wait, last acting [28,19]
pg 17.319 is stuck undersized for 29653.035805, current state
active+undersized+degraded+remapped+backfill_wait, last acting [27,0]
pg 17.322 is stuck undersized for 29653.048491, current state
active+undersized+degraded+remapped+backfill_wait, last acting [16,28]
pg 17.324 is stuck undersized for 29651.973616, current state
active+undersized+degraded+remapped+backfill_wait, last acting [20,3]
pg 32.2c5 is stuck undersized for 40852.685144, current state
active+undersized+degraded+remapped+backfill_wait, last acting [37,20]
pg 32.2c6 is stuck undersized for 40852.635275, current state
active+undersized+degraded+remapped+backfill_wait, last acting [22,34]
pg 32.2cc is stuck undersized for 40928.723375, current state
active+undersized+degraded+remapped+backfill_wait, last acting [35,11]
pg 32.2cf is stuck undersized for 29651.975687, current state
active+undersized+degraded+remapped+backfill_wait, last acting [21,3]
pg 32.2dd is stuck undersized for 40850.500730, current state
active+undersized+degraded+remapped+backfill_wait, last acting [17,28]
pg 32.2df is stuck undersized for 40852.665581, current state
active+undersized+degraded+remapped+backfill_wait, last acting [30,10]
pg 32.326 is stuck undersized for 40953.205257, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [21,13]
pg 32.32d is stuck undersized for 40944.651277, current state
active+undersized+degraded+remapped+backfill_wait, last acting [39,11]
pg 32.337 is stuck undersized for 29652.993700, current state
active+undersized+degraded+remapped+backfill_wait, last acting [22,31]
pg 38.2e9 is stuck undersized for 40853.307863, current state
active+undersized+degraded+remapped+backfill_wait, last acting [23,36]
pg 38.312 is stuck undersized for 40853.732048, current state
active+undersized+degraded+remapped+backfill_wait, last acting [36,27]
pg 38.321 is stuck undersized for 40931.896842, current state
active+undersized+degraded+remapped+backfill_wait, last acting [23,8]
pg 38.33f is stuck undersized for 29653.021370, current state
active+undersized+degraded+remapped+backfill_wait, last acting [39,22]
pg 40.2c7 is stuck undersized for 40852.708839, current state
active+undersized+degraded+remapped+backfill_wait, last acting [32,15]
pg 40.2c8 is stuck undersized for 40947.619067, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [4,19]
pg 40.2c9 is stuck undersized for 40852.572681, current state
active+undersized+degraded+remapped+backfill_wait, last acting [23,28]
pg 40.2d1 is stuck undersized for 40943.572427, current state
active+undersized+degraded+remapped+backfill_wait, last acting [39,11]
pg 40.2dd is stuck undersized for 29653.027625, current state
active+undersized+degraded+remapped+backfill_wait, last acting [5,22]
pg 40.308 is stuck undersized for 40849.940027, current state
active+undersized+degraded+remapped+backfill_wait, last acting [23,11]
pg 40.30b is stuck undersized for 40853.608444, current state
active+undersized+degraded+remapped+backfill_wait, last acting [16,24]
pg 40.30d is stuck undersized for 29653.023739, current state
active+undersized+degraded+remapped+backfill_wait, last acting [22,25]
pg 40.31a is stuck undersized for 40951.103943, current state
active+undersized+degraded+remapped+backfill_wait, last acting [18,31]
pg 40.31c is stuck undersized for 40850.502975, current state
active+undersized+degraded+remapped+backfill_wait, last acting [17,30]
pg 40.31d is stuck undersized for 40852.659455, current state
active+undersized+degraded+remapped+backfill_wait, last acting [28,15]
pg 40.32a is stuck undersized for 40908.632328, current state
active+undersized+degraded+remapped+backfill_wait, last acting [5,32]
pg 40.32c is stuck undersized for 29651.973754, current state
active+undersized+degraded+remapped+backfill_wait, last acting [30,20]
pg 40.32d is stuck undersized for 40947.478757, current state
active+undersized+degraded+remapped+backfill_wait, last acting [39,0]
pg 40.32e is stuck undersized for 29653.057911, current state
active+undersized+degraded+remapped+backfill_wait, last acting [28,17]
pg 40.334 is stuck undersized for 40845.126881, current state
active+undersized+degraded+remapped+backfill_wait, last acting [39,9]
pg 40.335 is stuck undersized for 40951.140746, current state
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, last
acting [17,10]
pg 40.33c is stuck undersized for 40852.645060, current state
active+undersized+degraded+remapped+backfill_wait, last acting [23,34]
PG_DEGRADED_FULL Degraded data redundancy (low space): 161 pgs
backfill_toofull
pg 16.30c is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[8,24]
pg 17.1ea is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[21,9]
pg 17.1ee is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[9,19]
pg 17.200 is active+remapped+backfill_wait+backfill_toofull, acting
[7,11,32]
pg 17.208 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[10,37]
pg 17.21c is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[8,33]
pg 17.221 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[18,31]
pg 17.24e is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[8,21]
pg 17.253 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[10,26]
pg 17.268 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[9,22]
pg 17.269 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[1,22]
pg 17.2a0 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[2,27]
pg 17.2a8 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[8,31]
pg 17.2c0 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[2,19]
pg 17.2c6 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[21,33]
pg 17.2df is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[20,31]
pg 17.2e5 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[2,14]
pg 17.2e6 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[15,27]
pg 17.2f0 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[13,27]
pg 17.302 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[0,13]
pg 32.1de is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[15,28]
pg 32.212 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[10,18]
pg 32.216 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[15,25]
pg 32.23a is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[18,32]
pg 32.257 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[22,0]
pg 32.285 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[21,30]
pg 32.290 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[19,14]
pg 32.2aa is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[2,38]
pg 32.2b0 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[8,37]
pg 32.2e0 is active+remapped+backfill_wait+backfill_toofull, acting
[14,25,20]
pg 32.326 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[21,13]
pg 38.237 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[11,20]
pg 38.24f is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[16,9]
pg 38.2a2 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[1,22]
pg 40.1de is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[22,14]
pg 40.209 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[0,27]
pg 40.20a is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[22,13]
pg 40.225 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[10,36]
pg 40.233 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[18,29]
pg 40.235 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[19,27]
pg 40.246 is active+remapped+backfill_wait+backfill_toofull, acting
[7,27,38]
pg 40.249 is active+remapped+backfill_wait+backfill_toofull, acting
[14,27,33]
pg 40.267 is active+remapped+backfill_wait+backfill_toofull, acting
[7,23,38]
pg 40.272 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[1,35]
pg 40.287 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[15,35]
pg 40.290 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[8,34]
pg 40.2a9 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[9,38]
pg 40.2bc is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[8,37]
pg 40.2c8 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[4,19]
pg 40.2ff is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[10,20]
pg 40.335 is
active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting
[17,10]
Regards,
Munna
On Thu, Dec 9, 2021 at 12:39 AM Prayank Saxena <pr31189@xxxxxxxxx> wrote:
> Hi Munna,
>
> Have you added osd’s in the cluster recently?
> If yes, i think you have to re-weight the osd’s which you have added to
> lower values and slowly increase the weight one by one.
>
> Also, please share output of ‘ceph osd df’ and ‘ceph health details’
>
> On Wed, 8 Dec 2021 at 11:56 PM, Md. Hejbul Tawhid MUNNA <
> munnaeebd@xxxxxxxxx> wrote:
>
>> Hi,
>>
>> Overall status: HEALTH_ERR
>> PG_DEGRADED_FULL: Degraded data redundancy (low space): 19 pgs
>> backfill_toofull
>> OBJECT_MISPLACED: 12359314/17705640 objects misplaced (69.804%)
>> PG_DEGRADED: Degraded data redundancy: 1707105/17705640 objects degraded
>> (9.642%), 1979 pgs degraded, 1155 pgs undersized
>>
>> Any advice to resolve the issue. Its running in production
>>
>> Please let me know for any further information.
>>
>> Regards,
>> Munna
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
> --
>
>
>
>
> Regards
> Prayank Saxena
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
