You should just need to restart the relavent OSDs for the new backfill threshold to kick in.On 20 Dec 2017 00:14, "Nghia Than" <contact@xxxxxxxxxxxxxxx> wrote:I added more OSDs few days ago to reduce usage under 70% (nearfull and full ratio is higher than this value) and it still stuck at backfill_toofull while rebalance data.I tried to change backfill full ratio and it show error (unchangeable) as below:[root@storcp ~]# ceph tell osd.\* injectargs '--osd_backfill_full_ratio 0.92'
osd.0: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.1: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.2: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.3: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.4: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.5: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.6: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.7: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.8: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.9: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.10: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.11: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.12: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.13: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.14: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.15: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.16: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.17: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.18: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.19: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.20: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.21: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.22: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.23: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.24: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.25: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.26: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.27: osd_backfill_full_ratio = '0.92' (unchangeable)
osd.28: osd_backfill_full_ratio = '0.92' (unchangeable)
[root@storcp ~]#
--On Wed, Dec 20, 2017 at 1:57 AM, David C <dcsysengineer@xxxxxxxxx> wrote:What's your backfill full ratio? You may be able to get healthy by increasing your backfill full ratio (in small increments). But your next immediate task should be to add more OSDs or remove data.On 19 Dec 2017 4:26 p.m., "Nghia Than" <contact@xxxxxxxxxxxxxxx> wrote:_______________________________________________Hi,My CEPH is stuck at this for few days, we added new OSD and nothing changed:
- 17 pgs backfill_toofull
- 17 pgs stuck unclean
- recovery 21/5156264 objects degraded (0.000%)
- recovery 52908/5156264 objects misplaced (1.026%)
- 8 near full osd(s)
And here is my ceph health detail:HEALTH_WARN 17 pgs backfill_toofull; 17 pgs stuck unclean; recovery 21/5156264 objects degraded (0.000%); recovery 52908/5156264 objects misplaced (1.026%); 8 near full osd(s)
pg 1.231 is stuck unclean for 4367.088889, current state active+remapped+backfill_toofull, last acting [24,9]
pg 1.1e8 is stuck unclean for 7316.364770, current state active+remapped+backfill_toofull, last acting [16,3]
pg 1.188 is stuck unclean for 7315.400227, current state active+remapped+backfill_toofull, last acting [11,7]
pg 1.158 is stuck unclean for 7321.511627, current state active+remapped+backfill_toofull, last acting [11,17]
pg 1.81 is stuck unclean for 4366.683703, current state active+remapped+backfill_toofull, last acting [10,24]
pg 1.332 is stuck unclean for 7315.248115, current state active+remapped+backfill_toofull, last acting [23,1]
pg 1.2c2 is stuck unclean for 4365.635413, current state active+remapped+backfill_toofull, last acting [24,13]
pg 1.3c6 is stuck unclean for 7320.816089, current state active+remapped+backfill_toofull, last acting [11,20]
pg 1.26f is stuck unclean for 7315.882215, current state active+remapped+backfill_toofull, last acting [28,8]
pg 1.236 is stuck unclean for 7322.152706, current state active+remapped+backfill_toofull, last acting [8,26]
pg 1.249 is stuck unclean for 4366.885751, current state active+remapped+backfill_toofull, last acting [9,24]
pg 1.7b is stuck unclean for 7315.353072, current state active+remapped+backfill_toofull, last acting [28,3]
pg 1.1ec is stuck unclean for 7315.981062, current state active+remapped+backfill_toofull, last acting [16,0]
pg 1.248 is stuck unclean for 7324.062482, current state active+remapped+backfill_toofull, last acting [16,3]
pg 1.e4 is stuck unclean for 4370.009328, current state active+remapped+backfill_toofull, last acting [21,24]
pg 1.144 is stuck unclean for 7317.998393, current state active+remapped+backfill_toofull, last acting [26,3]
pg 0.5f is stuck unclean for 5877.987814, current state active+remapped+backfill_toofull, last acting [24,5]
pg 1.3c6 is active+remapped+backfill_toofull, acting [11,20]
pg 1.332 is active+remapped+backfill_toofull, acting [23,1]
pg 1.2c2 is active+remapped+backfill_toofull, acting [24,13]
pg 1.26f is active+remapped+backfill_toofull, acting [28,8]
pg 1.249 is active+remapped+backfill_toofull, acting [9,24]
pg 1.248 is active+remapped+backfill_toofull, acting [16,3]
pg 1.236 is active+remapped+backfill_toofull, acting [8,26]
pg 1.e4 is active+remapped+backfill_toofull, acting [21,24]
pg 0.5f is active+remapped+backfill_toofull, acting [24,5]
pg 1.7b is active+remapped+backfill_toofull, acting [28,3]
pg 1.81 is active+remapped+backfill_toofull, acting [10,24]
pg 1.144 is active+remapped+backfill_toofull, acting [26,3]
pg 1.158 is active+remapped+backfill_toofull, acting [11,17]
pg 1.188 is active+remapped+backfill_toofull, acting [11,7]
pg 1.1e8 is active+remapped+backfill_toofull, acting [16,3]
pg 1.1ec is active+remapped+backfill_toofull, acting [16,0]
pg 1.231 is active+remapped+backfill_toofull, acting [24,9]
recovery 21/5156264 objects degraded (0.000%)
recovery 52908/5156264 objects misplaced (1.026%)
osd.3 is near full at 92%
osd.4 is near full at 91%
osd.12 is near full at 92%
osd.17 is near full at 86%
osd.18 is near full at 87%
osd.23 is near full at 90%
osd.27 is near full at 85%
osd.28 is near full at 85%I tried reweight OSD to smaller weight but nothing changed. This is my dump full_ratio:[root@storcp ~]# ceph pg dump |grep full_ratio
dumped all in format plain
full_ratio 0.95
nearfull_ratio 0.85
[root@storcp ~]#And ceph osd df:[root@storcp ~]# ceph osd df
ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
0 0.86800 1.00000 888G 754G 134G 84.91 1.09 102
1 0.86800 1.00000 888G 734G 154G 82.63 1.06 90
2 0.86800 1.00000 888G 548G 339G 61.77 0.79 75
9 0.86800 1.00000 888G 658G 230G 74.09 0.95 81
10 0.86800 1.00000 888G 659G 229G 74.17 0.95 79
11 0.86800 1.00000 888G 706G 182G 79.49 1.02 91
18 0.86800 1.00000 888G 774G 114G 87.14 1.12 94
3 0.86800 1.00000 888G 823G 67037M 92.63 1.19 99
4 0.86800 1.00000 888G 816G 73780M 91.89 1.18 102
5 0.86800 1.00000 888G 608G 279G 68.51 0.88 76
12 0.86800 1.00000 888G 818G 72144M 92.07 1.18 111
13 0.86800 1.00000 888G 657G 231G 73.94 0.95 84
14 0.86800 1.00000 888G 668G 220G 75.16 0.96 84
19 0.86800 1.00000 888G 547G 341G 61.59 0.79 75
6 0.86800 1.00000 888G 651G 237G 73.31 0.94 90
7 0.86800 1.00000 888G 542G 346G 61.01 0.78 68
8 0.86800 1.00000 888G 727G 160G 81.90 1.05 90
15 0.86800 1.00000 888G 627G 260G 70.64 0.91 91
16 0.86800 1.00000 888G 668G 220G 75.19 0.96 81
17 0.86800 1.00000 888G 764G 124G 86.04 1.10 92
20 0.86800 1.00000 888G 598G 289G 67.37 0.86 78
21 0.86800 1.00000 888G 726G 162G 81.73 1.05 87
22 0.86800 1.00000 888G 707G 181G 79.60 1.02 92
23 0.86800 1.00000 888G 804G 85861M 90.57 1.16 104
24 0.86800 0.79999 888G 726G 162G 81.73 1.05 90
25 0.86800 1.00000 888G 579G 308G 65.24 0.84 80
26 0.86800 1.00000 888G 696G 192G 78.36 1.00 95
27 0.86800 1.00000 888G 757G 131G 85.20 1.09 98
28 0.86800 1.00000 888G 758G 130G 85.29 1.09 104
TOTAL 25775G 20115G 5660G 78.04
MIN/MAX VAR: 0.78/1.19 STDDEV: 9.24
[root@storcp ~]# --May i know how to get over this?==============
Nghia Than
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
==============
Nghia Than
Nghia Than
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
