active+remapped+backfill_toofull

Nghia Than <contact@xxxxxxxxxxxxxxx> · Tue, 19 Dec 2017 23:26:40 +0700

Hi,

My CEPH is stuck at this for few days, we added new OSD and nothing changed:

17 pgs backfill_toofull
17 pgs stuck unclean
recovery 21/5156264 objects degraded (0.000%)
recovery 52908/5156264 objects misplaced (1.026%)
8 near full osd(s)

And here is my ceph health detail:

HEALTH_WARN 17 pgs backfill_toofull; 17 pgs stuck unclean; recovery 21/5156264 objects degraded (0.000%); recovery 52908/5156264 objects misplaced (1.026%); 8 near full osd(s)
pg 1.231 is stuck unclean for 4367.088889, current state active+remapped+backfill_toofull, last acting [24,9]
pg 1.1e8 is stuck unclean for 7316.364770, current state active+remapped+backfill_toofull, last acting [16,3]
pg 1.188 is stuck unclean for 7315.400227, current state active+remapped+backfill_toofull, last acting [11,7]
pg 1.158 is stuck unclean for 7321.511627, current state active+remapped+backfill_toofull, last acting [11,17]
pg 1.81 is stuck unclean for 4366.683703, current state active+remapped+backfill_toofull, last acting [10,24]
pg 1.332 is stuck unclean for 7315.248115, current state active+remapped+backfill_toofull, last acting [23,1]
pg 1.2c2 is stuck unclean for 4365.635413, current state active+remapped+backfill_toofull, last acting [24,13]
pg 1.3c6 is stuck unclean for 7320.816089, current state active+remapped+backfill_toofull, last acting [11,20]
pg 1.26f is stuck unclean for 7315.882215, current state active+remapped+backfill_toofull, last acting [28,8]
pg 1.236 is stuck unclean for 7322.152706, current state active+remapped+backfill_toofull, last acting [8,26]
pg 1.249 is stuck unclean for 4366.885751, current state active+remapped+backfill_toofull, last acting [9,24]
pg 1.7b is stuck unclean for 7315.353072, current state active+remapped+backfill_toofull, last acting [28,3]
pg 1.1ec is stuck unclean for 7315.981062, current state active+remapped+backfill_toofull, last acting [16,0]
pg 1.248 is stuck unclean for 7324.062482, current state active+remapped+backfill_toofull, last acting [16,3]
pg 1.e4 is stuck unclean for 4370.009328, current state active+remapped+backfill_toofull, last acting [21,24]
pg 1.144 is stuck unclean for 7317.998393, current state active+remapped+backfill_toofull, last acting [26,3]
pg 0.5f is stuck unclean for 5877.987814, current state active+remapped+backfill_toofull, last acting [24,5]
pg 1.3c6 is active+remapped+backfill_toofull, acting [11,20]
pg 1.332 is active+remapped+backfill_toofull, acting [23,1]
pg 1.2c2 is active+remapped+backfill_toofull, acting [24,13]
pg 1.26f is active+remapped+backfill_toofull, acting [28,8]
pg 1.249 is active+remapped+backfill_toofull, acting [9,24]
pg 1.248 is active+remapped+backfill_toofull, acting [16,3]
pg 1.236 is active+remapped+backfill_toofull, acting [8,26]
pg 1.e4 is active+remapped+backfill_toofull, acting [21,24]
pg 0.5f is active+remapped+backfill_toofull, acting [24,5]
pg 1.7b is active+remapped+backfill_toofull, acting [28,3]
pg 1.81 is active+remapped+backfill_toofull, acting [10,24]
pg 1.144 is active+remapped+backfill_toofull, acting [26,3]
pg 1.158 is active+remapped+backfill_toofull, acting [11,17]
pg 1.188 is active+remapped+backfill_toofull, acting [11,7]
pg 1.1e8 is active+remapped+backfill_toofull, acting [16,3]
pg 1.1ec is active+remapped+backfill_toofull, acting [16,0]
pg 1.231 is active+remapped+backfill_toofull, acting [24,9]
recovery 21/5156264 objects degraded (0.000%)
recovery 52908/5156264 objects misplaced (1.026%)
osd.3 is near full at 92%
osd.4 is near full at 91%
osd.12 is near full at 92%
osd.17 is near full at 86%
osd.18 is near full at 87%
osd.23 is near full at 90%
osd.27 is near full at 85%
osd.28 is near full at 85%

I tried reweight OSD to smaller weight but nothing changed. This is my dump full_ratio:

[root@storcp ~]# ceph pg dump |grep full_ratio
dumped all in format plain
full_ratio 0.95
nearfull_ratio 0.85
[root@storcp ~]# 

And ceph osd df:

[root@storcp ~]# ceph osd df
ID WEIGHT  REWEIGHT SIZE   USE    AVAIL  %USE  VAR  PGS 
 0 0.86800  1.00000   888G   754G   134G 84.91 1.09 102 
 1 0.86800  1.00000   888G   734G   154G 82.63 1.06  90 
 2 0.86800  1.00000   888G   548G   339G 61.77 0.79  75 
 9 0.86800  1.00000   888G   658G   230G 74.09 0.95  81 
10 0.86800  1.00000   888G   659G   229G 74.17 0.95  79 
11 0.86800  1.00000   888G   706G   182G 79.49 1.02  91 
18 0.86800  1.00000   888G   774G   114G 87.14 1.12  94 
 3 0.86800  1.00000   888G   823G 67037M 92.63 1.19  99 
 4 0.86800  1.00000   888G   816G 73780M 91.89 1.18 102 
 5 0.86800  1.00000   888G   608G   279G 68.51 0.88  76 
12 0.86800  1.00000   888G   818G 72144M 92.07 1.18 111 
13 0.86800  1.00000   888G   657G   231G 73.94 0.95  84 
14 0.86800  1.00000   888G   668G   220G 75.16 0.96  84 
19 0.86800  1.00000   888G   547G   341G 61.59 0.79  75 
 6 0.86800  1.00000   888G   651G   237G 73.31 0.94  90 
 7 0.86800  1.00000   888G   542G   346G 61.01 0.78  68 
 8 0.86800  1.00000   888G   727G   160G 81.90 1.05  90 
15 0.86800  1.00000   888G   627G   260G 70.64 0.91  91 
16 0.86800  1.00000   888G   668G   220G 75.19 0.96  81 
17 0.86800  1.00000   888G   764G   124G 86.04 1.10  92 
20 0.86800  1.00000   888G   598G   289G 67.37 0.86  78 
21 0.86800  1.00000   888G   726G   162G 81.73 1.05  87 
22 0.86800  1.00000   888G   707G   181G 79.60 1.02  92 
23 0.86800  1.00000   888G   804G 85861M 90.57 1.16 104 
24 0.86800  0.79999   888G   726G   162G 81.73 1.05  90 
25 0.86800  1.00000   888G   579G   308G 65.24 0.84  80 
26 0.86800  1.00000   888G   696G   192G 78.36 1.00  95 
27 0.86800  1.00000   888G   757G   131G 85.20 1.09  98 
28 0.86800  1.00000   888G   758G   130G 85.29 1.09 104 
              TOTAL 25775G 20115G  5660G 78.04          
MIN/MAX VAR: 0.78/1.19  STDDEV: 9.24
[root@storcp ~]# 

May i know how to get over this?

--  ==============
Nghia Than

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com