> On Dec 19, 2017, at 8:26 AM, Nghia Than <contact@xxxxxxxxxxxxxxx> wrote: > > Hi, > > My CEPH is stuck at this for few days, we added new OSD and nothing changed: Does the new osd show up in osd tree? I see all your osd’s at ~80%, the new ones should be at much lower percentage or did they get full too? > > • 17 pgs backfill_toofull > • 17 pgs stuck unclean > • recovery 21/5156264 objects degraded (0.000%) > • recovery 52908/5156264 objects misplaced (1.026%) > • 8 near full osd(s) > > And here is my ceph health detail: > > HEALTH_WARN 17 pgs backfill_toofull; 17 pgs stuck unclean; recovery 21/5156264 objects degraded (0.000%); recovery 52908/5156264 objects misplaced (1.026%); 8 near full osd(s) > pg 1.231 is stuck unclean for 4367.088889, current state active+remapped+backfill_toofull, last acting [24,9] > pg 1.1e8 is stuck unclean for 7316.364770, current state active+remapped+backfill_toofull, last acting [16,3] > pg 1.188 is stuck unclean for 7315.400227, current state active+remapped+backfill_toofull, last acting [11,7] > pg 1.158 is stuck unclean for 7321.511627, current state active+remapped+backfill_toofull, last acting [11,17] > pg 1.81 is stuck unclean for 4366.683703, current state active+remapped+backfill_toofull, last acting [10,24] > pg 1.332 is stuck unclean for 7315.248115, current state active+remapped+backfill_toofull, last acting [23,1] > pg 1.2c2 is stuck unclean for 4365.635413, current state active+remapped+backfill_toofull, last acting [24,13] > pg 1.3c6 is stuck unclean for 7320.816089, current state active+remapped+backfill_toofull, last acting [11,20] > pg 1.26f is stuck unclean for 7315.882215, current state active+remapped+backfill_toofull, last acting [28,8] > pg 1.236 is stuck unclean for 7322.152706, current state active+remapped+backfill_toofull, last acting [8,26] > pg 1.249 is stuck unclean for 4366.885751, current state active+remapped+backfill_toofull, last acting [9,24] > pg 1.7b is stuck unclean for 7315.353072, current state active+remapped+backfill_toofull, last acting [28,3] > pg 1.1ec is stuck unclean for 7315.981062, current state active+remapped+backfill_toofull, last acting [16,0] > pg 1.248 is stuck unclean for 7324.062482, current state active+remapped+backfill_toofull, last acting [16,3] > pg 1.e4 is stuck unclean for 4370.009328, current state active+remapped+backfill_toofull, last acting [21,24] > pg 1.144 is stuck unclean for 7317.998393, current state active+remapped+backfill_toofull, last acting [26,3] > pg 0.5f is stuck unclean for 5877.987814, current state active+remapped+backfill_toofull, last acting [24,5] > pg 1.3c6 is active+remapped+backfill_toofull, acting [11,20] > pg 1.332 is active+remapped+backfill_toofull, acting [23,1] > pg 1.2c2 is active+remapped+backfill_toofull, acting [24,13] > pg 1.26f is active+remapped+backfill_toofull, acting [28,8] > pg 1.249 is active+remapped+backfill_toofull, acting [9,24] > pg 1.248 is active+remapped+backfill_toofull, acting [16,3] > pg 1.236 is active+remapped+backfill_toofull, acting [8,26] > pg 1.e4 is active+remapped+backfill_toofull, acting [21,24] > pg 0.5f is active+remapped+backfill_toofull, acting [24,5] > pg 1.7b is active+remapped+backfill_toofull, acting [28,3] > pg 1.81 is active+remapped+backfill_toofull, acting [10,24] > pg 1.144 is active+remapped+backfill_toofull, acting [26,3] > pg 1.158 is active+remapped+backfill_toofull, acting [11,17] > pg 1.188 is active+remapped+backfill_toofull, acting [11,7] > pg 1.1e8 is active+remapped+backfill_toofull, acting [16,3] > pg 1.1ec is active+remapped+backfill_toofull, acting [16,0] > pg 1.231 is active+remapped+backfill_toofull, acting [24,9] > recovery 21/5156264 objects degraded (0.000%) > recovery 52908/5156264 objects misplaced (1.026%) > osd.3 is near full at 92% > osd.4 is near full at 91% > osd.12 is near full at 92% > osd.17 is near full at 86% > osd.18 is near full at 87% > osd.23 is near full at 90% > osd.27 is near full at 85% > osd.28 is near full at 85% > > I tried reweight OSD to smaller weight but nothing changed. This is my dump full_ratio: > > [root@storcp ~]# ceph pg dump |grep full_ratio > dumped all in format plain > full_ratio 0.95 > nearfull_ratio 0.85 > [root@storcp ~]# > > And ceph osd df: > > [root@storcp ~]# ceph osd df > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 0 0.86800 1.00000 888G 754G 134G 84.91 1.09 102 > 1 0.86800 1.00000 888G 734G 154G 82.63 1.06 90 > 2 0.86800 1.00000 888G 548G 339G 61.77 0.79 75 > 9 0.86800 1.00000 888G 658G 230G 74.09 0.95 81 > 10 0.86800 1.00000 888G 659G 229G 74.17 0.95 79 > 11 0.86800 1.00000 888G 706G 182G 79.49 1.02 91 > 18 0.86800 1.00000 888G 774G 114G 87.14 1.12 94 > 3 0.86800 1.00000 888G 823G 67037M 92.63 1.19 99 > 4 0.86800 1.00000 888G 816G 73780M 91.89 1.18 102 > 5 0.86800 1.00000 888G 608G 279G 68.51 0.88 76 > 12 0.86800 1.00000 888G 818G 72144M 92.07 1.18 111 > 13 0.86800 1.00000 888G 657G 231G 73.94 0.95 84 > 14 0.86800 1.00000 888G 668G 220G 75.16 0.96 84 > 19 0.86800 1.00000 888G 547G 341G 61.59 0.79 75 > 6 0.86800 1.00000 888G 651G 237G 73.31 0.94 90 > 7 0.86800 1.00000 888G 542G 346G 61.01 0.78 68 > 8 0.86800 1.00000 888G 727G 160G 81.90 1.05 90 > 15 0.86800 1.00000 888G 627G 260G 70.64 0.91 91 > 16 0.86800 1.00000 888G 668G 220G 75.19 0.96 81 > 17 0.86800 1.00000 888G 764G 124G 86.04 1.10 92 > 20 0.86800 1.00000 888G 598G 289G 67.37 0.86 78 > 21 0.86800 1.00000 888G 726G 162G 81.73 1.05 87 > 22 0.86800 1.00000 888G 707G 181G 79.60 1.02 92 > 23 0.86800 1.00000 888G 804G 85861M 90.57 1.16 104 > 24 0.86800 0.79999 888G 726G 162G 81.73 1.05 90 > 25 0.86800 1.00000 888G 579G 308G 65.24 0.84 80 > 26 0.86800 1.00000 888G 696G 192G 78.36 1.00 95 > 27 0.86800 1.00000 888G 757G 131G 85.20 1.09 98 > 28 0.86800 1.00000 888G 758G 130G 85.29 1.09 104 > TOTAL 25775G 20115G 5660G 78.04 > MIN/MAX VAR: 0.78/1.19 STDDEV: 9.24 > [root@storcp ~]# > > May i know how to get over this? > > -- > ============== > Nghia Than > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |