НА: ceph pg backfill_toofull

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



​Ok, I'll try these params. thx!


От: Maged Mokhtar <mmokhtar@xxxxxxxxxxx>
Отправлено: 12 декабря 2018 г. 10:51
Кому: Klimenko, Roman; ceph-users@xxxxxxxxxxxxxx
Тема: Re: ceph pg backfill_toofull
 


There are 2 relevant params

mon_osd_full_ratio     0.95

osd_backfill_full_ratio 0.85
you are probably hitting them both
As a short term/ temp fix you may increase these values and maybe adjust weights on osds if you have to.
However you really need to fix this by adding more osds to your cluster, else it will happen again and again. Also when planing for required storage capacity, you should plan when 1 or 2 hosts fail and their pgs will distributed on remaining nodes, else you will hit the same issue.

/Maged




On 12/12/2018 07:52, Klimenko, Roman wrote:

Hi everyone. Yesterday i found that on our overcrowded Hammer ceph cluster (83% used in HDD pool) several osds were in danger zone - near 95%. 

I reweighted them, and after several moments I got pgs stuck in backfill_toofull. 

After that, I reapplied reweight to osds - no luck. 

Currently, all reweights are equal 1.0, and ceph do nothing - no rebalance and recovering. 

How I can make ceph recover these pgs?

ceph -s 

     health HEALTH_WARN
            47 pgs backfill_toofull
            47 pgs stuck unclean
            recovery 16/9422472 objects degraded (0.000%)
            recovery 365332/9422472 objects misplaced (3.877%)
            7 near full osd(s)

ceph osd df tree
ID WEIGHT   REWEIGHT SIZE   USE    AVAIL %USE  VAR  TYPE NAME
-1 30.65996        - 37970G 29370G 8599G 77.35 1.00 root default
-6 18.65996        - 20100G 16681G 3419G 82.99 1.07     region HDD
-3  6.09000        -  6700G  5539G 1160G 82.68 1.07         host ceph03.HDD
 1  1.00000  1.00000  1116G   841G  274G 75.39 0.97             osd.1
 5  1.00000  1.00000  1116G   916G  200G 82.07 1.06             osd.5
 3  1.00000  1.00000  1116G   939G  177G 84.14 1.09             osd.3
 8  1.09000  1.00000  1116G   952G  164G 85.29 1.10             osd.8
 7  1.00000  1.00000  1116G   972G  143G 87.11 1.13             osd.7
11  1.00000  1.00000  1116G   916G  200G 82.08 1.06             osd.11
-4  6.16998        -  6700G  5612G 1088G 83.76 1.08         host ceph02.HDD
14  1.09000  1.00000  1116G   950G  165G 85.16 1.10             osd.14
13  0.89999  1.00000  1116G   949G  167G 85.03 1.10             osd.13
16  1.09000  1.00000  1116G   921G  195G 82.50 1.07             osd.16
17  1.00000  1.00000  1116G   899G  216G 80.59 1.04             osd.17
10  1.09000  1.00000  1116G   952G  164G 85.28 1.10             osd.10
15  1.00000  1.00000  1116G   938G  178G 84.02 1.09             osd.15
-2  6.39998        -  6700G  5529G 1170G 82.53 1.07         host ceph01.HDD
12  1.09000  1.00000  1116G   953G  163G 85.39 1.10             osd.12
 9  0.95000  1.00000  1116G   939G  177G 84.14 1.09             osd.9
 2  1.09000  1.00000  1116G   911G  204G 81.64 1.06             osd.2
 0  1.09000  1.00000  1116G   951G  165G 85.22 1.10             osd.0
 6  1.09000  1.00000  1116G   917G  199G 82.12 1.06             osd.6
 4  1.09000  1.00000  1116G   856G  260G 76.67 0.99             osd.4








_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-- 
Maged Mokhtar
CEO PetaSAN
4 Emad El Deen Kamel
Cairo 11371, Egypt
www.petasan.org
+201006979931
skype: maged.mokhtar
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux