PGs activating+remapped, PG overdose protection?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello people :)

we are facing a situation quite similar to the one described here:
http://tracker.ceph.com/issues/23117

Namely:

we have a Luminous cluster consisting of 16 hosts, where each host holds
12 OSDs on spinning disks and 4 OSDs on SSDs. Let's forget the SSDs for
now since they're not used atm.

We have a Erasure Coding pool (k=6, m=3) with 4096 PGs, residing on the
spinning disks, with failure domain the host.

After getting a host (and their OSDs) out for maintenance, we're trying
to put the OSDs back in. While cluster starts recovering we observe

> Reduced data availability: 170 pgs inactive

and

> 170  activating+remapped

This eventually leads to slow/stucked requests and we have to get the
OSDs out again.

While searching around we came across the already mentioned issue on
tracker [1] and we're wondering "PG overdose protection" [2] is what
we're really facing now.

Our cluster features:

"mon_max_pg_per_osd": "200",
"osd_max_pg_per_osd_hard_ratio": "2.000000",

What is more, we observed that the PGs distribution among the OSDs is
not uniform, eg:

> ID  CLASS WEIGHT    REWEIGHT SIZE   USE    AVAIL  %USE  VAR  PGS TYPE NAME         
>  -1       711.29004        -   666T   165T   500T     0    0   - root default      
> -17        44.68457        - 45757G 11266G 34491G 24.62 0.99   -     host rd3-1427 
>   9   hdd   3.66309  1.00000  3751G   976G  2774G 26.03 1.05 212         osd.9     
>  30   hdd   3.66309  1.00000  3751G   961G  2789G 25.64 1.03 209         osd.30    
>  46   hdd   3.66309  1.00000  3751G   902G  2848G 24.07 0.97 196         osd.46    
>  61   hdd   3.66309  1.00000  3751G   877G  2873G 23.40 0.94 190         osd.61    
>  76   hdd   3.66309  1.00000  3751G   984G  2766G 26.24 1.05 214         osd.76    
>  92   hdd   3.66309  1.00000  3751G   894G  2856G 23.84 0.96 194         osd.92    
> 107   hdd   3.66309  1.00000  3751G   881G  2869G 23.50 0.94 191         osd.107   
> 123   hdd   3.66309  1.00000  3751G   973G  2777G 25.97 1.04 212         osd.123   
> 138   hdd   3.66309  1.00000  3751G   975G  2775G 26.01 1.05 212         osd.138   
> 156   hdd   3.66309  1.00000  3751G   813G  2937G 21.69 0.87 176         osd.156   
> 172   hdd   3.66309  1.00000  3751G  1016G  2734G 27.09 1.09 221         osd.172   
> 188   hdd   3.66309  1.00000  3751G   998G  2752G 26.62 1.07 217         osd.188 

Could these OSDs, holding more than 200 PGs, contribute to the problem?

Is there any way to confirm that we're hitting the "PG overdose
protection"? If that's true how can restore our cluster back to normal.

Apart from getting these OSDs back to work, we're concerned about the
overall choices regarding the number of PGs (4096) for that (6,3) EC pool.

Any help appreciated,
Alex

[1] http://tracker.ceph.com/issues/23117
[2] https://ceph.com/community/new-luminous-pg-overdose-protection/
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux