Re: Ceph is moving data ONLY to near-full OSDs [BUG]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I was following the pg autoscaler recommendations and I did not get a recommendation to raise the PGs there.
I'll try that, I am raising it already. But still seems weird why it would move data onto almost full OSDs, see the data distribution, it's horrible, ranges from 60 to almost 90% of full. PGs are not equally distributed otherwise it'd be a PG size issue.


Thanks

Am Sonntag, 27. Oktober 2019, 20:33:11 OEZ hat Wido den Hollander <wido@xxxxxxxx> Folgendes geschrieben:




On 10/26/19 8:01 AM, Philippe D'Anjou wrote:
> V14.2.4
> So, this is not new, this happens every time there is a rebalance, now
> because of raising PGs. PG balancer is disabled because I thought it was
> the reason but apparently it's not, but it ain't helping either.
>
> Ceph is totally borged, it's only moving data on nearfull OSDs causing
> issues. See this after PG raise.
>
>     health: HEALTH_WARN
>             3 nearfull osd(s)
>             2 pool(s) nearfull
>
>
> 08:44 am
>
> 92   hdd  9.09470  1.00000 9.1 TiB  7.5 TiB  7.5 TiB  48 KiB  19 GiB 1.6
> TiB 82.79 1.25  39     up
> 71   hdd  9.09470  1.00000 9.1 TiB  7.6 TiB  7.6 TiB  88 KiB  20 GiB 1.4
> TiB 84.09 1.27  38     up
> 21   hdd  9.09470  1.00000 9.1 TiB  7.6 TiB  7.6 TiB  60 KiB  20 GiB 1.5
> TiB 84.05 1.27  36     up
>  
>  
> 08:54 am
>
> 92   hdd  9.09470  1.00000 9.1 TiB  7.5 TiB  7.5 TiB  48 KiB  19 GiB 1.6
> TiB 82.81 1.25  39     up
> 71   hdd  9.09470  1.00000 9.1 TiB  7.7 TiB  7.6 TiB  88 KiB  20 GiB 1.4
> TiB 84.14 1.27  38     up
> 21   hdd  9.09470  1.00000 9.1 TiB  7.6 TiB  7.6 TiB  60 KiB  20 GiB 1.4
> TiB 84.10 1.27  36     up
>    
>    
>    
>  14   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  76 KiB  17 GiB
> 2.6 TiB 71.33 1.07  32     up
>  19   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  52 KiB  17 GiB
> 2.8 TiB 68.81 1.04  30     up
>  22   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  92 KiB  17 GiB
> 2.8 TiB 68.90 1.04  32     up
>  25   hdd  9.09470  1.00000 9.1 TiB  6.2 TiB  6.2 TiB 219 KiB  17 GiB
> 2.9 TiB 68.11 1.03  31     up
>  30   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  20 KiB  17 GiB
> 2.6 TiB 71.41 1.08  33     up
>  33   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  40 KiB  17 GiB
> 2.6 TiB 71.30 1.07  32     up
>  34   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  36 KiB  17 GiB
> 2.6 TiB 71.33 1.07  30     up
>  35   hdd  9.09470  1.00000 9.1 TiB  6.6 TiB  6.6 TiB 124 KiB  17 GiB
> 2.5 TiB 72.61 1.09  32     up
>  12   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  24 KiB  18 GiB
> 2.4 TiB 73.84 1.11  32     up
>  16   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.4 TiB  96 KiB  17 GiB
> 2.6 TiB 71.08 1.07  29     up
>  17   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  60 KiB  17 GiB
> 2.6 TiB 71.41 1.08  31     up
>  20   hdd  9.09470  1.00000 9.1 TiB  6.2 TiB  6.2 TiB  92 KiB  17 GiB
> 2.9 TiB 68.57 1.03  28     up
>  23   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  36 KiB  17 GiB
> 2.6 TiB 71.37 1.08  29     up
>  26   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.4 TiB  84 KiB  17 GiB
> 2.7 TiB 70.02 1.06  30     up
>  28   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.4 TiB  28 KiB  17 GiB
> 2.7 TiB 70.11 1.06  30     up
>  31   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  56 KiB  17 GiB
> 2.6 TiB 71.26 1.07  32     up
>  13   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  24 KiB  18 GiB
> 2.4 TiB 73.84 1.11  31     up
>  15   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  44 KiB  17 GiB
> 2.6 TiB 71.35 1.08  29     up
>  18   hdd  9.09470  1.00000 9.1 TiB  5.8 TiB  5.8 TiB  76 KiB  16 GiB
> 3.3 TiB 63.70 0.96  26     up
>  21   hdd  9.09470  1.00000 9.1 TiB  7.6 TiB  7.6 TiB  60 KiB  20 GiB
> 1.4 TiB 84.10 1.27  36     up
>  24   hdd  9.09470  1.00000 9.1 TiB  5.8 TiB  5.8 TiB  64 KiB  15 GiB
> 3.3 TiB 63.67 0.96  28     up
>  27   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  48 KiB  17 GiB
> 3.1 TiB 66.03 1.00  28     up
>  29   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.3 TiB  28 KiB  18 GiB
> 2.7 TiB 69.93 1.05  34     up
>  32   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  20 KiB  17 GiB
> 3.1 TiB 66.20 1.00  28     up
>  37   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.4 TiB  32 KiB  18 GiB
> 2.7 TiB 70.59 1.06  31     up
>  39   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.4 TiB  32 KiB  19 GiB
> 2.7 TiB 70.50 1.06  29     up
>  41   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  52 KiB  17 GiB
> 2.8 TiB 68.79 1.04  30     up
>  43   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  48 KiB  17 GiB
> 2.8 TiB 68.84 1.04  28     up
>  45   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  80 KiB  18 GiB
> 2.4 TiB 74.02 1.12  33     up
>  46   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  36 KiB  18 GiB
> 2.4 TiB 73.88 1.11  30     up
>  48   hdd  9.09470  1.00000 9.1 TiB  6.6 TiB  6.6 TiB 101 KiB  17 GiB
> 2.5 TiB 72.57 1.09  31     up
>  50   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  96 KiB  17 GiB
> 2.8 TiB 68.86 1.04  31     up
>  36   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  60 KiB  18 GiB
> 2.4 TiB 73.82 1.11  32     up
>  38   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.4 TiB  76 KiB  17 GiB
> 2.7 TiB 70.62 1.06  33     up
>  40   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB   8 KiB  18 GiB
> 2.4 TiB 73.89 1.11  33     up
>  42   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB 120 KiB  17 GiB
> 2.8 TiB 68.82 1.04  28     up
>  44   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  68 KiB  17 GiB
> 2.6 TiB 71.39 1.08  29     up
>  47   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.4 TiB  68 KiB  17 GiB
> 2.7 TiB 70.06 1.06  29     up
>  49   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  40 KiB  16 GiB
> 3.1 TiB 66.25 1.00  28     up
>  51   hdd  9.09470  1.00000 9.1 TiB  5.8 TiB  5.8 TiB  60 KiB  15 GiB
> 3.3 TiB 63.73 0.96  27     up
>  52   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.4 TiB 200 KiB  17 GiB
> 2.6 TiB 71.09 1.07  33     up
>  53   hdd  9.09470  0.79999 9.1 TiB  7.2 TiB  7.2 TiB  48 KiB  19 GiB
> 1.9 TiB 79.03 1.19  33     up
>  54   hdd  9.09470  1.00000 9.1 TiB  5.6 TiB  5.5 TiB  44 KiB  15 GiB
> 3.5 TiB 61.14 0.92  25     up
>  55   hdd  9.09470  1.00000 9.1 TiB  5.8 TiB  5.8 TiB  36 KiB  15 GiB
> 3.3 TiB 63.72 0.96  26     up
>  56   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  49 KiB  18 GiB
> 2.4 TiB 73.91 1.11  31     up
>  57   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  72 KiB  17 GiB
> 2.8 TiB 68.88 1.04  29     up
>  58   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  20 KiB  18 GiB
> 2.6 TiB 71.40 1.08  30     up
>  59   hdd  9.09470  1.00000 9.1 TiB  6.1 TiB  6.1 TiB  60 KiB  16 GiB
> 3.0 TiB 67.51 1.02  29     up
>  60   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  32 KiB  16 GiB
> 3.1 TiB 66.14 1.00  26     up
>  61   hdd  9.09470  1.00000 9.1 TiB  6.2 TiB  6.2 TiB  32 KiB  17 GiB
> 2.9 TiB 68.59 1.03  32     up
>  62   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.4 TiB  80 KiB  17 GiB
> 2.7 TiB 70.06 1.06  29     up
>  63   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  76 KiB  18 GiB
> 2.4 TiB 73.85 1.11  33     up
>  64   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  44 KiB  17 GiB
> 2.8 TiB 68.80 1.04  31     up
>  65   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.4 TiB  68 KiB  17 GiB
> 2.7 TiB 70.10 1.06  32     up
>  66   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  52 KiB  17 GiB
> 2.8 TiB 68.77 1.04  30     up
>  67   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB 112 KiB  17 GiB
> 2.6 TiB 71.36 1.08  31     up
>  84   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  60 KiB  17 GiB
> 2.8 TiB 68.84 1.04  30     up
>  85   hdd  9.09470  1.00000 9.1 TiB  7.0 TiB  6.9 TiB  48 KiB  19 GiB
> 2.1 TiB 76.46 1.15  32     up
>  86   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  76 KiB  16 GiB
> 2.8 TiB 68.80 1.04  29     up
>  87   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  60 KiB  16 GiB
> 3.1 TiB 66.28 1.00  27     up
>  88   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  5.9 TiB  28 KiB  17 GiB
> 3.1 TiB 65.51 0.99  27     up
>  89   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.3 TiB 116 KiB  17 GiB
> 2.8 TiB 68.90 1.04  28     up
>  90   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  56 KiB  16 GiB
> 3.1 TiB 66.29 1.00  28     up
>  91   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  16 KiB  16 GiB
> 3.1 TiB 66.24 1.00  30     up
>  76   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  76 KiB  17 GiB
> 2.8 TiB 68.78 1.04  27     up
>  77   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  44 KiB  17 GiB
> 2.6 TiB 71.39 1.08  29     up
>  78   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  40 KiB  17 GiB
> 2.6 TiB 71.38 1.08  32     up
>  79   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  56 KiB  17 GiB
> 2.6 TiB 71.29 1.07  33     up
>  80   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  32 KiB  18 GiB
> 2.6 TiB 71.36 1.08  30     up
>  81   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  44 KiB  17 GiB
> 2.6 TiB 71.36 1.08  32     up
>  82   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  36 KiB  17 GiB
> 2.8 TiB 68.82 1.04  30     up
>  83   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.4 TiB  36 KiB  17 GiB
> 2.7 TiB 70.07 1.06  29     up
>  68   hdd  9.09470  1.00000 9.1 TiB  6.4 TiB  6.4 TiB  20 KiB  17 GiB
> 2.7 TiB 70.06 1.06  31     up
>  69   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  16 KiB  18 GiB
> 2.4 TiB 73.96 1.11  32     up
>  70   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  64 KiB  17 GiB
> 2.8 TiB 68.84 1.04  29     up
>  71   hdd  9.09470  1.00000 9.1 TiB  7.7 TiB  7.6 TiB  88 KiB  20 GiB
> 1.4 TiB 84.14 1.27  38     up
>  72   hdd  9.09470  1.00000 9.1 TiB  6.1 TiB  6.1 TiB  48 KiB  16 GiB
> 3.0 TiB 67.55 1.02  28     up
>  73   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  80 KiB  16 GiB
> 3.1 TiB 66.20 1.00  30     up
>  74   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  32 KiB  17 GiB
> 2.6 TiB 71.25 1.07  31     up
>  75   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  44 KiB  16 GiB
> 3.1 TiB 66.17 1.00  26     up
> 121   hdd  9.09470  1.00000 9.1 TiB  5.6 TiB  5.6 TiB  76 KiB  15 GiB
> 3.5 TiB 61.18 0.92  28     up
> 123   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  72 KiB  17 GiB
> 2.6 TiB 71.47 1.08  29     up
> 124   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  76 KiB  17 GiB
> 2.6 TiB 71.29 1.07  31     up
> 126   hdd  9.09470  1.00000 9.1 TiB  6.2 TiB  6.2 TiB  64 KiB  16 GiB
> 2.9 TiB 68.65 1.03  29     up
> 128   hdd  9.09470  1.00000 9.1 TiB  6.6 TiB  6.6 TiB 104 KiB  17 GiB
> 2.5 TiB 72.65 1.09  32     up
> 130   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  96 KiB  16 GiB
> 3.1 TiB 66.33 1.00  28     up
> 132   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  84 KiB  17 GiB
> 2.4 TiB 73.99 1.11  31     up
> 134   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.6 TiB  48 KiB  18 GiB
> 2.4 TiB 73.22 1.10  30     up
> 112   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  64 KiB  18 GiB
> 2.4 TiB 73.86 1.11  31     up
> 113   hdd  9.09470  1.00000 9.1 TiB  5.8 TiB  5.8 TiB 104 KiB  15 GiB
> 3.3 TiB 63.77 0.96  28     up
> 114   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  48 KiB  16 GiB
> 3.1 TiB 66.29 1.00  28     up
> 115   hdd  9.09470  0.89999 9.1 TiB  6.7 TiB  6.7 TiB  40 KiB  18 GiB
> 2.4 TiB 73.70 1.11  29     up
> 116   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB   4 KiB  17 GiB
> 2.6 TiB 71.44 1.08  29     up
> 117   hdd  9.09470  0.89999 9.1 TiB  6.7 TiB  6.7 TiB 120 KiB  18 GiB
> 2.4 TiB 73.90 1.11  29     up
> 118   hdd  9.09470  1.00000 9.1 TiB  5.7 TiB  5.7 TiB  16 KiB  15 GiB
> 3.4 TiB 62.88 0.95  25     up
> 119   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB 108 KiB  16 GiB
> 2.8 TiB 68.83 1.04  30     up
> 120   hdd  9.09470  1.00000 9.1 TiB  6.0 TiB  5.9 TiB  68 KiB  16 GiB
> 3.1 TiB 65.48 0.99  29     up
> 122   hdd  9.09470  1.00000 9.1 TiB  6.6 TiB  6.6 TiB 100 KiB  17 GiB
> 2.5 TiB 72.63 1.09  31     up
> 125   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  64 KiB  17 GiB
> 2.4 TiB 73.85 1.11  32     up
> 127   hdd  9.09470  1.00000 9.1 TiB  6.6 TiB  6.6 TiB 171 KiB  17 GiB
> 2.5 TiB 72.58 1.09  34     up
> 129   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  48 KiB  17 GiB
> 2.6 TiB 71.39 1.08  30     up
> 131   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.5 TiB  68 KiB  17 GiB
> 2.6 TiB 71.38 1.08  29     up
> 133   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  28 KiB  17 GiB
> 2.4 TiB 73.97 1.11  31     up
> 135   hdd  9.09470  1.00000 9.1 TiB  6.6 TiB  6.6 TiB 104 KiB  17 GiB
> 2.5 TiB 72.58 1.09  30     up
>  92   hdd  9.09470  1.00000 9.1 TiB  7.5 TiB  7.5 TiB  48 KiB  19 GiB
> 1.6 TiB 82.81 1.25  39     up
>  93   hdd  9.09470  1.00000 9.1 TiB  6.3 TiB  6.2 TiB  88 KiB  16 GiB
> 2.8 TiB 68.83 1.04  30     up
>  94   hdd  9.09470  1.00000 9.1 TiB  4.9 TiB  4.9 TiB  32 KiB  13 GiB
> 4.2 TiB 53.51 0.81  25     up
>  95   hdd  8.00000  1.00000 9.1 TiB  6.0 TiB  6.0 TiB  60 KiB  16 GiB
> 3.1 TiB 66.28 1.00  27     up
>  96   hdd  9.09470  1.00000 9.1 TiB  5.8 TiB  5.8 TiB  76 KiB  15 GiB
> 3.3 TiB 63.72 0.96  27     up
>  97   hdd  9.09470  1.00000 9.1 TiB  5.8 TiB  5.8 TiB  68 KiB  15 GiB
> 3.3 TiB 63.61 0.96  27     up
>  98   hdd  9.09470  1.00000 9.1 TiB  6.7 TiB  6.7 TiB  24 KiB  17 GiB
> 2.4 TiB 73.82 1.11  29     up
>  99   hdd  9.09470  1.00000 9.1 TiB  6.5 TiB  6.4 TiB  76 KiB  17 GiB
> 2.6 TiB 71.09 1.07  33     up
>
>
> I really need some help with this, it's becoming unbearable to work
> with, I can't attend 24/7 to manually rebalance, this is not how its
> supposed to be. This is a Nautilus only issue

This is not an issue, but you seem to be very low on PGs per OSD.

This causes CRUSH not to be able to balance in a very optimal way.

You have ~30 PGs per OSD on 10TB large OSDs. Try increasing this to 100
~ 150 PGs or even ~200 per OSD. This will allow for a better data
distribution.

Wido


>
> Thanks

>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux