I was following the pg autoscaler recommendations and I did not get a recommendation to raise the PGs there.
I'll try that, I am raising it already. But still seems weird why it would move data onto almost full OSDs, see the data distribution, it's horrible, ranges from 60 to almost 90% of full. PGs are not equally distributed otherwise it'd be a PG size issue.
Thanks
Am Sonntag, 27. Oktober 2019, 20:33:11 OEZ hat Wido den Hollander <wido@xxxxxxxx> Folgendes geschrieben:
On 10/26/19 8:01 AM, Philippe D'Anjou wrote:
> V14.2.4
> So, this is not new, this happens every time there is a rebalance, now
> because of raising PGs. PG balancer is disabled because I thought it was
> the reason but apparently it's not, but it ain't helping either.
>
> Ceph is totally borged, it's only moving data on nearfull OSDs causing
> issues. See this after PG raise.
>
> health: HEALTH_WARN
> 3 nearfull osd(s)
> 2 pool(s) nearfull
>
>
> 08:44 am
>
> 92 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 48 KiB 19 GiB 1.6
> TiB 82.79 1.25 39 up
> 71 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 88 KiB 20 GiB 1.4
> TiB 84.09 1.27 38 up
> 21 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 60 KiB 20 GiB 1.5
> TiB 84.05 1.27 36 up
>
>
> 08:54 am
>
> 92 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 48 KiB 19 GiB 1.6
> TiB 82.81 1.25 39 up
> 71 hdd 9.09470 1.00000 9.1 TiB 7.7 TiB 7.6 TiB 88 KiB 20 GiB 1.4
> TiB 84.14 1.27 38 up
> 21 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 60 KiB 20 GiB 1.4
> TiB 84.10 1.27 36 up
>
>
>
> 14 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 76 KiB 17 GiB
> 2.6 TiB 71.33 1.07 32 up
> 19 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 52 KiB 17 GiB
> 2.8 TiB 68.81 1.04 30 up
> 22 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 92 KiB 17 GiB
> 2.8 TiB 68.90 1.04 32 up
> 25 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 219 KiB 17 GiB
> 2.9 TiB 68.11 1.03 31 up
> 30 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 20 KiB 17 GiB
> 2.6 TiB 71.41 1.08 33 up
> 33 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 40 KiB 17 GiB
> 2.6 TiB 71.30 1.07 32 up
> 34 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 36 KiB 17 GiB
> 2.6 TiB 71.33 1.07 30 up
> 35 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 124 KiB 17 GiB
> 2.5 TiB 72.61 1.09 32 up
> 12 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 24 KiB 18 GiB
> 2.4 TiB 73.84 1.11 32 up
> 16 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 96 KiB 17 GiB
> 2.6 TiB 71.08 1.07 29 up
> 17 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 60 KiB 17 GiB
> 2.6 TiB 71.41 1.08 31 up
> 20 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 92 KiB 17 GiB
> 2.9 TiB 68.57 1.03 28 up
> 23 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 36 KiB 17 GiB
> 2.6 TiB 71.37 1.08 29 up
> 26 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 84 KiB 17 GiB
> 2.7 TiB 70.02 1.06 30 up
> 28 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 28 KiB 17 GiB
> 2.7 TiB 70.11 1.06 30 up
> 31 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 56 KiB 17 GiB
> 2.6 TiB 71.26 1.07 32 up
> 13 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 24 KiB 18 GiB
> 2.4 TiB 73.84 1.11 31 up
> 15 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 44 KiB 17 GiB
> 2.6 TiB 71.35 1.08 29 up
> 18 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 76 KiB 16 GiB
> 3.3 TiB 63.70 0.96 26 up
> 21 hdd 9.09470 1.00000 9.1 TiB 7.6 TiB 7.6 TiB 60 KiB 20 GiB
> 1.4 TiB 84.10 1.27 36 up
> 24 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 64 KiB 15 GiB
> 3.3 TiB 63.67 0.96 28 up
> 27 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 48 KiB 17 GiB
> 3.1 TiB 66.03 1.00 28 up
> 29 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.3 TiB 28 KiB 18 GiB
> 2.7 TiB 69.93 1.05 34 up
> 32 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 20 KiB 17 GiB
> 3.1 TiB 66.20 1.00 28 up
> 37 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 32 KiB 18 GiB
> 2.7 TiB 70.59 1.06 31 up
> 39 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 32 KiB 19 GiB
> 2.7 TiB 70.50 1.06 29 up
> 41 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 52 KiB 17 GiB
> 2.8 TiB 68.79 1.04 30 up
> 43 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 48 KiB 17 GiB
> 2.8 TiB 68.84 1.04 28 up
> 45 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 80 KiB 18 GiB
> 2.4 TiB 74.02 1.12 33 up
> 46 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 36 KiB 18 GiB
> 2.4 TiB 73.88 1.11 30 up
> 48 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 101 KiB 17 GiB
> 2.5 TiB 72.57 1.09 31 up
> 50 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 96 KiB 17 GiB
> 2.8 TiB 68.86 1.04 31 up
> 36 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 60 KiB 18 GiB
> 2.4 TiB 73.82 1.11 32 up
> 38 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 76 KiB 17 GiB
> 2.7 TiB 70.62 1.06 33 up
> 40 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 8 KiB 18 GiB
> 2.4 TiB 73.89 1.11 33 up
> 42 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 120 KiB 17 GiB
> 2.8 TiB 68.82 1.04 28 up
> 44 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 68 KiB 17 GiB
> 2.6 TiB 71.39 1.08 29 up
> 47 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 68 KiB 17 GiB
> 2.7 TiB 70.06 1.06 29 up
> 49 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 40 KiB 16 GiB
> 3.1 TiB 66.25 1.00 28 up
> 51 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 60 KiB 15 GiB
> 3.3 TiB 63.73 0.96 27 up
> 52 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 200 KiB 17 GiB
> 2.6 TiB 71.09 1.07 33 up
> 53 hdd 9.09470 0.79999 9.1 TiB 7.2 TiB 7.2 TiB 48 KiB 19 GiB
> 1.9 TiB 79.03 1.19 33 up
> 54 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.5 TiB 44 KiB 15 GiB
> 3.5 TiB 61.14 0.92 25 up
> 55 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 36 KiB 15 GiB
> 3.3 TiB 63.72 0.96 26 up
> 56 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 49 KiB 18 GiB
> 2.4 TiB 73.91 1.11 31 up
> 57 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 72 KiB 17 GiB
> 2.8 TiB 68.88 1.04 29 up
> 58 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 20 KiB 18 GiB
> 2.6 TiB 71.40 1.08 30 up
> 59 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 60 KiB 16 GiB
> 3.0 TiB 67.51 1.02 29 up
> 60 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 32 KiB 16 GiB
> 3.1 TiB 66.14 1.00 26 up
> 61 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 32 KiB 17 GiB
> 2.9 TiB 68.59 1.03 32 up
> 62 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 80 KiB 17 GiB
> 2.7 TiB 70.06 1.06 29 up
> 63 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 76 KiB 18 GiB
> 2.4 TiB 73.85 1.11 33 up
> 64 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 44 KiB 17 GiB
> 2.8 TiB 68.80 1.04 31 up
> 65 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 68 KiB 17 GiB
> 2.7 TiB 70.10 1.06 32 up
> 66 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 52 KiB 17 GiB
> 2.8 TiB 68.77 1.04 30 up
> 67 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 112 KiB 17 GiB
> 2.6 TiB 71.36 1.08 31 up
> 84 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 60 KiB 17 GiB
> 2.8 TiB 68.84 1.04 30 up
> 85 hdd 9.09470 1.00000 9.1 TiB 7.0 TiB 6.9 TiB 48 KiB 19 GiB
> 2.1 TiB 76.46 1.15 32 up
> 86 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 76 KiB 16 GiB
> 2.8 TiB 68.80 1.04 29 up
> 87 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 60 KiB 16 GiB
> 3.1 TiB 66.28 1.00 27 up
> 88 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 5.9 TiB 28 KiB 17 GiB
> 3.1 TiB 65.51 0.99 27 up
> 89 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 116 KiB 17 GiB
> 2.8 TiB 68.90 1.04 28 up
> 90 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 56 KiB 16 GiB
> 3.1 TiB 66.29 1.00 28 up
> 91 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 16 KiB 16 GiB
> 3.1 TiB 66.24 1.00 30 up
> 76 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 76 KiB 17 GiB
> 2.8 TiB 68.78 1.04 27 up
> 77 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 44 KiB 17 GiB
> 2.6 TiB 71.39 1.08 29 up
> 78 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 40 KiB 17 GiB
> 2.6 TiB 71.38 1.08 32 up
> 79 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 56 KiB 17 GiB
> 2.6 TiB 71.29 1.07 33 up
> 80 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 32 KiB 18 GiB
> 2.6 TiB 71.36 1.08 30 up
> 81 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 44 KiB 17 GiB
> 2.6 TiB 71.36 1.08 32 up
> 82 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 36 KiB 17 GiB
> 2.8 TiB 68.82 1.04 30 up
> 83 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 36 KiB 17 GiB
> 2.7 TiB 70.07 1.06 29 up
> 68 hdd 9.09470 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 20 KiB 17 GiB
> 2.7 TiB 70.06 1.06 31 up
> 69 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 16 KiB 18 GiB
> 2.4 TiB 73.96 1.11 32 up
> 70 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 64 KiB 17 GiB
> 2.8 TiB 68.84 1.04 29 up
> 71 hdd 9.09470 1.00000 9.1 TiB 7.7 TiB 7.6 TiB 88 KiB 20 GiB
> 1.4 TiB 84.14 1.27 38 up
> 72 hdd 9.09470 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 48 KiB 16 GiB
> 3.0 TiB 67.55 1.02 28 up
> 73 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 80 KiB 16 GiB
> 3.1 TiB 66.20 1.00 30 up
> 74 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 32 KiB 17 GiB
> 2.6 TiB 71.25 1.07 31 up
> 75 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 44 KiB 16 GiB
> 3.1 TiB 66.17 1.00 26 up
> 121 hdd 9.09470 1.00000 9.1 TiB 5.6 TiB 5.6 TiB 76 KiB 15 GiB
> 3.5 TiB 61.18 0.92 28 up
> 123 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 72 KiB 17 GiB
> 2.6 TiB 71.47 1.08 29 up
> 124 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 76 KiB 17 GiB
> 2.6 TiB 71.29 1.07 31 up
> 126 hdd 9.09470 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 64 KiB 16 GiB
> 2.9 TiB 68.65 1.03 29 up
> 128 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 104 KiB 17 GiB
> 2.5 TiB 72.65 1.09 32 up
> 130 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 96 KiB 16 GiB
> 3.1 TiB 66.33 1.00 28 up
> 132 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 84 KiB 17 GiB
> 2.4 TiB 73.99 1.11 31 up
> 134 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.6 TiB 48 KiB 18 GiB
> 2.4 TiB 73.22 1.10 30 up
> 112 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 64 KiB 18 GiB
> 2.4 TiB 73.86 1.11 31 up
> 113 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 104 KiB 15 GiB
> 3.3 TiB 63.77 0.96 28 up
> 114 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 48 KiB 16 GiB
> 3.1 TiB 66.29 1.00 28 up
> 115 hdd 9.09470 0.89999 9.1 TiB 6.7 TiB 6.7 TiB 40 KiB 18 GiB
> 2.4 TiB 73.70 1.11 29 up
> 116 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 4 KiB 17 GiB
> 2.6 TiB 71.44 1.08 29 up
> 117 hdd 9.09470 0.89999 9.1 TiB 6.7 TiB 6.7 TiB 120 KiB 18 GiB
> 2.4 TiB 73.90 1.11 29 up
> 118 hdd 9.09470 1.00000 9.1 TiB 5.7 TiB 5.7 TiB 16 KiB 15 GiB
> 3.4 TiB 62.88 0.95 25 up
> 119 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 108 KiB 16 GiB
> 2.8 TiB 68.83 1.04 30 up
> 120 hdd 9.09470 1.00000 9.1 TiB 6.0 TiB 5.9 TiB 68 KiB 16 GiB
> 3.1 TiB 65.48 0.99 29 up
> 122 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 100 KiB 17 GiB
> 2.5 TiB 72.63 1.09 31 up
> 125 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 64 KiB 17 GiB
> 2.4 TiB 73.85 1.11 32 up
> 127 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 171 KiB 17 GiB
> 2.5 TiB 72.58 1.09 34 up
> 129 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 48 KiB 17 GiB
> 2.6 TiB 71.39 1.08 30 up
> 131 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 68 KiB 17 GiB
> 2.6 TiB 71.38 1.08 29 up
> 133 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 28 KiB 17 GiB
> 2.4 TiB 73.97 1.11 31 up
> 135 hdd 9.09470 1.00000 9.1 TiB 6.6 TiB 6.6 TiB 104 KiB 17 GiB
> 2.5 TiB 72.58 1.09 30 up
> 92 hdd 9.09470 1.00000 9.1 TiB 7.5 TiB 7.5 TiB 48 KiB 19 GiB
> 1.6 TiB 82.81 1.25 39 up
> 93 hdd 9.09470 1.00000 9.1 TiB 6.3 TiB 6.2 TiB 88 KiB 16 GiB
> 2.8 TiB 68.83 1.04 30 up
> 94 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 32 KiB 13 GiB
> 4.2 TiB 53.51 0.81 25 up
> 95 hdd 8.00000 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 60 KiB 16 GiB
> 3.1 TiB 66.28 1.00 27 up
> 96 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 76 KiB 15 GiB
> 3.3 TiB 63.72 0.96 27 up
> 97 hdd 9.09470 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 68 KiB 15 GiB
> 3.3 TiB 63.61 0.96 27 up
> 98 hdd 9.09470 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 24 KiB 17 GiB
> 2.4 TiB 73.82 1.11 29 up
> 99 hdd 9.09470 1.00000 9.1 TiB 6.5 TiB 6.4 TiB 76 KiB 17 GiB
> 2.6 TiB 71.09 1.07 33 up
>
>
> I really need some help with this, it's becoming unbearable to work
> with, I can't attend 24/7 to manually rebalance, this is not how its
> supposed to be. This is a Nautilus only issue
This is not an issue, but you seem to be very low on PGs per OSD.
This causes CRUSH not to be able to balance in a very optimal way.
You have ~30 PGs per OSD on 10TB large OSDs. Try increasing this to 100
~ 150 PGs or even ~200 per OSD. This will allow for a better data
distribution.
Wido
>
> Thanks
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
