Issues with data distribution on Nautilus / weird filling behavior

["Philippe D'Anjou" <danjou.philippe@xxxxxxxx>]

Hi,

this is a fresh Nautilus cluster, but there is a second old one that was upgraded from Luminous to Nautilus, both experience the same symptoms.

First of all the data distribution on the OSDs is very bad. Now that could be due to low PGs although I get no recommendation to raise the PG number so far.

 14   hdd  9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  68 KiB  15 GiB 2.9 TiB 67.83 1.07  32     up
 19   hdd  9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  56 KiB  16 GiB 2.4 TiB 73.56 1.16  35     up
 22   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  88 KiB  15 GiB 2.8 TiB 69.39 1.09  32     up
 25   hdd  9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  88 KiB  16 GiB 3.1 TiB 66.01 1.04  31     up
 30   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  84 KiB  16 GiB 2.8 TiB 69.36 1.09  33     up
 33   hdd  9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  60 KiB  15 GiB 3.1 TiB 65.84 1.03  31     up
 34   hdd  9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB 124 KiB  14 GiB 3.3 TiB 63.84 1.00  29     up
 35   hdd  9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB  32 KiB  15 GiB 3.1 TiB 65.82 1.03  31     up
 12   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  44 KiB  15 GiB 3.0 TiB 67.32 1.06  31     up
 16   hdd  9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  72 KiB  17 GiB 2.4 TiB 73.52 1.16  35     up
 17   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  80 KiB  15 GiB 3.0 TiB 67.24 1.06  32     up
 20   hdd  9.09470  1.00000 9.1 TiB 6.8 TiB 6.8 TiB  64 KiB  17 GiB 2.3 TiB 74.45 1.17  35     up
 23   hdd  9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  80 KiB  16 GiB 2.6 TiB 71.46 1.12  34     up
 26   hdd  9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  68 KiB  16 GiB 2.7 TiB 70.45 1.11  33     up
 28   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.0 TiB  64 KiB  15 GiB 3.0 TiB 66.52 1.05  31     up
 31   hdd  9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  52 KiB  16 GiB 2.5 TiB 72.21 1.13  34     up
 13   hdd  9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  68 KiB  16 GiB 2.4 TiB 73.56 1.16  35     up
 15   hdd  9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  44 KiB  17 GiB 2.4 TiB 73.58 1.16  35     up
 18   hdd  9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  56 KiB  16 GiB 2.6 TiB 71.46 1.12  34     up
 21   hdd  8.00000  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  84 KiB  15 GiB 3.2 TiB 65.14 1.02  31     up
 24   hdd  9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  76 KiB  16 GiB 2.5 TiB 72.21 1.13  34     up
 27   hdd  9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  64 KiB  16 GiB 2.6 TiB 71.42 1.12  34     up
 29   hdd  9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  80 KiB  15 GiB 2.9 TiB 68.16 1.07  32     up
 32   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB 132 KiB  15 GiB 3.0 TiB 66.91 1.05  31     up
 37   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  16 KiB  17 GiB 2.8 TiB 69.38 1.09  33     up
 39   hdd  9.09470  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  92 KiB  16 GiB 2.7 TiB 70.44 1.11  33     up
 41   hdd  9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB 108 KiB  15 GiB 2.9 TiB 67.95 1.07  32     up
 43   hdd  9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  24 KiB  15 GiB 3.2 TiB 65.20 1.02  31     up
 45   hdd  9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  72 KiB  15 GiB 3.2 TiB 65.35 1.03  31     up
 46   hdd  9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  60 KiB  15 GiB 2.9 TiB 68.08 1.07  32     up
 48   hdd  9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  64 KiB  15 GiB 2.9 TiB 67.96 1.07  32     up
 50   hdd  9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB  48 KiB  15 GiB 3.4 TiB 63.06 0.99  30     up
 36   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB 116 KiB  15 GiB 3.0 TiB 67.45 1.06  32     up
 38   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  80 KiB  16 GiB 2.8 TiB 69.36 1.09  33     up
 40   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  84 KiB  15 GiB 3.0 TiB 67.31 1.06  32     up
 42   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.0 TiB 104 KiB  15 GiB 3.0 TiB 66.59 1.05  31     up
 44   hdd  9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  68 KiB  15 GiB 3.2 TiB 65.12 1.02  31     up
 47   hdd  9.09470  1.00000 9.1 TiB 5.9 TiB 5.9 TiB  88 KiB  15 GiB 3.2 TiB 65.09 1.02  31     up
 49   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB  15 GiB 3.0 TiB 67.19 1.06  32     up
 51   hdd  9.09470  1.00000 9.1 TiB 6.2 TiB 6.1 TiB  40 KiB  15 GiB 2.9 TiB 67.75 1.06  32     up
 52   hdd  9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  80 KiB  15 GiB 3.3 TiB 63.91 1.00  30     up
 53   hdd  7.00000  1.00000 9.1 TiB 7.3 TiB 7.2 TiB  88 KiB  18 GiB 1.8 TiB 79.84 1.25  38     up
 54   hdd  9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  96 KiB  14 GiB 3.3 TiB 63.67 1.00  30     up
 55   hdd  9.09470  1.00000 9.1 TiB 5.3 TiB 5.3 TiB  48 KiB  13 GiB 3.8 TiB 58.20 0.91  27     up
 56   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  64 KiB  16 GiB 2.8 TiB 69.35 1.09  33     up
 57   hdd  6.00000  1.00000 9.1 TiB 4.2 TiB 4.2 TiB  36 KiB  11 GiB 4.9 TiB 46.27 0.73  22     up
 58   hdd  9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB 104 KiB  15 GiB 2.9 TiB 68.20 1.07  32     up
 59   hdd  9.09470  1.00000 9.1 TiB 6.4 TiB 6.3 TiB  92 KiB  16 GiB 2.7 TiB 69.99 1.10  33     up
 60   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB 108 KiB  15 GiB 3.0 TiB 67.18 1.06  32     up
 61   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB 112 KiB  16 GiB 2.8 TiB 69.34 1.09  33     up
 62   hdd  9.09470  1.00000 9.1 TiB 3.9 TiB 3.9 TiB  32 KiB  10 GiB 5.2 TiB 42.98 0.68  20     up
 63   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB 108 KiB  16 GiB 2.8 TiB 69.62 1.09  33     up
 64   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  64 KiB  16 GiB 3.0 TiB 67.26 1.06  32     up
 65   hdd  9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  36 KiB  17 GiB 2.6 TiB 71.46 1.12  34     up
 66   hdd  9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  80 KiB  16 GiB 2.6 TiB 71.37 1.12  34     up
 67   hdd  9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  48 KiB  15 GiB 2.9 TiB 68.60 1.08  32     up
 84   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  68 KiB  16 GiB 2.8 TiB 69.40 1.09  33     up
 85   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.2 TiB 116 KiB  15 GiB 2.8 TiB 68.80 1.08  32     up
 86   hdd  9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  48 KiB  15 GiB 3.2 TiB 64.30 1.01  30     up
 87   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB 125 KiB  15 GiB 3.0 TiB 67.23 1.06  32     up
 88   hdd  9.09470  1.00000 9.1 TiB 5.9 TiB 5.8 TiB  68 KiB  15 GiB 3.2 TiB 64.41 1.01  30     up
 89   hdd  9.09470  1.00000 9.1 TiB 5.7 TiB 5.7 TiB 100 KiB  14 GiB 3.4 TiB 63.11 0.99  30     up
 90   hdd  9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  96 KiB  15 GiB 2.9 TiB 68.02 1.07  32     up
 91   hdd  9.09470  1.00000 9.1 TiB 7.0 TiB 6.9 TiB  68 KiB  17 GiB 2.1 TiB 76.43 1.20  36     up
 76   hdd  9.09470  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  84 KiB  16 GiB 2.5 TiB 72.40 1.14  34     up
 77   hdd  9.09470  1.00000 9.1 TiB 6.6 TiB 6.5 TiB  56 KiB  17 GiB 2.5 TiB 72.06 1.13  34     up
 78   hdd  9.09470  1.00000 9.1 TiB 6.4 TiB 6.3 TiB 132 KiB  16 GiB 2.7 TiB 69.98 1.10  33     up
 79   hdd  9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  32 KiB  18 GiB 2.4 TiB 73.50 1.16  35     up
 80   hdd  9.09470  1.00000 9.1 TiB 5.4 TiB 5.4 TiB  76 KiB  14 GiB 3.7 TiB 59.80 0.94  28     up
 81   hdd  9.09470  1.00000 9.1 TiB 6.2 TiB 6.2 TiB  44 KiB  15 GiB 2.9 TiB 68.20 1.07  32     up
 82   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  68 KiB  16 GiB 2.8 TiB 69.34 1.09  33     up
 83   hdd  9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB 124 KiB  15 GiB 3.1 TiB 65.92 1.04  31     up
 68   hdd  9.09470  1.00000 9.1 TiB 3.6 TiB 3.6 TiB 104 KiB 8.8 GiB 5.5 TiB 39.45 0.62  15     up
 69   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  56 KiB  15 GiB 3.0 TiB 67.43 1.06  31     up
 70   hdd  9.09470  1.00000 9.1 TiB 7.0 TiB 7.0 TiB 100 KiB  17 GiB 2.1 TiB 77.39 1.22  36     up
 71   hdd  7.50000  1.00000 9.1 TiB 7.3 TiB 7.3 TiB  52 KiB  19 GiB 1.8 TiB 80.73 1.27  38     up
 72   hdd  9.09470  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  52 KiB  16 GiB 2.8 TiB 69.28 1.09  33     up
 73   hdd  9.09470  1.00000 9.1 TiB 7.6 TiB 7.5 TiB 100 KiB  18 GiB 1.5 TiB 83.02 1.30  39     up
 74   hdd  9.09470  1.00000 9.1 TiB 6.1 TiB 6.1 TiB  72 KiB  15 GiB 3.0 TiB 66.91 1.05  31     up
 75   hdd  9.09470  1.00000 9.1 TiB 5.9 TiB 5.8 TiB  92 KiB  14 GiB 3.2 TiB 64.41 1.01  29     up
112   hdd  9.09470  1.00000 9.1 TiB 5.8 TiB 5.8 TiB  36 KiB  14 GiB 3.2 TiB 64.28 1.01  30     up
113   hdd  9.09470  1.00000 9.1 TiB 7.0 TiB 6.9 TiB  64 KiB  17 GiB 2.1 TiB 76.47 1.20  36     up
114   hdd  9.09470  1.00000 9.1 TiB 6.8 TiB 6.8 TiB 128 KiB  16 GiB 2.3 TiB 74.86 1.18  35     up
115   hdd  7.00000  1.00000 9.1 TiB 6.4 TiB 6.4 TiB  96 KiB  16 GiB 2.7 TiB 70.35 1.11  33     up
116   hdd  9.09470  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  84 KiB  16 GiB 2.4 TiB 73.84 1.16  35     up
117   hdd  5.00000  1.00000 9.1 TiB 7.7 TiB 7.7 TiB  64 KiB  18 GiB 1.4 TiB 84.55 1.33  40     up
118   hdd  9.09470  1.00000 9.1 TiB 6.0 TiB 6.0 TiB 116 KiB  14 GiB 3.1 TiB 65.59 1.03  30     up
119   hdd  9.09470  1.00000 9.1 TiB 7.1 TiB 7.0 TiB 108 KiB  17 GiB 2.0 TiB 77.52 1.22  36     up
 92   hdd  9.09470  1.00000 9.1 TiB 6.5 TiB 6.5 TiB  68 KiB  16 GiB 2.6 TiB 71.51 1.12  34     up
 93   hdd 10.00000  1.00000 9.1 TiB 6.3 TiB 6.3 TiB  60 KiB  15 GiB 2.8 TiB 69.58 1.09  32     up
 94   hdd 10.00000  1.00000 9.1 TiB 6.6 TiB 6.6 TiB  92 KiB  16 GiB 2.5 TiB 72.79 1.14  34     up
 95   hdd 10.00000  1.00000 9.1 TiB 7.1 TiB 7.1 TiB 100 KiB  17 GiB 2.0 TiB 77.71 1.22  37     up
 96   hdd 10.00000  1.00000 9.1 TiB 6.7 TiB 6.7 TiB  32 KiB  16 GiB 2.4 TiB 73.37 1.15  34     up
 97   hdd 10.00000  1.00000 9.1 TiB 6.9 TiB 6.9 TiB  76 KiB  17 GiB 2.2 TiB 76.35 1.20  36     up

STDDEV: 24.83

Second, and biggest issue is that whenever we add new storage to the cluster ceph begins to move data from the old OSDs to other old OSDs which causes them to fill up dangerously high. Then I have to either reweight them or kill them completely to avoid a full OSD.

The problem is that reweight is ignored too very often, even halfing the weight doesn't change anything, the OSD just fills up more and more during rebalance/recovery.

So in a nutshell: I fill up the tank in my car and when I leave the station my fuel gauge says its empty. That's what happens if we add new storage, starting a whack-a-mole game of reweight OSDs to avoid further issues.

Pretty much using all default values for everything, no special OSD configs.

I reported this but I was told to ask here.

Thanks for any hints.

ps. never happend under Luminous

PG autoscaler is on recommendation mode only

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com