Re: Unbalanced Cluster

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Josh,

Thanks for getting back to me so soon!

This cluster was upgraded from 13.x to 14.2.9 some time ago.  The entire 
cluster was installed at the 13.x time and was upgraded together so all 
OSDs should have the same formatting etc.

Below is pasted the ceph osd df tree output.

Sincerely

-Dave


# ceph osd df tree
ID  CLASS WEIGHT     REWEIGHT SIZE    RAW USE DATA    OMAP META    
AVAIL   %USE  VAR  PGS STATUS TYPE NAME
  -1       1773.46399        - 1.8 PiB 1.2 PiB 1.2 PiB 1.1 TiB  11 TiB 
570 TiB 68.71 1.00   -        root default
  -3        202.48462        - 202 TiB 135 TiB 131 TiB 122 GiB 1.2 TiB  
68 TiB 66.50 0.97   -            host trex-ceph1
   0   hdd   12.65529  1.00000  13 TiB 7.7 TiB 7.5 TiB 4.4 GiB  73 GiB 
5.0 TiB 60.73 0.88 186     up         osd.0
   1   hdd   12.65529  1.00000  13 TiB 8.3 TiB 8.1 TiB 7.2 GiB  77 GiB 
4.3 TiB 65.64 0.96 185     up         osd.1
   2   hdd   12.65529  1.00000  13 TiB 6.8 TiB 6.6 TiB 6.1 GiB  67 GiB 
5.9 TiB 53.51 0.78 183     up         osd.2
   3   hdd   12.65529  1.00000  13 TiB 8.6 TiB 8.3 TiB 7.2 GiB  76 GiB 
4.1 TiB 67.57 0.98 182     up         osd.3
   4   hdd   12.65529  1.00000  13 TiB 8.5 TiB 8.3 TiB 8.6 GiB  76 GiB 
4.2 TiB 67.19 0.98 178     up         osd.4
   5   hdd   12.65529  1.00000  13 TiB  10 TiB  10 TiB  13 GiB  86 GiB 
2.3 TiB 81.88 1.19 180     up         osd.5
   6   hdd   12.65529  1.00000  13 TiB 6.8 TiB 6.6 TiB  10 GiB  67 GiB 
5.9 TiB 53.47 0.78 183     up         osd.6
   7   hdd   12.65529  1.00000  13 TiB  10 TiB 9.9 TiB 8.7 GiB  86 GiB 
2.6 TiB 79.82 1.16 184     up         osd.7
   8   hdd   12.65529  1.00000  13 TiB 7.8 TiB 7.6 TiB 7.2 GiB  72 GiB 
4.9 TiB 61.60 0.90 181     up         osd.8
   9   hdd   12.65529  1.00000  13 TiB 7.7 TiB 7.5 TiB 8.4 GiB  73 GiB 
4.9 TiB 60.93 0.89 186     up         osd.9
  10   hdd   12.65529  1.00000  13 TiB 8.4 TiB 8.2 TiB 5.8 GiB  75 GiB 
4.3 TiB 66.30 0.96 179     up         osd.10
  11   hdd   12.65529  1.00000  13 TiB 7.3 TiB 7.1 TiB 6.0 GiB  69 GiB 
5.3 TiB 57.86 0.84 182     up         osd.11
  12   hdd   12.65529  1.00000  13 TiB 9.0 TiB 8.8 TiB  10 GiB  78 GiB 
3.6 TiB 71.41 1.04 183     up         osd.12
  13   hdd   12.65529  1.00000  13 TiB 7.8 TiB 7.6 TiB 2.8 GiB  74 GiB 
4.9 TiB 61.52 0.90 188     up         osd.13
  14   hdd   12.65529  1.00000  13 TiB  10 TiB  10 TiB 5.9 GiB  87 GiB 
2.3 TiB 81.61 1.19 185     up         osd.14
  15   hdd   12.65529  1.00000  13 TiB 9.2 TiB 9.0 TiB  10 GiB  81 GiB 
3.4 TiB 72.99 1.06 184     up         osd.15
  -5        197.82933        - 202 TiB 140 TiB 136 TiB 130 GiB 1.2 TiB  
63 TiB 68.97 1.00   -            host trex-ceph2
  16   hdd   12.65529  1.00000  13 TiB 7.6 TiB 7.4 TiB 9.9 GiB  73 GiB 
5.0 TiB 60.42 0.88 191     up         osd.16
  17   hdd   12.65529  1.00000  13 TiB 8.6 TiB 8.4 TiB  18 GiB  78 GiB 
4.0 TiB 68.29 0.99 191     up         osd.17
  18   hdd   12.65529  1.00000  13 TiB 8.5 TiB 8.3 TiB 8.7 GiB  79 GiB 
4.2 TiB 67.01 0.98 194     up         osd.18
  19   hdd   12.65529  1.00000  13 TiB 9.9 TiB 9.6 TiB 9.0 GiB  85 GiB 
2.8 TiB 78.02 1.14 188     up         osd.19
  20   hdd   12.65529  1.00000  13 TiB 8.7 TiB 8.5 TiB  10 GiB  79 GiB 
3.9 TiB 69.02 1.00 191     up         osd.20
  21   hdd   12.65529  0.79160  13 TiB 8.4 TiB 8.2 TiB 4.5 GiB  71 GiB 
4.2 TiB 66.56 0.97 139     up         osd.21
  22   hdd   12.65529  0.80670  13 TiB 9.9 TiB 9.7 TiB 2.7 GiB  83 GiB 
2.8 TiB 78.26 1.14 155     up         osd.22
  23   hdd   12.65529  1.00000  13 TiB 8.8 TiB 8.6 TiB  12 GiB  80 GiB 
3.9 TiB 69.54 1.01 194     up         osd.23
  24   hdd   12.65529  1.00000  13 TiB 8.0 TiB 7.8 TiB 7.3 GiB  76 GiB 
4.7 TiB 63.13 0.92 190     up         osd.24
  25   hdd    8.00000  0.42892  13 TiB  11 TiB  10 TiB 6.9 GiB  77 GiB 
2.0 TiB 83.82 1.22  86     up         osd.25
  26   hdd   12.65529  1.00000  13 TiB 7.2 TiB 7.0 TiB  10 GiB  70 GiB 
5.4 TiB 57.28 0.83 187     up         osd.26
  27   hdd   12.65529  1.00000  13 TiB  10 TiB 9.9 TiB 1.5 GiB  86 GiB 
2.5 TiB 79.90 1.16 182     up         osd.27
  28   hdd   12.65529  1.00000  13 TiB 9.2 TiB 9.0 TiB 8.8 GiB  81 GiB 
3.4 TiB 72.79 1.06 190     up         osd.28
  29   hdd   12.65529  1.00000  13 TiB 6.8 TiB 6.6 TiB 7.2 GiB  69 GiB 
5.9 TiB 53.53 0.78 194     up         osd.29
  30   hdd   12.65529  1.00000  13 TiB 9.0 TiB 8.8 TiB 4.5 GiB  80 GiB 
3.7 TiB 71.05 1.03 186     up         osd.30
  31   hdd   12.65529  1.00000  13 TiB 8.2 TiB 8.0 TiB 8.8 GiB  76 GiB 
4.4 TiB 64.92 0.94 191     up         osd.31
  -7        202.48462        - 202 TiB 134 TiB 131 TiB 146 GiB 1.2 TiB  
68 TiB 66.33 0.97   -            host trex-ceph3
  32   hdd   12.65529  1.00000  13 TiB 9.1 TiB 8.8 TiB  10 GiB  80 GiB 
3.6 TiB 71.63 1.04 182     up         osd.32
  33   hdd   12.65529  1.00000  13 TiB 8.9 TiB 8.7 TiB 4.3 GiB  80 GiB 
3.8 TiB 70.15 1.02 180     up         osd.33
  34   hdd   12.65529  1.00000  13 TiB 6.4 TiB 6.2 TiB  13 GiB  67 GiB 
6.3 TiB 50.43 0.73 183     up         osd.34
  35   hdd   12.65529  1.00000  13 TiB 8.0 TiB 7.8 TiB 7.1 GiB  75 GiB 
4.6 TiB 63.51 0.92 186     up         osd.35
  36   hdd   12.65529  1.00000  13 TiB 7.8 TiB 7.6 TiB  11 GiB  73 GiB 
4.9 TiB 61.41 0.89 183     up         osd.36
  37   hdd   12.65529  1.00000  13 TiB 8.3 TiB 8.1 TiB  10 GiB  76 GiB 
4.4 TiB 65.47 0.95 186     up         osd.37
  38   hdd   12.65529  1.00000  13 TiB 7.5 TiB 7.3 TiB  16 GiB  72 GiB 
5.1 TiB 59.40 0.86 188     up         osd.38
  39   hdd   12.65529  1.00000  13 TiB 8.0 TiB 7.7 TiB  12 GiB  76 GiB 
4.7 TiB 62.87 0.91 186     up         osd.39
  40   hdd   12.65529  1.00000  13 TiB 8.5 TiB 8.3 TiB  10 GiB  78 GiB 
4.1 TiB 67.41 0.98 184     up         osd.40
  41   hdd   12.65529  1.00000  13 TiB 8.6 TiB 8.4 TiB 8.9 GiB  78 GiB 
4.0 TiB 68.29 0.99 181     up         osd.41
  42   hdd   12.65529  1.00000  13 TiB 9.3 TiB 9.1 TiB 4.4 GiB  82 GiB 
3.4 TiB 73.48 1.07 184     up         osd.42
  43   hdd   12.65529  1.00000  13 TiB 9.8 TiB 9.6 TiB 2.9 GiB  85 GiB 
2.8 TiB 77.49 1.13 183     up         osd.43
  44   hdd   12.65529  1.00000  13 TiB  10 TiB  10 TiB  10 GiB  87 GiB 
2.4 TiB 80.79 1.18 185     up         osd.44
  45   hdd   12.65529  1.00000  13 TiB 8.3 TiB 8.1 TiB 5.9 GiB  76 GiB 
4.4 TiB 65.36 0.95 186     up         osd.45
  46   hdd   12.65529  1.00000  13 TiB 6.2 TiB 6.1 TiB  12 GiB  64 GiB 
6.4 TiB 49.36 0.72 184     up         osd.46
  47   hdd   12.65529  1.00000  13 TiB 9.4 TiB 9.2 TiB 8.2 GiB  83 GiB 
3.3 TiB 74.30 1.08 186     up         osd.47
  -9        193.17404        - 202 TiB 141 TiB 137 TiB 120 GiB 1.2 TiB  
62 TiB 69.61 1.01   -            host trex-ceph4
  48   hdd   12.65529  1.00000  13 TiB 8.8 TiB 8.6 TiB 2.8 GiB  81 GiB 
3.8 TiB 69.88 1.02 197     up         osd.48
  49   hdd   12.65529  1.00000  13 TiB 7.4 TiB 7.2 TiB 8.7 GiB  74 GiB 
5.2 TiB 58.53 0.85 190     up         osd.49
  50   hdd   12.65529  1.00000  13 TiB 9.0 TiB 8.7 TiB 2.9 GiB  81 GiB 
3.7 TiB 70.78 1.03 193     up         osd.50
  51   hdd   12.65529  1.00000  13 TiB 8.8 TiB 8.6 TiB  12 GiB  80 GiB 
3.8 TiB 69.71 1.01 192     up         osd.51
  52   hdd   12.65529  1.00000  13 TiB 8.9 TiB 8.7 TiB  10 GiB  81 GiB 
3.7 TiB 70.41 1.02 193     up         osd.52
  53   hdd    8.00000  0.40524  13 TiB  10 TiB  10 TiB  32 KiB  74 GiB 
2.2 TiB 82.72 1.20  84     up         osd.53
  54   hdd   12.65529  1.00000  13 TiB 9.6 TiB 9.3 TiB 6.9 GiB  84 GiB 
3.1 TiB 75.72 1.10 194     up         osd.54
  55   hdd   12.65529  1.00000  13 TiB 9.3 TiB 9.1 TiB  13 GiB  84 GiB 
3.3 TiB 73.58 1.07 192     up         osd.55
  56   hdd   12.65529  1.00000  13 TiB 7.4 TiB 7.2 TiB  10 GiB  72 GiB 
5.2 TiB 58.63 0.85 195     up         osd.56
  57   hdd   12.65529  1.00000  13 TiB 8.7 TiB 8.4 TiB 7.3 GiB  80 GiB 
4.0 TiB 68.53 1.00 194     up         osd.57
  58   hdd   12.65529  1.00000  13 TiB 7.6 TiB 7.4 TiB 8.7 GiB  74 GiB 
5.0 TiB 60.38 0.88 186     up         osd.58
  59   hdd   12.65529  0.51837  13 TiB 7.3 TiB 7.1 TiB 3.0 GiB  59 GiB 
5.3 TiB 57.97 0.84  87     up         osd.59
  60   hdd   12.65529  1.00000  13 TiB 8.7 TiB 8.5 TiB 7.4 GiB  80 GiB 
4.0 TiB 68.69 1.00 191     up         osd.60
  61   hdd    8.00000  1.00000  13 TiB  11 TiB  11 TiB 3.9 GiB  94 GiB 
1.7 TiB 86.91 1.26 184     up         osd.61
  62   hdd   12.65529  1.00000  13 TiB 8.6 TiB 8.3 TiB  11 GiB  78 GiB 
4.1 TiB 67.78 0.99 190     up         osd.62
  63   hdd   12.65529  1.00000  13 TiB 9.3 TiB 9.1 TiB  12 GiB  82 GiB 
3.3 TiB 73.54 1.07 187     up         osd.63
-11        202.48462        - 202 TiB 135 TiB 131 TiB 138 GiB 1.2 TiB  
68 TiB 66.58 0.97   -            host trex-ceph5
  64   hdd   12.65529  1.00000  13 TiB 8.2 TiB 7.9 TiB  10 GiB  76 GiB 
4.5 TiB 64.53 0.94 185     up         osd.64
  65   hdd   12.65529  1.00000  13 TiB 7.5 TiB 7.2 TiB  16 GiB  70 GiB 
5.2 TiB 58.87 0.86 186     up         osd.65
  66   hdd   12.65529  1.00000  13 TiB 9.7 TiB 9.4 TiB 4.4 GiB  84 GiB 
3.0 TiB 76.45 1.11 185     up         osd.66
  67   hdd   12.65529  1.00000  13 TiB 9.3 TiB 9.0 TiB  13 GiB  81 GiB 
3.4 TiB 73.32 1.07 187     up         osd.67
  68   hdd   12.65529  1.00000  13 TiB  10 TiB 9.7 TiB  13 GiB  85 GiB 
2.7 TiB 78.87 1.15 186     up         osd.68
  69   hdd   12.65529  1.00000  13 TiB 9.8 TiB 9.6 TiB 5.7 GiB  84 GiB 
2.8 TiB 77.61 1.13 188     up         osd.69
  70   hdd   12.65529  1.00000  13 TiB 7.5 TiB 7.3 TiB 8.6 GiB  71 GiB 
5.1 TiB 59.52 0.87 188     up         osd.70
  71   hdd   12.65529  1.00000  13 TiB 8.3 TiB 8.1 TiB  10 GiB  76 GiB 
4.3 TiB 65.73 0.96 189     up         osd.71
  72   hdd   12.65529  1.00000  13 TiB 8.0 TiB 7.8 TiB 4.6 GiB  74 GiB 
4.6 TiB 63.51 0.92 187     up         osd.72
  73   hdd   12.65529  1.00000  13 TiB 8.5 TiB 8.3 TiB 8.7 GiB  77 GiB 
4.1 TiB 67.52 0.98 190     up         osd.73
  74   hdd   12.65529  1.00000  13 TiB 6.8 TiB 6.5 TiB 4.5 GiB  68 GiB 
5.9 TiB 53.41 0.78 185     up         osd.74
  75   hdd   12.65529  1.00000  13 TiB 9.3 TiB 9.1 TiB 7.3 GiB  81 GiB 
3.4 TiB 73.38 1.07 186     up         osd.75
  76   hdd   12.65529  1.00000  13 TiB 6.8 TiB 6.6 TiB 8.8 GiB  67 GiB 
5.9 TiB 53.53 0.78 188     up         osd.76
  77   hdd   12.65529  1.00000  13 TiB 7.6 TiB 7.4 TiB 4.4 GiB  72 GiB 
5.1 TiB 59.85 0.87 183     up         osd.77
  78   hdd   12.65529  0.80603  13 TiB  10 TiB 9.8 TiB 5.9 GiB  82 GiB 
2.6 TiB 79.65 1.16 155     up         osd.78
  79   hdd   12.65529  1.00000  13 TiB 7.5 TiB 7.3 TiB  13 GiB  71 GiB 
5.1 TiB 59.50 0.87 185     up         osd.79
-13        202.48462        - 202 TiB 135 TiB 131 TiB 147 GiB 1.2 TiB  
67 TiB 66.74 0.97   -            host trex-ceph6
  80   hdd   12.65529  1.00000  13 TiB 7.5 TiB 7.2 TiB  16 GiB  71 GiB 
5.2 TiB 58.88 0.86 187     up         osd.80
  81   hdd   12.65529  1.00000  13 TiB 9.3 TiB 9.1 TiB  10 GiB  81 GiB 
3.3 TiB 73.55 1.07 187     up         osd.81
  82   hdd   12.65529  1.00000  13 TiB 9.8 TiB 9.6 TiB  10 GiB  83 GiB 
2.8 TiB 77.82 1.13 180     up         osd.82
  83   hdd   12.65529  1.00000  13 TiB 6.7 TiB 6.5 TiB 6.1 GiB  66 GiB 
6.0 TiB 52.96 0.77 183     up         osd.83
  84   hdd   12.65529  1.00000  13 TiB 8.9 TiB 8.6 TiB 5.9 GiB  79 GiB 
3.8 TiB 69.94 1.02 187     up         osd.84
  85   hdd   12.65529  1.00000  13 TiB 8.1 TiB 7.9 TiB  11 GiB  75 GiB 
4.6 TiB 63.92 0.93 182     up         osd.85
  86   hdd   12.65529  1.00000  13 TiB 9.0 TiB 8.8 TiB 5.6 GiB  80 GiB 
3.7 TiB 71.02 1.03 189     up         osd.86
  87   hdd   12.65529  1.00000  13 TiB 7.8 TiB 7.6 TiB  10 GiB  74 GiB 
4.9 TiB 61.53 0.90 189     up         osd.87
  88   hdd   12.65529  1.00000  13 TiB 7.5 TiB 7.2 TiB 8.6 GiB  72 GiB 
5.2 TiB 58.89 0.86 191     up         osd.88
  89   hdd   12.65529  1.00000  13 TiB 9.1 TiB 8.8 TiB 7.3 GiB  80 GiB 
3.6 TiB 71.57 1.04 186     up         osd.89
  90   hdd   12.65529  1.00000  13 TiB 8.0 TiB 7.8 TiB  10 GiB  75 GiB 
4.6 TiB 63.50 0.92 186     up         osd.90
  91   hdd   12.65529  0.80237  13 TiB 9.1 TiB 8.9 TiB 4.4 GiB  75 GiB 
3.5 TiB 72.28 1.05 145     up         osd.91
  92   hdd   12.65529  1.00000  13 TiB 8.7 TiB 8.5 TiB  12 GiB  78 GiB 
3.9 TiB 68.90 1.00 187     up         osd.92
  93   hdd   12.65529  1.00000  13 TiB 7.8 TiB 7.6 TiB  10 GiB  73 GiB 
4.8 TiB 61.94 0.90 182     up         osd.93
  94   hdd   12.65529  1.00000  13 TiB 9.1 TiB 8.8 TiB  13 GiB  79 GiB 
3.6 TiB 71.54 1.04 188     up         osd.94
  95   hdd   12.65529  1.00000  13 TiB 8.8 TiB 8.6 TiB 5.8 GiB  78 GiB 
3.8 TiB 69.59 1.01 186     up         osd.95
-15        197.82933        - 202 TiB 139 TiB 136 TiB 125 GiB 1.2 TiB  
63 TiB 68.78 1.00   -            host trex-ceph7
  96   hdd    8.00000  1.00000  13 TiB  11 TiB  11 TiB 8.3 GiB  93 GiB 
1.7 TiB 86.57 1.26 193     up         osd.96
  97   hdd   12.65529  0.42383  13 TiB 6.5 TiB 6.3 TiB  44 KiB  55 GiB 
6.1 TiB 51.44 0.75  80     up         osd.97
  98   hdd   12.65529  1.00000  13 TiB 7.4 TiB 7.2 TiB 8.8 GiB  72 GiB 
5.3 TiB 58.30 0.85 195     up         osd.98
  99   hdd   12.65529  1.00000  13 TiB 9.2 TiB 9.0 TiB  11 GiB  84 GiB 
3.4 TiB 72.88 1.06 208     up         osd.99
100   hdd   12.65529  1.00000  13 TiB  11 TiB  10 TiB 5.9 GiB  90 GiB 
2.0 TiB 84.32 1.23 197     up         osd.100
101   hdd   12.65529  1.00000  13 TiB 8.1 TiB 7.8 TiB  19 GiB  76 GiB 
4.6 TiB 63.73 0.93 200     up         osd.101
102   hdd   12.65529  1.00000  13 TiB 7.3 TiB 7.0 TiB 4.2 GiB  72 GiB 
5.4 TiB 57.41 0.84 199     up         osd.102
103   hdd   12.65529  1.00000  13 TiB 9.4 TiB 9.1 TiB 5.7 GiB  84 GiB 
3.3 TiB 74.06 1.08 198     up         osd.103
104   hdd   12.65529  1.00000  13 TiB 6.7 TiB 6.4 TiB 7.3 GiB  67 GiB 
6.0 TiB 52.64 0.77 184     up         osd.104
105   hdd   12.65529  1.00000  13 TiB 7.5 TiB 7.3 TiB  16 GiB  73 GiB 
5.1 TiB 59.52 0.87 200     up         osd.105
106   hdd   12.65529  1.00000  13 TiB 9.9 TiB 9.7 TiB 8.6 GiB  87 GiB 
2.7 TiB 78.49 1.14 200     up         osd.106
107   hdd   12.65529  1.00000  13 TiB 9.3 TiB 9.1 TiB 8.1 GiB  83 GiB 
3.3 TiB 73.54 1.07 199     up         osd.107
108   hdd   12.65529  1.00000  13 TiB  10 TiB  10 TiB 4.3 GiB  89 GiB 
2.2 TiB 82.29 1.20 193     up         osd.108
109   hdd   12.65529  1.00000  13 TiB 9.2 TiB 9.0 TiB  10 GiB  82 GiB 
3.5 TiB 72.67 1.06 192     up         osd.109
110   hdd   12.65529  0.44075  13 TiB 6.7 TiB 6.5 TiB 4.2 GiB  57 GiB 
6.0 TiB 52.95 0.77  85     up         osd.110
111   hdd   12.65529  0.81145  13 TiB  10 TiB 9.8 TiB 4.3 GiB  85 GiB 
2.6 TiB 79.66 1.16 169     up         osd.111
-17        172.20818        - 202 TiB 159 TiB 155 TiB  81 GiB 1.3 TiB  
44 TiB 78.37 1.14   -            host trex-ceph8
112   hdd   12.65529  0.42731  13 TiB 9.0 TiB 8.8 TiB 1.6 GiB  72 GiB 
3.6 TiB 71.32 1.04  98     up         osd.112
113   hdd   12.65529  0.78836  13 TiB 9.7 TiB 9.4 TiB 5.7 GiB  82 GiB 
3.0 TiB 76.33 1.11 158     up         osd.113
114   hdd    8.00000  0.29999  13 TiB  11 TiB  11 TiB 2.9 GiB  91 GiB 
1.7 TiB 86.24 1.26 176     up         osd.114
115   hdd    8.00000  0.29999  13 TiB  11 TiB  11 TiB 2.9 GiB  96 GiB 
1.7 TiB 86.27 1.26 208     up         osd.115
116   hdd    5.00000  0.07227  13 TiB 9.9 TiB 9.6 TiB  28 KiB  68 GiB 
2.8 TiB 78.13 1.14  39     up         osd.116
117   hdd   12.65529  1.00000  13 TiB  11 TiB  10 TiB  12 GiB  99 GiB 
1.9 TiB 84.68 1.23 243     up         osd.117
118   hdd   12.65529  0.63712  13 TiB 9.8 TiB 9.6 TiB  98 MiB  82 GiB 
2.9 TiB 77.45 1.13 147     up         osd.118
119   hdd   12.65529  1.00000  13 TiB  11 TiB  10 TiB  10 GiB  96 GiB 
2.0 TiB 84.12 1.22 223     up         osd.119
120   hdd    4.00000  0.29999  13 TiB  11 TiB  11 TiB 1.4 GiB  85 GiB 
1.6 TiB 87.43 1.27 129     up         osd.120
121   hdd   12.65529  1.00000  13 TiB 8.6 TiB 8.4 TiB 6.7 GiB  68 GiB 
4.1 TiB 67.82 0.99 111     up         osd.121
122   hdd    8.00000  0.39594  13 TiB  11 TiB  11 TiB 1.6 GiB  82 GiB 
1.8 TiB 85.62 1.25 101     up         osd.122
123   hdd   12.65529  0.49266  13 TiB 9.0 TiB 8.8 TiB 549 MiB  77 GiB 
3.6 TiB 71.45 1.04 134     up         osd.123
124   hdd   12.65529  1.00000  13 TiB  10 TiB 9.8 TiB 9.6 GiB  94 GiB 
2.6 TiB 79.37 1.16 221     up         osd.124
125   hdd   12.65529  1.00000  13 TiB  10 TiB 9.9 TiB 8.5 GiB  96 GiB 
2.5 TiB 80.01 1.16 253     up         osd.125
126   hdd   12.65529  0.52419  13 TiB 8.3 TiB 8.0 TiB 2.9 GiB  70 GiB 
4.4 TiB 65.46 0.95 113     up         osd.126
127   hdd   12.65529  1.00000  13 TiB 9.1 TiB 8.9 TiB  15 GiB  90 GiB 
3.5 TiB 72.18 1.05 243     up         osd.127
-19        202.48462        - 202 TiB 135 TiB 131 TiB 120 GiB 1.2 TiB  
68 TiB 66.54 0.97   -            host trex-ceph9
128   hdd   12.65529  1.00000  13 TiB 6.5 TiB 6.3 TiB 2.8 GiB  66 GiB 
6.1 TiB 51.41 0.75 184     up         osd.128
129   hdd   12.65529  1.00000  13 TiB 7.8 TiB 7.6 TiB 8.8 GiB  72 GiB 
4.9 TiB 61.43 0.89 183     up         osd.129
130   hdd   12.65529  0.79724  13 TiB 8.6 TiB 8.3 TiB 4.3 GiB  71 GiB 
4.1 TiB 67.65 0.98 130     up         osd.130
131   hdd   12.65529  1.00000  13 TiB 8.3 TiB 8.1 TiB 5.9 GiB  77 GiB 
4.3 TiB 65.88 0.96 191     up         osd.131
132   hdd   12.65529  1.00000  13 TiB 7.0 TiB 6.8 TiB 8.7 GiB  69 GiB 
5.6 TiB 55.52 0.81 186     up         osd.132
133   hdd   12.65529  0.78160  13 TiB 8.6 TiB 8.4 TiB 2.9 GiB  72 GiB 
4.1 TiB 67.93 0.99 131     up         osd.133
134   hdd   12.65529  1.00000  13 TiB 8.0 TiB 7.8 TiB 7.4 GiB  74 GiB 
4.6 TiB 63.61 0.93 186     up         osd.134
135   hdd   12.65529  1.00000  13 TiB 9.8 TiB 9.6 TiB  14 GiB  85 GiB 
2.8 TiB 77.54 1.13 185     up         osd.135
136   hdd   12.65529  1.00000  13 TiB 9.3 TiB 9.1 TiB 5.7 GiB  83 GiB 
3.3 TiB 73.59 1.07 195     up         osd.136
137   hdd   12.65529  1.00000  13 TiB 9.7 TiB 9.4 TiB 5.9 GiB  84 GiB 
3.0 TiB 76.46 1.11 187     up         osd.137
138   hdd   12.65529  1.00000  13 TiB  10 TiB 9.8 TiB 5.6 GiB  85 GiB 
2.7 TiB 78.99 1.15 188     up         osd.138
139   hdd   12.65529  1.00000  13 TiB 8.2 TiB 7.9 TiB 5.8 GiB  75 GiB 
4.5 TiB 64.51 0.94 185     up         osd.139
140   hdd   12.65529  1.00000  13 TiB 9.8 TiB 9.6 TiB  11 GiB  84 GiB 
2.8 TiB 77.59 1.13 186     up         osd.140
141   hdd   12.65529  1.00000  13 TiB 6.8 TiB 6.6 TiB  16 GiB  69 GiB 
5.8 TiB 54.01 0.79 185     up         osd.141
142   hdd   12.65529  1.00000  13 TiB 7.9 TiB 7.7 TiB  11 GiB  74 GiB 
4.8 TiB 62.41 0.91 192     up         osd.142
143   hdd   12.65529  1.00000  13 TiB 8.4 TiB 8.1 TiB 4.4 GiB  75 GiB 
4.3 TiB 66.15 0.96 181     up         osd.143
                         TOTAL 1.8 PiB 1.2 PiB 1.2 PiB 1.1 TiB  11 TiB 
570 TiB 68.71
MIN/MAX VAR: 0.72/1.27  STDDEV: 8.88


On 2022-05-04 1:26 p.m., Josh Baergen wrote:
> [△EXTERNAL]
>
>
>
> Hey Dave,
>
> What version of Ceph is this cluster currently running, and was it
> running any previous versions? Were the more full and less full OSDs
> deployed on different Ceph versions?
>
> Could you also provide the output of "ceph osd df tree"?
>
> Josh
>
> On Wed, May 4, 2022 at 1:21 PM David Schulz <dschulz@xxxxxxxxxxx> wrote:
>> Hi Everyone,
>>
>> I'm looking for a bit of guidance on a 9 server * 16 OSDs per server=
>> 144 OSDs system.
>>
>> This cluster has 143 OSDs in it but ceph osd df shows that they are very
>> unbalanced in their utilization.  Some are around 50% full and yet
>> others are pushing 85% full.  The balancer was on and that was making it
>> worse.  At one point we had an osd at 89.7% nearly triggering a full
>> cluster.
>>
>> We're running cephfs on erasure coded data pool k=7, m=2.  We have two
>> options to proceed here and I'm not sure which is appropriate to be done
>> next.
>>
>> Option 1: Add another server -- I'm currently unconvinced that this will
>> help much as there is so much skew.  I fear that the cluster will just
>> continue to be skewed even if we add another 16 or so OSDs.
>>
>> Option 2: I've read that the the reason for having skewed OSD usage is
>> having too few PGs in the pool.    Currently our data pool is set to 512
>> but I think this number is inflated when using erasure coding which is
>> why it's set only at 512.  If I recall k=7 is like having 512*7=3584
>> PGs.  And is it safe to suddenly double the number of pgs on a cluster
>> that is close to the edge of going full?
>>
>> Here's a chart of our 12 highest usage OSDs plus the one that was the
>> worst early on in this process:
>>
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpasteboard.co%2FqiaYnDxScEHM.png&amp;data=05%7C01%7Cdschulz%40ucalgary.ca%7Cbf36039589704dcd4d4508da2e03f5cf%7Cc609a0eca5e346319686192280bd9151%7C1%7C0%7C637872891789883757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=67SKwyxo%2FAvtzkq%2BsNKY916p91cM1Pbd0zZWUlVyYMc%3D&amp;reserved=0
>>
>> Here's a chart showing the distribution of OSD usage:
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpasteboard.co%2FNKITkmZYaGiH.png&amp;data=05%7C01%7Cdschulz%40ucalgary.ca%7Cbf36039589704dcd4d4508da2e03f5cf%7Cc609a0eca5e346319686192280bd9151%7C1%7C0%7C637872891789883757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=9RLCbe9RL0cEBz6k9N3R19eL0BI0pxm%2BO1xwpYQmFqo%3D&amp;reserved=0
>>
>> It is more or less gaussian but the tails seem really long for a cluster
>> that's been running the balancer for a long time.
>>
>> Any guidance would be very much appreciated.
>>
>> Sincerely
>>
>> -Dave
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux