Hi Josh, Thanks for getting back to me so soon! This cluster was upgraded from 13.x to 14.2.9 some time ago. The entire cluster was installed at the 13.x time and was upgraded together so all OSDs should have the same formatting etc. Below is pasted the ceph osd df tree output. Sincerely -Dave # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1773.46399 - 1.8 PiB 1.2 PiB 1.2 PiB 1.1 TiB 11 TiB 570 TiB 68.71 1.00 - root default -3 202.48462 - 202 TiB 135 TiB 131 TiB 122 GiB 1.2 TiB 68 TiB 66.50 0.97 - host trex-ceph1 0 hdd 12.65529 1.00000 13 TiB 7.7 TiB 7.5 TiB 4.4 GiB 73 GiB 5.0 TiB 60.73 0.88 186 up osd.0 1 hdd 12.65529 1.00000 13 TiB 8.3 TiB 8.1 TiB 7.2 GiB 77 GiB 4.3 TiB 65.64 0.96 185 up osd.1 2 hdd 12.65529 1.00000 13 TiB 6.8 TiB 6.6 TiB 6.1 GiB 67 GiB 5.9 TiB 53.51 0.78 183 up osd.2 3 hdd 12.65529 1.00000 13 TiB 8.6 TiB 8.3 TiB 7.2 GiB 76 GiB 4.1 TiB 67.57 0.98 182 up osd.3 4 hdd 12.65529 1.00000 13 TiB 8.5 TiB 8.3 TiB 8.6 GiB 76 GiB 4.2 TiB 67.19 0.98 178 up osd.4 5 hdd 12.65529 1.00000 13 TiB 10 TiB 10 TiB 13 GiB 86 GiB 2.3 TiB 81.88 1.19 180 up osd.5 6 hdd 12.65529 1.00000 13 TiB 6.8 TiB 6.6 TiB 10 GiB 67 GiB 5.9 TiB 53.47 0.78 183 up osd.6 7 hdd 12.65529 1.00000 13 TiB 10 TiB 9.9 TiB 8.7 GiB 86 GiB 2.6 TiB 79.82 1.16 184 up osd.7 8 hdd 12.65529 1.00000 13 TiB 7.8 TiB 7.6 TiB 7.2 GiB 72 GiB 4.9 TiB 61.60 0.90 181 up osd.8 9 hdd 12.65529 1.00000 13 TiB 7.7 TiB 7.5 TiB 8.4 GiB 73 GiB 4.9 TiB 60.93 0.89 186 up osd.9 10 hdd 12.65529 1.00000 13 TiB 8.4 TiB 8.2 TiB 5.8 GiB 75 GiB 4.3 TiB 66.30 0.96 179 up osd.10 11 hdd 12.65529 1.00000 13 TiB 7.3 TiB 7.1 TiB 6.0 GiB 69 GiB 5.3 TiB 57.86 0.84 182 up osd.11 12 hdd 12.65529 1.00000 13 TiB 9.0 TiB 8.8 TiB 10 GiB 78 GiB 3.6 TiB 71.41 1.04 183 up osd.12 13 hdd 12.65529 1.00000 13 TiB 7.8 TiB 7.6 TiB 2.8 GiB 74 GiB 4.9 TiB 61.52 0.90 188 up osd.13 14 hdd 12.65529 1.00000 13 TiB 10 TiB 10 TiB 5.9 GiB 87 GiB 2.3 TiB 81.61 1.19 185 up osd.14 15 hdd 12.65529 1.00000 13 TiB 9.2 TiB 9.0 TiB 10 GiB 81 GiB 3.4 TiB 72.99 1.06 184 up osd.15 -5 197.82933 - 202 TiB 140 TiB 136 TiB 130 GiB 1.2 TiB 63 TiB 68.97 1.00 - host trex-ceph2 16 hdd 12.65529 1.00000 13 TiB 7.6 TiB 7.4 TiB 9.9 GiB 73 GiB 5.0 TiB 60.42 0.88 191 up osd.16 17 hdd 12.65529 1.00000 13 TiB 8.6 TiB 8.4 TiB 18 GiB 78 GiB 4.0 TiB 68.29 0.99 191 up osd.17 18 hdd 12.65529 1.00000 13 TiB 8.5 TiB 8.3 TiB 8.7 GiB 79 GiB 4.2 TiB 67.01 0.98 194 up osd.18 19 hdd 12.65529 1.00000 13 TiB 9.9 TiB 9.6 TiB 9.0 GiB 85 GiB 2.8 TiB 78.02 1.14 188 up osd.19 20 hdd 12.65529 1.00000 13 TiB 8.7 TiB 8.5 TiB 10 GiB 79 GiB 3.9 TiB 69.02 1.00 191 up osd.20 21 hdd 12.65529 0.79160 13 TiB 8.4 TiB 8.2 TiB 4.5 GiB 71 GiB 4.2 TiB 66.56 0.97 139 up osd.21 22 hdd 12.65529 0.80670 13 TiB 9.9 TiB 9.7 TiB 2.7 GiB 83 GiB 2.8 TiB 78.26 1.14 155 up osd.22 23 hdd 12.65529 1.00000 13 TiB 8.8 TiB 8.6 TiB 12 GiB 80 GiB 3.9 TiB 69.54 1.01 194 up osd.23 24 hdd 12.65529 1.00000 13 TiB 8.0 TiB 7.8 TiB 7.3 GiB 76 GiB 4.7 TiB 63.13 0.92 190 up osd.24 25 hdd 8.00000 0.42892 13 TiB 11 TiB 10 TiB 6.9 GiB 77 GiB 2.0 TiB 83.82 1.22 86 up osd.25 26 hdd 12.65529 1.00000 13 TiB 7.2 TiB 7.0 TiB 10 GiB 70 GiB 5.4 TiB 57.28 0.83 187 up osd.26 27 hdd 12.65529 1.00000 13 TiB 10 TiB 9.9 TiB 1.5 GiB 86 GiB 2.5 TiB 79.90 1.16 182 up osd.27 28 hdd 12.65529 1.00000 13 TiB 9.2 TiB 9.0 TiB 8.8 GiB 81 GiB 3.4 TiB 72.79 1.06 190 up osd.28 29 hdd 12.65529 1.00000 13 TiB 6.8 TiB 6.6 TiB 7.2 GiB 69 GiB 5.9 TiB 53.53 0.78 194 up osd.29 30 hdd 12.65529 1.00000 13 TiB 9.0 TiB 8.8 TiB 4.5 GiB 80 GiB 3.7 TiB 71.05 1.03 186 up osd.30 31 hdd 12.65529 1.00000 13 TiB 8.2 TiB 8.0 TiB 8.8 GiB 76 GiB 4.4 TiB 64.92 0.94 191 up osd.31 -7 202.48462 - 202 TiB 134 TiB 131 TiB 146 GiB 1.2 TiB 68 TiB 66.33 0.97 - host trex-ceph3 32 hdd 12.65529 1.00000 13 TiB 9.1 TiB 8.8 TiB 10 GiB 80 GiB 3.6 TiB 71.63 1.04 182 up osd.32 33 hdd 12.65529 1.00000 13 TiB 8.9 TiB 8.7 TiB 4.3 GiB 80 GiB 3.8 TiB 70.15 1.02 180 up osd.33 34 hdd 12.65529 1.00000 13 TiB 6.4 TiB 6.2 TiB 13 GiB 67 GiB 6.3 TiB 50.43 0.73 183 up osd.34 35 hdd 12.65529 1.00000 13 TiB 8.0 TiB 7.8 TiB 7.1 GiB 75 GiB 4.6 TiB 63.51 0.92 186 up osd.35 36 hdd 12.65529 1.00000 13 TiB 7.8 TiB 7.6 TiB 11 GiB 73 GiB 4.9 TiB 61.41 0.89 183 up osd.36 37 hdd 12.65529 1.00000 13 TiB 8.3 TiB 8.1 TiB 10 GiB 76 GiB 4.4 TiB 65.47 0.95 186 up osd.37 38 hdd 12.65529 1.00000 13 TiB 7.5 TiB 7.3 TiB 16 GiB 72 GiB 5.1 TiB 59.40 0.86 188 up osd.38 39 hdd 12.65529 1.00000 13 TiB 8.0 TiB 7.7 TiB 12 GiB 76 GiB 4.7 TiB 62.87 0.91 186 up osd.39 40 hdd 12.65529 1.00000 13 TiB 8.5 TiB 8.3 TiB 10 GiB 78 GiB 4.1 TiB 67.41 0.98 184 up osd.40 41 hdd 12.65529 1.00000 13 TiB 8.6 TiB 8.4 TiB 8.9 GiB 78 GiB 4.0 TiB 68.29 0.99 181 up osd.41 42 hdd 12.65529 1.00000 13 TiB 9.3 TiB 9.1 TiB 4.4 GiB 82 GiB 3.4 TiB 73.48 1.07 184 up osd.42 43 hdd 12.65529 1.00000 13 TiB 9.8 TiB 9.6 TiB 2.9 GiB 85 GiB 2.8 TiB 77.49 1.13 183 up osd.43 44 hdd 12.65529 1.00000 13 TiB 10 TiB 10 TiB 10 GiB 87 GiB 2.4 TiB 80.79 1.18 185 up osd.44 45 hdd 12.65529 1.00000 13 TiB 8.3 TiB 8.1 TiB 5.9 GiB 76 GiB 4.4 TiB 65.36 0.95 186 up osd.45 46 hdd 12.65529 1.00000 13 TiB 6.2 TiB 6.1 TiB 12 GiB 64 GiB 6.4 TiB 49.36 0.72 184 up osd.46 47 hdd 12.65529 1.00000 13 TiB 9.4 TiB 9.2 TiB 8.2 GiB 83 GiB 3.3 TiB 74.30 1.08 186 up osd.47 -9 193.17404 - 202 TiB 141 TiB 137 TiB 120 GiB 1.2 TiB 62 TiB 69.61 1.01 - host trex-ceph4 48 hdd 12.65529 1.00000 13 TiB 8.8 TiB 8.6 TiB 2.8 GiB 81 GiB 3.8 TiB 69.88 1.02 197 up osd.48 49 hdd 12.65529 1.00000 13 TiB 7.4 TiB 7.2 TiB 8.7 GiB 74 GiB 5.2 TiB 58.53 0.85 190 up osd.49 50 hdd 12.65529 1.00000 13 TiB 9.0 TiB 8.7 TiB 2.9 GiB 81 GiB 3.7 TiB 70.78 1.03 193 up osd.50 51 hdd 12.65529 1.00000 13 TiB 8.8 TiB 8.6 TiB 12 GiB 80 GiB 3.8 TiB 69.71 1.01 192 up osd.51 52 hdd 12.65529 1.00000 13 TiB 8.9 TiB 8.7 TiB 10 GiB 81 GiB 3.7 TiB 70.41 1.02 193 up osd.52 53 hdd 8.00000 0.40524 13 TiB 10 TiB 10 TiB 32 KiB 74 GiB 2.2 TiB 82.72 1.20 84 up osd.53 54 hdd 12.65529 1.00000 13 TiB 9.6 TiB 9.3 TiB 6.9 GiB 84 GiB 3.1 TiB 75.72 1.10 194 up osd.54 55 hdd 12.65529 1.00000 13 TiB 9.3 TiB 9.1 TiB 13 GiB 84 GiB 3.3 TiB 73.58 1.07 192 up osd.55 56 hdd 12.65529 1.00000 13 TiB 7.4 TiB 7.2 TiB 10 GiB 72 GiB 5.2 TiB 58.63 0.85 195 up osd.56 57 hdd 12.65529 1.00000 13 TiB 8.7 TiB 8.4 TiB 7.3 GiB 80 GiB 4.0 TiB 68.53 1.00 194 up osd.57 58 hdd 12.65529 1.00000 13 TiB 7.6 TiB 7.4 TiB 8.7 GiB 74 GiB 5.0 TiB 60.38 0.88 186 up osd.58 59 hdd 12.65529 0.51837 13 TiB 7.3 TiB 7.1 TiB 3.0 GiB 59 GiB 5.3 TiB 57.97 0.84 87 up osd.59 60 hdd 12.65529 1.00000 13 TiB 8.7 TiB 8.5 TiB 7.4 GiB 80 GiB 4.0 TiB 68.69 1.00 191 up osd.60 61 hdd 8.00000 1.00000 13 TiB 11 TiB 11 TiB 3.9 GiB 94 GiB 1.7 TiB 86.91 1.26 184 up osd.61 62 hdd 12.65529 1.00000 13 TiB 8.6 TiB 8.3 TiB 11 GiB 78 GiB 4.1 TiB 67.78 0.99 190 up osd.62 63 hdd 12.65529 1.00000 13 TiB 9.3 TiB 9.1 TiB 12 GiB 82 GiB 3.3 TiB 73.54 1.07 187 up osd.63 -11 202.48462 - 202 TiB 135 TiB 131 TiB 138 GiB 1.2 TiB 68 TiB 66.58 0.97 - host trex-ceph5 64 hdd 12.65529 1.00000 13 TiB 8.2 TiB 7.9 TiB 10 GiB 76 GiB 4.5 TiB 64.53 0.94 185 up osd.64 65 hdd 12.65529 1.00000 13 TiB 7.5 TiB 7.2 TiB 16 GiB 70 GiB 5.2 TiB 58.87 0.86 186 up osd.65 66 hdd 12.65529 1.00000 13 TiB 9.7 TiB 9.4 TiB 4.4 GiB 84 GiB 3.0 TiB 76.45 1.11 185 up osd.66 67 hdd 12.65529 1.00000 13 TiB 9.3 TiB 9.0 TiB 13 GiB 81 GiB 3.4 TiB 73.32 1.07 187 up osd.67 68 hdd 12.65529 1.00000 13 TiB 10 TiB 9.7 TiB 13 GiB 85 GiB 2.7 TiB 78.87 1.15 186 up osd.68 69 hdd 12.65529 1.00000 13 TiB 9.8 TiB 9.6 TiB 5.7 GiB 84 GiB 2.8 TiB 77.61 1.13 188 up osd.69 70 hdd 12.65529 1.00000 13 TiB 7.5 TiB 7.3 TiB 8.6 GiB 71 GiB 5.1 TiB 59.52 0.87 188 up osd.70 71 hdd 12.65529 1.00000 13 TiB 8.3 TiB 8.1 TiB 10 GiB 76 GiB 4.3 TiB 65.73 0.96 189 up osd.71 72 hdd 12.65529 1.00000 13 TiB 8.0 TiB 7.8 TiB 4.6 GiB 74 GiB 4.6 TiB 63.51 0.92 187 up osd.72 73 hdd 12.65529 1.00000 13 TiB 8.5 TiB 8.3 TiB 8.7 GiB 77 GiB 4.1 TiB 67.52 0.98 190 up osd.73 74 hdd 12.65529 1.00000 13 TiB 6.8 TiB 6.5 TiB 4.5 GiB 68 GiB 5.9 TiB 53.41 0.78 185 up osd.74 75 hdd 12.65529 1.00000 13 TiB 9.3 TiB 9.1 TiB 7.3 GiB 81 GiB 3.4 TiB 73.38 1.07 186 up osd.75 76 hdd 12.65529 1.00000 13 TiB 6.8 TiB 6.6 TiB 8.8 GiB 67 GiB 5.9 TiB 53.53 0.78 188 up osd.76 77 hdd 12.65529 1.00000 13 TiB 7.6 TiB 7.4 TiB 4.4 GiB 72 GiB 5.1 TiB 59.85 0.87 183 up osd.77 78 hdd 12.65529 0.80603 13 TiB 10 TiB 9.8 TiB 5.9 GiB 82 GiB 2.6 TiB 79.65 1.16 155 up osd.78 79 hdd 12.65529 1.00000 13 TiB 7.5 TiB 7.3 TiB 13 GiB 71 GiB 5.1 TiB 59.50 0.87 185 up osd.79 -13 202.48462 - 202 TiB 135 TiB 131 TiB 147 GiB 1.2 TiB 67 TiB 66.74 0.97 - host trex-ceph6 80 hdd 12.65529 1.00000 13 TiB 7.5 TiB 7.2 TiB 16 GiB 71 GiB 5.2 TiB 58.88 0.86 187 up osd.80 81 hdd 12.65529 1.00000 13 TiB 9.3 TiB 9.1 TiB 10 GiB 81 GiB 3.3 TiB 73.55 1.07 187 up osd.81 82 hdd 12.65529 1.00000 13 TiB 9.8 TiB 9.6 TiB 10 GiB 83 GiB 2.8 TiB 77.82 1.13 180 up osd.82 83 hdd 12.65529 1.00000 13 TiB 6.7 TiB 6.5 TiB 6.1 GiB 66 GiB 6.0 TiB 52.96 0.77 183 up osd.83 84 hdd 12.65529 1.00000 13 TiB 8.9 TiB 8.6 TiB 5.9 GiB 79 GiB 3.8 TiB 69.94 1.02 187 up osd.84 85 hdd 12.65529 1.00000 13 TiB 8.1 TiB 7.9 TiB 11 GiB 75 GiB 4.6 TiB 63.92 0.93 182 up osd.85 86 hdd 12.65529 1.00000 13 TiB 9.0 TiB 8.8 TiB 5.6 GiB 80 GiB 3.7 TiB 71.02 1.03 189 up osd.86 87 hdd 12.65529 1.00000 13 TiB 7.8 TiB 7.6 TiB 10 GiB 74 GiB 4.9 TiB 61.53 0.90 189 up osd.87 88 hdd 12.65529 1.00000 13 TiB 7.5 TiB 7.2 TiB 8.6 GiB 72 GiB 5.2 TiB 58.89 0.86 191 up osd.88 89 hdd 12.65529 1.00000 13 TiB 9.1 TiB 8.8 TiB 7.3 GiB 80 GiB 3.6 TiB 71.57 1.04 186 up osd.89 90 hdd 12.65529 1.00000 13 TiB 8.0 TiB 7.8 TiB 10 GiB 75 GiB 4.6 TiB 63.50 0.92 186 up osd.90 91 hdd 12.65529 0.80237 13 TiB 9.1 TiB 8.9 TiB 4.4 GiB 75 GiB 3.5 TiB 72.28 1.05 145 up osd.91 92 hdd 12.65529 1.00000 13 TiB 8.7 TiB 8.5 TiB 12 GiB 78 GiB 3.9 TiB 68.90 1.00 187 up osd.92 93 hdd 12.65529 1.00000 13 TiB 7.8 TiB 7.6 TiB 10 GiB 73 GiB 4.8 TiB 61.94 0.90 182 up osd.93 94 hdd 12.65529 1.00000 13 TiB 9.1 TiB 8.8 TiB 13 GiB 79 GiB 3.6 TiB 71.54 1.04 188 up osd.94 95 hdd 12.65529 1.00000 13 TiB 8.8 TiB 8.6 TiB 5.8 GiB 78 GiB 3.8 TiB 69.59 1.01 186 up osd.95 -15 197.82933 - 202 TiB 139 TiB 136 TiB 125 GiB 1.2 TiB 63 TiB 68.78 1.00 - host trex-ceph7 96 hdd 8.00000 1.00000 13 TiB 11 TiB 11 TiB 8.3 GiB 93 GiB 1.7 TiB 86.57 1.26 193 up osd.96 97 hdd 12.65529 0.42383 13 TiB 6.5 TiB 6.3 TiB 44 KiB 55 GiB 6.1 TiB 51.44 0.75 80 up osd.97 98 hdd 12.65529 1.00000 13 TiB 7.4 TiB 7.2 TiB 8.8 GiB 72 GiB 5.3 TiB 58.30 0.85 195 up osd.98 99 hdd 12.65529 1.00000 13 TiB 9.2 TiB 9.0 TiB 11 GiB 84 GiB 3.4 TiB 72.88 1.06 208 up osd.99 100 hdd 12.65529 1.00000 13 TiB 11 TiB 10 TiB 5.9 GiB 90 GiB 2.0 TiB 84.32 1.23 197 up osd.100 101 hdd 12.65529 1.00000 13 TiB 8.1 TiB 7.8 TiB 19 GiB 76 GiB 4.6 TiB 63.73 0.93 200 up osd.101 102 hdd 12.65529 1.00000 13 TiB 7.3 TiB 7.0 TiB 4.2 GiB 72 GiB 5.4 TiB 57.41 0.84 199 up osd.102 103 hdd 12.65529 1.00000 13 TiB 9.4 TiB 9.1 TiB 5.7 GiB 84 GiB 3.3 TiB 74.06 1.08 198 up osd.103 104 hdd 12.65529 1.00000 13 TiB 6.7 TiB 6.4 TiB 7.3 GiB 67 GiB 6.0 TiB 52.64 0.77 184 up osd.104 105 hdd 12.65529 1.00000 13 TiB 7.5 TiB 7.3 TiB 16 GiB 73 GiB 5.1 TiB 59.52 0.87 200 up osd.105 106 hdd 12.65529 1.00000 13 TiB 9.9 TiB 9.7 TiB 8.6 GiB 87 GiB 2.7 TiB 78.49 1.14 200 up osd.106 107 hdd 12.65529 1.00000 13 TiB 9.3 TiB 9.1 TiB 8.1 GiB 83 GiB 3.3 TiB 73.54 1.07 199 up osd.107 108 hdd 12.65529 1.00000 13 TiB 10 TiB 10 TiB 4.3 GiB 89 GiB 2.2 TiB 82.29 1.20 193 up osd.108 109 hdd 12.65529 1.00000 13 TiB 9.2 TiB 9.0 TiB 10 GiB 82 GiB 3.5 TiB 72.67 1.06 192 up osd.109 110 hdd 12.65529 0.44075 13 TiB 6.7 TiB 6.5 TiB 4.2 GiB 57 GiB 6.0 TiB 52.95 0.77 85 up osd.110 111 hdd 12.65529 0.81145 13 TiB 10 TiB 9.8 TiB 4.3 GiB 85 GiB 2.6 TiB 79.66 1.16 169 up osd.111 -17 172.20818 - 202 TiB 159 TiB 155 TiB 81 GiB 1.3 TiB 44 TiB 78.37 1.14 - host trex-ceph8 112 hdd 12.65529 0.42731 13 TiB 9.0 TiB 8.8 TiB 1.6 GiB 72 GiB 3.6 TiB 71.32 1.04 98 up osd.112 113 hdd 12.65529 0.78836 13 TiB 9.7 TiB 9.4 TiB 5.7 GiB 82 GiB 3.0 TiB 76.33 1.11 158 up osd.113 114 hdd 8.00000 0.29999 13 TiB 11 TiB 11 TiB 2.9 GiB 91 GiB 1.7 TiB 86.24 1.26 176 up osd.114 115 hdd 8.00000 0.29999 13 TiB 11 TiB 11 TiB 2.9 GiB 96 GiB 1.7 TiB 86.27 1.26 208 up osd.115 116 hdd 5.00000 0.07227 13 TiB 9.9 TiB 9.6 TiB 28 KiB 68 GiB 2.8 TiB 78.13 1.14 39 up osd.116 117 hdd 12.65529 1.00000 13 TiB 11 TiB 10 TiB 12 GiB 99 GiB 1.9 TiB 84.68 1.23 243 up osd.117 118 hdd 12.65529 0.63712 13 TiB 9.8 TiB 9.6 TiB 98 MiB 82 GiB 2.9 TiB 77.45 1.13 147 up osd.118 119 hdd 12.65529 1.00000 13 TiB 11 TiB 10 TiB 10 GiB 96 GiB 2.0 TiB 84.12 1.22 223 up osd.119 120 hdd 4.00000 0.29999 13 TiB 11 TiB 11 TiB 1.4 GiB 85 GiB 1.6 TiB 87.43 1.27 129 up osd.120 121 hdd 12.65529 1.00000 13 TiB 8.6 TiB 8.4 TiB 6.7 GiB 68 GiB 4.1 TiB 67.82 0.99 111 up osd.121 122 hdd 8.00000 0.39594 13 TiB 11 TiB 11 TiB 1.6 GiB 82 GiB 1.8 TiB 85.62 1.25 101 up osd.122 123 hdd 12.65529 0.49266 13 TiB 9.0 TiB 8.8 TiB 549 MiB 77 GiB 3.6 TiB 71.45 1.04 134 up osd.123 124 hdd 12.65529 1.00000 13 TiB 10 TiB 9.8 TiB 9.6 GiB 94 GiB 2.6 TiB 79.37 1.16 221 up osd.124 125 hdd 12.65529 1.00000 13 TiB 10 TiB 9.9 TiB 8.5 GiB 96 GiB 2.5 TiB 80.01 1.16 253 up osd.125 126 hdd 12.65529 0.52419 13 TiB 8.3 TiB 8.0 TiB 2.9 GiB 70 GiB 4.4 TiB 65.46 0.95 113 up osd.126 127 hdd 12.65529 1.00000 13 TiB 9.1 TiB 8.9 TiB 15 GiB 90 GiB 3.5 TiB 72.18 1.05 243 up osd.127 -19 202.48462 - 202 TiB 135 TiB 131 TiB 120 GiB 1.2 TiB 68 TiB 66.54 0.97 - host trex-ceph9 128 hdd 12.65529 1.00000 13 TiB 6.5 TiB 6.3 TiB 2.8 GiB 66 GiB 6.1 TiB 51.41 0.75 184 up osd.128 129 hdd 12.65529 1.00000 13 TiB 7.8 TiB 7.6 TiB 8.8 GiB 72 GiB 4.9 TiB 61.43 0.89 183 up osd.129 130 hdd 12.65529 0.79724 13 TiB 8.6 TiB 8.3 TiB 4.3 GiB 71 GiB 4.1 TiB 67.65 0.98 130 up osd.130 131 hdd 12.65529 1.00000 13 TiB 8.3 TiB 8.1 TiB 5.9 GiB 77 GiB 4.3 TiB 65.88 0.96 191 up osd.131 132 hdd 12.65529 1.00000 13 TiB 7.0 TiB 6.8 TiB 8.7 GiB 69 GiB 5.6 TiB 55.52 0.81 186 up osd.132 133 hdd 12.65529 0.78160 13 TiB 8.6 TiB 8.4 TiB 2.9 GiB 72 GiB 4.1 TiB 67.93 0.99 131 up osd.133 134 hdd 12.65529 1.00000 13 TiB 8.0 TiB 7.8 TiB 7.4 GiB 74 GiB 4.6 TiB 63.61 0.93 186 up osd.134 135 hdd 12.65529 1.00000 13 TiB 9.8 TiB 9.6 TiB 14 GiB 85 GiB 2.8 TiB 77.54 1.13 185 up osd.135 136 hdd 12.65529 1.00000 13 TiB 9.3 TiB 9.1 TiB 5.7 GiB 83 GiB 3.3 TiB 73.59 1.07 195 up osd.136 137 hdd 12.65529 1.00000 13 TiB 9.7 TiB 9.4 TiB 5.9 GiB 84 GiB 3.0 TiB 76.46 1.11 187 up osd.137 138 hdd 12.65529 1.00000 13 TiB 10 TiB 9.8 TiB 5.6 GiB 85 GiB 2.7 TiB 78.99 1.15 188 up osd.138 139 hdd 12.65529 1.00000 13 TiB 8.2 TiB 7.9 TiB 5.8 GiB 75 GiB 4.5 TiB 64.51 0.94 185 up osd.139 140 hdd 12.65529 1.00000 13 TiB 9.8 TiB 9.6 TiB 11 GiB 84 GiB 2.8 TiB 77.59 1.13 186 up osd.140 141 hdd 12.65529 1.00000 13 TiB 6.8 TiB 6.6 TiB 16 GiB 69 GiB 5.8 TiB 54.01 0.79 185 up osd.141 142 hdd 12.65529 1.00000 13 TiB 7.9 TiB 7.7 TiB 11 GiB 74 GiB 4.8 TiB 62.41 0.91 192 up osd.142 143 hdd 12.65529 1.00000 13 TiB 8.4 TiB 8.1 TiB 4.4 GiB 75 GiB 4.3 TiB 66.15 0.96 181 up osd.143 TOTAL 1.8 PiB 1.2 PiB 1.2 PiB 1.1 TiB 11 TiB 570 TiB 68.71 MIN/MAX VAR: 0.72/1.27 STDDEV: 8.88 On 2022-05-04 1:26 p.m., Josh Baergen wrote: > [△EXTERNAL] > > > > Hey Dave, > > What version of Ceph is this cluster currently running, and was it > running any previous versions? Were the more full and less full OSDs > deployed on different Ceph versions? > > Could you also provide the output of "ceph osd df tree"? > > Josh > > On Wed, May 4, 2022 at 1:21 PM David Schulz <dschulz@xxxxxxxxxxx> wrote: >> Hi Everyone, >> >> I'm looking for a bit of guidance on a 9 server * 16 OSDs per server= >> 144 OSDs system. >> >> This cluster has 143 OSDs in it but ceph osd df shows that they are very >> unbalanced in their utilization. Some are around 50% full and yet >> others are pushing 85% full. The balancer was on and that was making it >> worse. At one point we had an osd at 89.7% nearly triggering a full >> cluster. >> >> We're running cephfs on erasure coded data pool k=7, m=2. We have two >> options to proceed here and I'm not sure which is appropriate to be done >> next. >> >> Option 1: Add another server -- I'm currently unconvinced that this will >> help much as there is so much skew. I fear that the cluster will just >> continue to be skewed even if we add another 16 or so OSDs. >> >> Option 2: I've read that the the reason for having skewed OSD usage is >> having too few PGs in the pool. Currently our data pool is set to 512 >> but I think this number is inflated when using erasure coding which is >> why it's set only at 512. If I recall k=7 is like having 512*7=3584 >> PGs. And is it safe to suddenly double the number of pgs on a cluster >> that is close to the edge of going full? >> >> Here's a chart of our 12 highest usage OSDs plus the one that was the >> worst early on in this process: >> >> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpasteboard.co%2FqiaYnDxScEHM.png&data=05%7C01%7Cdschulz%40ucalgary.ca%7Cbf36039589704dcd4d4508da2e03f5cf%7Cc609a0eca5e346319686192280bd9151%7C1%7C0%7C637872891789883757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=67SKwyxo%2FAvtzkq%2BsNKY916p91cM1Pbd0zZWUlVyYMc%3D&reserved=0 >> >> Here's a chart showing the distribution of OSD usage: >> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpasteboard.co%2FNKITkmZYaGiH.png&data=05%7C01%7Cdschulz%40ucalgary.ca%7Cbf36039589704dcd4d4508da2e03f5cf%7Cc609a0eca5e346319686192280bd9151%7C1%7C0%7C637872891789883757%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9RLCbe9RL0cEBz6k9N3R19eL0BI0pxm%2BO1xwpYQmFqo%3D&reserved=0 >> >> It is more or less gaussian but the tails seem really long for a cluster >> that's been running the balancer for a long time. >> >> Any guidance would be very much appreciated. >> >> Sincerely >> >> -Dave >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |