Hi Joseph,
Here's some of the larger pools. Notable the largest (pool 51, 32 TiB
CephFS data) doesn't have the highest number of PGs.
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
pool28 28 256 9.9 TiB 2.61M 30 TiB 43.28 13 TiB
pool29 29 256 9.5 TiB 2.48M 28 TiB 42.13 13 TiB
pool36 36 128 6.0 TiB 1.58M 18 TiB 31.67 13 TiB
pool39 39 256 20 TiB 5.20M 60 TiB 60.37 13 TiB
pool43 43 32 1.9 TiB 503.92k 5.7 TiB 12.79 13 TiB
pool46 46 32 1.3 TiB 236.34k 3.9 TiB 9.14 13 TiB
pool47 47 128 4.0 TiB 1.04M 12 TiB 23.35 13 TiB
pool51 51 128 32 TiB 32.47M 55 TiB 58.30 26 TiB
pool53 53 128 3.3 TiB 864.88k 9.9 TiB 20.21 13 TiB
pool57 57 128 14 TiB 3.55M 21 TiB 34.80 26 TiB
It does sounds like I need to increase that, but I had assume the
autoscaler would have produced a warning if that was the case... it
certainly has for some pools in the past, and I've adjusted as per its
recommendation.
Tim.
On Mon, Oct 24, 2022 at 09:24:58AM -0400, Joseph Mundackal wrote:
> Hi Tim,
> You might want to check you pool utilization and see if there are
> enough pg's in that pool. Higher GB per pg can result in this scenario.
>
> I am also assuming that you have the balancer module turn on (ceph balancer
> status) should tell you that as well.
>
> If you have enough pgs in the bigger pools and the balancer module is on,
> you shouldht have to manually reweight osd's.
>
> -Joseph
>
> On Mon, Oct 24, 2022 at 9:13 AM Tim Bishop <tim-lists@xxxxxxxxxxx> wrote:
>
> > Hi all,
> >
> > ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific
> > (stable)
> >
> > We're having an issue with the spread of data across our OSDs. We have
> > 108 OSDs in our cluster, all identical disk size, same number in each
> > server, and the same number of servers in each rack. So I'd hoped we'd
> > end up with a pretty balanced distribution of data across the disks.
> > However, the fullest is at 85% full and the most empty is at 40% full.
> >
> > I've included the osd df output below, along with pool and crush rules.
> >
> > I've also looked at the reweight-by-utilization command which would
> > apparently help:
> >
> > # ceph osd test-reweight-by-utilization
> > moved 16 / 5715 (0.279965%)
> > avg 52.9167
> > stddev 7.20998 -> 7.15325 (expected baseline 7.24063)
> > min osd.45 with 31 -> 31 pgs (0.585827 -> 0.585827 * mean)
> > max osd.22 with 70 -> 68 pgs (1.32283 -> 1.28504 * mean)
> >
> > oload 120
> > max_change 0.05
> > max_change_osds 4
> > average_utilization 0.6229
> > overload_utilization 0.7474
> > osd.22 weight 1.0000 -> 0.9500
> > osd.23 weight 1.0000 -> 0.9500
> > osd.53 weight 1.0000 -> 0.9500
> > osd.78 weight 1.0000 -> 0.9500
> > no change
> >
> > But I'd like to make sure I understand why the problem is occuring
> > first so I can rule out a configuration issue, since it feels like the
> > cluster shouldn't be getting in to this state in the first place.
> >
> > I have some suspicions that the number of PGs may be a bit low on some
> > pools, but autoscale-status is set to "on" or "warn" for every pool, so
> > it's happy with the current numbers. Does it play nice with CephFS?
> >
> > Thanks for any advice.
> > Tim.
> >
> > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
> > AVAIL %USE VAR PGS STATUS
> > 22 hdd 3.63199 1.00000 3.6 TiB 3.1 TiB 3.1 TiB 450 MiB 7.6
> > GiB 557 GiB 85.04 1.37 70 up
> > 23 hdd 3.63199 1.00000 3.6 TiB 2.9 TiB 2.9 TiB 459 MiB 7.5
> > GiB 759 GiB 79.64 1.28 64 up
> > 53 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 703 MiB 8.0
> > GiB 823 GiB 77.91 1.25 66 up
> > 78 hdd 3.63799 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 187 MiB 5.9
> > GiB 851 GiB 77.15 1.24 61 up
> > 26 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 432 MiB 7.7
> > GiB 854 GiB 77.07 1.24 61 up
> > 39 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 503 MiB 7.2
> > GiB 874 GiB 76.55 1.23 65 up
> > 42 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.7 TiB 439 MiB 6.6
> > GiB 909 GiB 75.59 1.21 60 up
> > 101 hdd 3.63820 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 306 MiB 7.1
> > GiB 913 GiB 75.50 1.21 61 up
> > 87 hdd 3.63820 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 539 MiB 7.5
> > GiB 921 GiB 75.27 1.21 61 up
> > 59 hdd 3.63799 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 721 MiB 7.9
> > GiB 957 GiB 74.30 1.19 64 up
> > 79 hdd 3.63799 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 950 MiB 9.0
> > GiB 970 GiB 73.95 1.19 58 up
> > 34 hdd 3.63199 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 202 MiB 6.0
> > GiB 974 GiB 73.85 1.19 57 up
> > 60 hdd 3.63799 1.00000 3.6 TiB 2.7 TiB 2.6 TiB 668 MiB 7.2
> > GiB 1009 GiB 72.91 1.17 59 up
> > 18 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 453 MiB 6.5
> > GiB 1021 GiB 72.59 1.17 60 up
> > 74 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 693 MiB 7.5
> > GiB 1.0 TiB 72.12 1.16 62 up
> > 19 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 655 MiB 7.9
> > GiB 1.0 TiB 71.71 1.15 63 up
> > 69 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 445 MiB 6.2
> > GiB 1.0 TiB 71.70 1.15 65 up
> > 43 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 170 MiB 4.7
> > GiB 1.0 TiB 71.62 1.15 63 up
> > 97 hdd 3.63820 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 276 MiB 5.7
> > GiB 1.0 TiB 71.33 1.15 66 up
> > 67 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 430 MiB 6.3
> > GiB 1.0 TiB 71.22 1.14 54 up
> > 68 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 419 MiB 6.6
> > GiB 1.1 TiB 70.68 1.13 58 up
> > 31 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.5 TiB 419 MiB 5.2
> > GiB 1.1 TiB 70.16 1.13 63 up
> > 48 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.5 TiB 211 MiB 5.0
> > GiB 1.1 TiB 70.13 1.13 56 up
> > 73 hdd 3.63799 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 765 MiB 7.1
> > GiB 1.1 TiB 69.52 1.12 57 up
> > 98 hdd 3.63820 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 552 MiB 7.1
> > GiB 1.1 TiB 68.72 1.10 60 up
> > 58 hdd 3.63799 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 427 MiB 6.3
> > GiB 1.2 TiB 68.39 1.10 53 up
> > 14 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 409 MiB 6.0
> > GiB 1.2 TiB 68.06 1.09 65 up
> > 47 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 166 MiB 5.5
> > GiB 1.2 TiB 67.84 1.09 55 up
> > 9 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 419 MiB 5.9
> > GiB 1.2 TiB 67.78 1.09 58 up
> > 90 hdd 3.63820 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 277 MiB 6.3
> > GiB 1.2 TiB 67.56 1.08 57 up
> > 12 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.4 TiB 649 MiB 6.6
> > GiB 1.2 TiB 67.41 1.08 63 up
> > 49 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 169 MiB 5.1
> > GiB 1.2 TiB 66.95 1.07 54 up
> > 41 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 203 MiB 5.5
> > GiB 1.2 TiB 66.95 1.07 54 up
> > 55 hdd 3.63799 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 179 MiB 5.1
> > GiB 1.2 TiB 66.79 1.07 47 up
> > 2 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 207 MiB 5.2
> > GiB 1.2 TiB 66.79 1.07 59 up
> > 36 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 194 MiB 5.4
> > GiB 1.2 TiB 66.71 1.07 52 up
> > 62 hdd 3.63799 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 167 MiB 5.1
> > GiB 1.2 TiB 66.66 1.07 50 up
> > 17 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 200 MiB 5.0
> > GiB 1.2 TiB 66.53 1.07 53 up
> > 52 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 404 MiB 5.0
> > GiB 1.2 TiB 66.13 1.06 61 up
> > 1 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 438 MiB 5.4
> > GiB 1.2 TiB 66.12 1.06 64 up
> > 96 hdd 3.63820 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 15 MiB 4.2
> > GiB 1.2 TiB 65.92 1.06 53 up
> > 54 hdd 3.63799 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 666 MiB 6.4
> > GiB 1.3 TiB 65.47 1.05 63 up
> > 37 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 412 MiB 6.6
> > GiB 1.3 TiB 64.87 1.04 54 up
> > 93 hdd 3.63820 1.00000 3.6 TiB 2.4 TiB 2.3 TiB 304 MiB 6.0
> > GiB 1.3 TiB 64.63 1.04 57 up
> > 15 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 200 MiB 5.8
> > GiB 1.3 TiB 64.22 1.03 51 up
> > 66 hdd 3.63799 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 427 MiB 5.2
> > GiB 1.3 TiB 63.98 1.03 50 up
> > 28 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 169 MiB 4.6
> > GiB 1.3 TiB 63.75 1.02 54 up
> > 30 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 400 MiB 5.6
> > GiB 1.3 TiB 63.28 1.02 53 up
> > 81 hdd 3.63820 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 55 MiB 5.6
> > GiB 1.3 TiB 63.02 1.01 47 up
> > 21 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 180 MiB 4.2
> > GiB 1.3 TiB 63.00 1.01 51 up
> > 64 hdd 3.63799 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 667 MiB 7.0
> > GiB 1.4 TiB 62.73 1.01 56 up
> > 57 hdd 3.63799 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 167 MiB 4.9
> > GiB 1.4 TiB 62.49 1.00 48 up
> > 8 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 682 MiB 6.9
> > GiB 1.4 TiB 62.25 1.00 52 up
> > 95 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 791 MiB 7.8
> > GiB 1.4 TiB 61.74 0.99 57 up
> > 75 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 1.1 GiB 8.1
> > GiB 1.4 TiB 61.30 0.98 53 up
> > 103 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 34 MiB 4.5
> > GiB 1.4 TiB 61.29 0.98 50 up
> > 89 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 25 MiB 4.3
> > GiB 1.4 TiB 60.99 0.98 53 up
> > 71 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 421 MiB 5.3
> > GiB 1.4 TiB 60.95 0.98 56 up
> > 5 hdd 3.63199 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 158 MiB 3.8
> > GiB 1.4 TiB 60.69 0.97 50 up
> > 61 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 166 MiB 4.6
> > GiB 1.4 TiB 60.29 0.97 55 up
> > 94 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 302 MiB 6.1
> > GiB 1.4 TiB 60.26 0.97 55 up
> > 72 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 170 MiB 4.0
> > GiB 1.5 TiB 59.91 0.96 51 up
> > 91 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 18 MiB 3.9
> > GiB 1.5 TiB 59.75 0.96 51 up
> > 38 hdd 3.63199 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 429 MiB 5.4
> > GiB 1.5 TiB 59.42 0.95 49 up
> > 33 hdd 3.63199 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 980 MiB 6.9
> > GiB 1.5 TiB 59.36 0.95 52 up
> > 3 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 174 MiB 4.5
> > GiB 1.5 TiB 58.79 0.94 49 up
> > 13 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 177 MiB 3.9
> > GiB 1.5 TiB 58.75 0.94 54 up
> > 0 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 179 MiB 4.7
> > GiB 1.5 TiB 58.63 0.94 50 up
> > 77 hdd 3.63799 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 429 MiB 5.3
> > GiB 1.5 TiB 58.59 0.94 50 up
> > 46 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 406 MiB 5.8
> > GiB 1.5 TiB 58.25 0.94 49 up
> > 84 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 849 MiB 7.1
> > GiB 1.5 TiB 57.97 0.93 52 up
> > 63 hdd 3.63799 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 144 MiB 4.3
> > GiB 1.5 TiB 57.94 0.93 55 up
> > 25 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 153 MiB 4.8
> > GiB 1.5 TiB 57.87 0.93 53 up
> > 4 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 682 MiB 6.4
> > GiB 1.5 TiB 57.76 0.93 47 up
> > 102 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 49 MiB 4.9
> > GiB 1.6 TiB 57.25 0.92 45 up
> > 10 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 446 MiB 4.5
> > GiB 1.6 TiB 57.24 0.92 53 up
> > 88 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 550 MiB 6.6
> > GiB 1.6 TiB 56.57 0.91 50 up
> > 92 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 39 MiB 4.5
> > GiB 1.6 TiB 56.53 0.91 44 up
> > 56 hdd 3.63799 1.00000 3.6 TiB 2.1 TiB 2.0 TiB 156 MiB 4.0
> > GiB 1.6 TiB 56.40 0.91 54 up
> > 85 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.0 TiB 35 MiB 4.3
> > GiB 1.6 TiB 56.38 0.91 47 up
> > 40 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 156 MiB 3.9
> > GiB 1.6 TiB 56.22 0.90 47 up
> > 76 hdd 3.63799 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 166 MiB 4.6
> > GiB 1.6 TiB 56.18 0.90 46 up
> > 83 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 35 MiB 4.3
> > GiB 1.6 TiB 56.17 0.90 49 up
> > 50 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 419 MiB 5.2
> > GiB 1.6 TiB 55.29 0.89 51 up
> > 35 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 415 MiB 5.5
> > GiB 1.6 TiB 55.27 0.89 45 up
> > 106 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 266 MiB 4.8
> > GiB 1.6 TiB 55.19 0.89 47 up
> > 44 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 378 MiB 5.2
> > GiB 1.6 TiB 55.11 0.88 45 up
> > 29 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 416 MiB 4.8
> > GiB 1.6 TiB 54.87 0.88 52 up
> > 86 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 559 MiB 6.7
> > GiB 1.6 TiB 54.80 0.88 47 up
> > 82 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 298 MiB 5.2
> > GiB 1.6 TiB 54.67 0.88 49 up
> > 65 hdd 3.63799 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 918 MiB 6.7
> > GiB 1.7 TiB 54.15 0.87 51 up
> > 70 hdd 3.63799 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 675 MiB 5.7
> > GiB 1.7 TiB 53.49 0.86 50 up
> > 11 hdd 3.63199 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 165 MiB 4.6
> > GiB 1.7 TiB 52.97 0.85 46 up
> > 24 hdd 3.63199 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 156 MiB 4.2
> > GiB 1.7 TiB 52.80 0.85 44 up
> > 105 hdd 3.63820 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 38 MiB 4.4
> > GiB 1.7 TiB 52.45 0.84 47 up
> > 104 hdd 3.63820 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 288 MiB 5.5
> > GiB 1.7 TiB 52.09 0.84 38 up
> > 107 hdd 3.63820 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 290 MiB 5.5
> > GiB 1.8 TiB 51.57 0.83 41 up
> > 20 hdd 3.63199 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 381 MiB 5.2
> > GiB 1.8 TiB 51.48 0.83 51 up
> > 32 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 116 MiB 4.1
> > GiB 1.8 TiB 50.37 0.81 48 up
> > 100 hdd 3.63820 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 539 MiB 6.1
> > GiB 1.8 TiB 50.21 0.81 47 up
> > 7 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 173 MiB 3.9
> > GiB 1.8 TiB 50.14 0.81 45 up
> > 51 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 640 MiB 5.9
> > GiB 1.8 TiB 49.97 0.80 46 up
> > 16 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 158 MiB 3.9
> > GiB 1.8 TiB 49.87 0.80 41 up
> > 99 hdd 3.63820 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 258 MiB 4.4
> > GiB 1.8 TiB 49.81 0.80 44 up
> > 6 hdd 3.63199 1.00000 3.6 TiB 1.7 TiB 1.7 TiB 149 MiB 3.8
> > GiB 1.9 TiB 47.96 0.77 42 up
> > 80 hdd 3.63799 1.00000 3.6 TiB 1.7 TiB 1.7 TiB 640 MiB 5.1
> > GiB 1.9 TiB 46.48 0.75 37 up
> > 27 hdd 3.63199 1.00000 3.6 TiB 1.6 TiB 1.6 TiB 338 MiB 3.9
> > GiB 2.1 TiB 42.72 0.69 36 up
> > 45 hdd 3.63199 1.00000 3.6 TiB 1.5 TiB 1.5 TiB 150 MiB 3.2
> > GiB 2.2 TiB 40.28 0.65 31 up
> > TOTAL 393 TiB 245 TiB 244 TiB 38 GiB 605
> > GiB 148 TiB 62.28
> > MIN/MAX VAR: 0.65/1.37 STDDEV: 8.55
> >
> >
> > --- RAW STORAGE ---
> > CLASS SIZE AVAIL USED RAW USED %RAW USED
> > hdd 393 TiB 148 TiB 245 TiB 245 TiB 62.29
> > TOTAL 393 TiB 148 TiB 245 TiB 245 TiB 62.29
> >
> > --- POOLS ---
> > POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
> > pool28 28 256 9.9 TiB 2.61M 30 TiB 43.28 13 TiB
> > pool29 29 256 9.5 TiB 2.48M 28 TiB 42.13 13 TiB
> > pool36 36 128 6.0 TiB 1.58M 18 TiB 31.67 13 TiB
> > pool39 39 256 20 TiB 5.20M 60 TiB 60.37 13 TiB
> > pool43 43 32 1.9 TiB 503.92k 5.7 TiB 12.79 13 TiB
> > pool44 44 16 385 GiB 133.10k 1.1 TiB 2.80 13 TiB
> > pool45 45 32 9.1 KiB 0 27 KiB 0 13 TiB
> > pool46 46 32 1.3 TiB 236.34k 3.9 TiB 9.14 13 TiB
> > pool47 47 128 4.0 TiB 1.04M 12 TiB 23.35 13 TiB
> > pool48 48 32 6.0 MiB 0 18 MiB 0 13 TiB
> > pool49 49 32 8.3 GiB 1.49M 25 GiB 0.06 13 TiB
> > pool50 50 32 3.0 MiB 3 9.2 MiB 0 13 TiB
> > pool51 51 128 32 TiB 32.47M 55 TiB 58.30 26 TiB
> > pool53 52 1 56 KiB 108 169 KiB 0 13 TiB
> > pool53 53 128 3.3 TiB 864.88k 9.9 TiB 20.21 13 TiB
> > pool54 54 32 2.2 MiB 4 7.0 MiB 0 13 TiB
> > pool57 57 128 14 TiB 3.55M 21 TiB 34.80 26 TiB
> >
> >
> > pool 28 'pool28' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 150301
> > flags hashpspool,nearfull stripe_width 0 application rbd
> > pool 29 'pool29' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 150301
> > flags hashpspool,nearfull stripe_width 0 application rbd
> > pool 36 'pool36' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 150301
> > flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application rbd
> > pool 39 'pool39' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 150301
> > flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application rbd
> > pool 43 'pool43' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor
> > 0/140049/140047 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0
> > application rbd
> > pool 44 'pool44' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 150301 lfor
> > 0/66919/66917 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0
> > application rbd
> > pool 45 'pool45' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor
> > 0/58489/67317 flags hashpspool,nearfull stripe_width 0 application rbd
> > pool 46 'pool46' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor
> > 0/66943/66941 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0
> > application rbd
> > pool 47 'pool47' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 150301
> > flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application rbd
> > pool 48 'pool48' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor
> > 0/64382/67327 flags hashpspool,nearfull stripe_width 0 application cephfs
> > pool 49 'pool49' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor
> > 0/66981/67337 flags hashpspool,nearfull stripe_width 0 application cephfs
> > pool 50 'pool50' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301 lfor
> > 0/66733/67322 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0
> > application benchmarks
> > pool 51 'pool51' erasure profile ec42 size 6 min_size 4 crush_rule 1
> > object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change
> > 150301 lfor 0/80117/81361 flags
> > hashpspool,ec_overwrites,nearfull,selfmanaged_snaps stripe_width 16384
> > application cephfs
> > pool 52 'pool52' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 1 pgp_num 1 autoscale_mode warn last_change 150301 flags
> > hashpspool,nearfull stripe_width 0 pg_num_min 1 application mgr_devicehealth
> > pool 53 'pool53' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 150301 lfor
> > 0/92746/101633 flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0
> > application rbd
> > pool 54 'pool54' replicated size 3 min_size 2 crush_rule 0 object_hash
> > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 150301 lfor
> > 0/96348/96346 flags hashpspool,nearfull stripe_width 0 application rbd
> > pool 57 'pool57' erasure profile ec42 size 6 min_size 4 crush_rule 1
> > object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change
> > 150301 lfor 0/96618/101635 flags
> > hashpspool,ec_overwrites,nearfull,selfmanaged_snaps stripe_width 16384
> > application rbd
> >
> > [
> > {
> > "rule_id": 0,
> > "rule_name": "replicated_ruleset",
> > "ruleset": 0,
> > "type": 1,
> > "min_size": 1,
> > "max_size": 10,
> > "steps": [
> > {
> > "op": "take",
> > "item": -1,
> > "item_name": "default"
> > },
> > {
> > "op": "chooseleaf_firstn",
> > "num": 0,
> > "type": "rack"
> > },
> > {
> > "op": "emit"
> > }
> > ]
> > },
> > {
> > "rule_id": 1,
> > "rule_name": "ec42_ruleset",
> > "ruleset": 1,
> > "type": 3,
> > "min_size": 3,
> > "max_size": 6,
> > "steps": [
> > {
> > "op": "set_chooseleaf_tries",
> > "num": 5
> > },
> > {
> > "op": "set_choose_tries",
> > "num": 100
> > },
> > {
> > "op": "take",
> > "item": -1,
> > "item_name": "default"
> > },
> > {
> > "op": "choose_indep",
> > "num": 3,
> > "type": "rack"
> > },
> > {
> > "op": "chooseleaf_indep",
> > "num": 2,
> > "type": "host"
> > },
> > {
> > "op": "emit"
> > }
> > ]
> > },
> > {
> > "rule_id": 2,
> > "rule_name": "erasure-code",
> > "ruleset": 2,
> > "type": 3,
> > "min_size": 3,
> > "max_size": 3,
> > "steps": [
> > {
> > "op": "set_chooseleaf_tries",
> > "num": 5
> > },
> > {
> > "op": "set_choose_tries",
> > "num": 100
> > },
> > {
> > "op": "take",
> > "item": -1,
> > "item_name": "default"
> > },
> > {
> > "op": "chooseleaf_indep",
> > "num": 0,
> > "type": "host"
> > },
> > {
> > "op": "emit"
> > }
> > ]
> > }
> > ]
> >
> >
> > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> > -1 392.58435 root default
> > -8 130.86145 rack R1
> > -4 43.62048 host C1
> > 1 hdd 3.63199 osd.1 up 1.00000 1.00000
> > 12 hdd 3.63199 osd.12 up 1.00000 1.00000
> > 16 hdd 3.63199 osd.16 up 1.00000 1.00000
> > 22 hdd 3.63199 osd.22 up 1.00000 1.00000
> > 27 hdd 3.63199 osd.27 up 1.00000 1.00000
> > 31 hdd 3.63199 osd.31 up 1.00000 1.00000
> > 55 hdd 3.63799 osd.55 up 1.00000 1.00000
> > 57 hdd 3.63799 osd.57 up 1.00000 1.00000
> > 58 hdd 3.63799 osd.58 up 1.00000 1.00000
> > 81 hdd 3.63820 osd.81 up 1.00000 1.00000
> > 82 hdd 3.63820 osd.82 up 1.00000 1.00000
> > 99 hdd 3.63820 osd.99 up 1.00000 1.00000
> > -12 43.62048 host C2
> > 37 hdd 3.63199 osd.37 up 1.00000 1.00000
> > 40 hdd 3.63199 osd.40 up 1.00000 1.00000
> > 44 hdd 3.63199 osd.44 up 1.00000 1.00000
> > 47 hdd 3.63199 osd.47 up 1.00000 1.00000
> > 50 hdd 3.63199 osd.50 up 1.00000 1.00000
> > 53 hdd 3.63199 osd.53 up 1.00000 1.00000
> > 56 hdd 3.63799 osd.56 up 1.00000 1.00000
> > 59 hdd 3.63799 osd.59 up 1.00000 1.00000
> > 60 hdd 3.63799 osd.60 up 1.00000 1.00000
> > 83 hdd 3.63820 osd.83 up 1.00000 1.00000
> > 84 hdd 3.63820 osd.84 up 1.00000 1.00000
> > 100 hdd 3.63820 osd.100 up 1.00000 1.00000
> > -2 43.62048 host C3
> > 0 hdd 3.63199 osd.0 up 1.00000 1.00000
> > 2 hdd 3.63199 osd.2 up 1.00000 1.00000
> > 3 hdd 3.63199 osd.3 up 1.00000 1.00000
> > 4 hdd 3.63199 osd.4 up 1.00000 1.00000
> > 5 hdd 3.63199 osd.5 up 1.00000 1.00000
> > 6 hdd 3.63199 osd.6 up 1.00000 1.00000
> > 54 hdd 3.63799 osd.54 up 1.00000 1.00000
> > 61 hdd 3.63799 osd.61 up 1.00000 1.00000
> > 62 hdd 3.63799 osd.62 up 1.00000 1.00000
> > 87 hdd 3.63820 osd.87 up 1.00000 1.00000
> > 88 hdd 3.63820 osd.88 up 1.00000 1.00000
> > 101 hdd 3.63820 osd.101 up 1.00000 1.00000
> > -9 130.86145 rack R2
> > -3 43.62048 host C4
> > 13 hdd 3.63199 osd.13 up 1.00000 1.00000
> > 20 hdd 3.63199 osd.20 up 1.00000 1.00000
> > 23 hdd 3.63199 osd.23 up 1.00000 1.00000
> > 28 hdd 3.63199 osd.28 up 1.00000 1.00000
> > 33 hdd 3.63199 osd.33 up 1.00000 1.00000
> > 35 hdd 3.63199 osd.35 up 1.00000 1.00000
> > 63 hdd 3.63799 osd.63 up 1.00000 1.00000
> > 65 hdd 3.63799 osd.65 up 1.00000 1.00000
> > 67 hdd 3.63799 osd.67 up 1.00000 1.00000
> > 93 hdd 3.63820 osd.93 up 1.00000 1.00000
> > 94 hdd 3.63820 osd.94 up 1.00000 1.00000
> > 103 hdd 3.63820 osd.103 up 1.00000 1.00000
> > -11 43.62048 host C5
> > 36 hdd 3.63199 osd.36 up 1.00000 1.00000
> > 39 hdd 3.63199 osd.39 up 1.00000 1.00000
> > 42 hdd 3.63199 osd.42 up 1.00000 1.00000
> > 45 hdd 3.63199 osd.45 up 1.00000 1.00000
> > 48 hdd 3.63199 osd.48 up 1.00000 1.00000
> > 51 hdd 3.63199 osd.51 up 1.00000 1.00000
> > 69 hdd 3.63799 osd.69 up 1.00000 1.00000
> > 70 hdd 3.63799 osd.70 up 1.00000 1.00000
> > 71 hdd 3.63799 osd.71 up 1.00000 1.00000
> > 95 hdd 3.63820 osd.95 up 1.00000 1.00000
> > 96 hdd 3.63820 osd.96 up 1.00000 1.00000
> > 104 hdd 3.63820 osd.104 up 1.00000 1.00000
> > -7 43.62048 host C6
> > 10 hdd 3.63199 osd.10 up 1.00000 1.00000
> > 15 hdd 3.63199 osd.15 up 1.00000 1.00000
> > 19 hdd 3.63199 osd.19 up 1.00000 1.00000
> > 25 hdd 3.63199 osd.25 up 1.00000 1.00000
> > 30 hdd 3.63199 osd.30 up 1.00000 1.00000
> > 34 hdd 3.63199 osd.34 up 1.00000 1.00000
> > 64 hdd 3.63799 osd.64 up 1.00000 1.00000
> > 66 hdd 3.63799 osd.66 up 1.00000 1.00000
> > 68 hdd 3.63799 osd.68 up 1.00000 1.00000
> > 91 hdd 3.63820 osd.91 up 1.00000 1.00000
> > 92 hdd 3.63820 osd.92 up 1.00000 1.00000
> > 102 hdd 3.63820 osd.102 up 1.00000 1.00000
> > -10 130.86145 rack R3
> > -5 43.62048 host C7
> > 9 hdd 3.63199 osd.9 up 1.00000 1.00000
> > 14 hdd 3.63199 osd.14 up 1.00000 1.00000
> > 17 hdd 3.63199 osd.17 up 1.00000 1.00000
> > 24 hdd 3.63199 osd.24 up 1.00000 1.00000
> > 29 hdd 3.63199 osd.29 up 1.00000 1.00000
> > 32 hdd 3.63199 osd.32 up 1.00000 1.00000
> > 73 hdd 3.63799 osd.73 up 1.00000 1.00000
> > 76 hdd 3.63799 osd.76 up 1.00000 1.00000
> > 79 hdd 3.63799 osd.79 up 1.00000 1.00000
> > 97 hdd 3.63820 osd.97 up 1.00000 1.00000
> > 98 hdd 3.63820 osd.98 up 1.00000 1.00000
> > 105 hdd 3.63820 osd.105 up 1.00000 1.00000
> > -13 43.62048 host C8
> > 38 hdd 3.63199 osd.38 up 1.00000 1.00000
> > 41 hdd 3.63199 osd.41 up 1.00000 1.00000
> > 43 hdd 3.63199 osd.43 up 1.00000 1.00000
> > 46 hdd 3.63199 osd.46 up 1.00000 1.00000
> > 49 hdd 3.63199 osd.49 up 1.00000 1.00000
> > 52 hdd 3.63199 osd.52 up 1.00000 1.00000
> > 72 hdd 3.63799 osd.72 up 1.00000 1.00000
> > 75 hdd 3.63799 osd.75 up 1.00000 1.00000
> > 78 hdd 3.63799 osd.78 up 1.00000 1.00000
> > 85 hdd 3.63820 osd.85 up 1.00000 1.00000
> > 86 hdd 3.63820 osd.86 up 1.00000 1.00000
> > 107 hdd 3.63820 osd.107 up 1.00000 1.00000
> > -6 43.62048 host C9
> > 7 hdd 3.63199 osd.7 up 1.00000 1.00000
> > 8 hdd 3.63199 osd.8 up 1.00000 1.00000
> > 11 hdd 3.63199 osd.11 up 1.00000 1.00000
> > 18 hdd 3.63199 osd.18 up 1.00000 1.00000
> > 21 hdd 3.63199 osd.21 up 1.00000 1.00000
> > 26 hdd 3.63199 osd.26 up 1.00000 1.00000
> > 74 hdd 3.63799 osd.74 up 1.00000 1.00000
> > 77 hdd 3.63799 osd.77 up 1.00000 1.00000
> > 80 hdd 3.63799 osd.80 up 1.00000 1.00000
> > 89 hdd 3.63820 osd.89 up 1.00000 1.00000
> > 90 hdd 3.63820 osd.90 up 1.00000 1.00000
> > 106 hdd 3.63820 osd.106 up 1.00000 1.00000
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
