Quick napkin math
for your 3 way replicated pool - eg: pool 28 - you have 9.9 TB across 256
pgs ~= 10137 GB across 256 pgs ~= 39 GB per PG
for 4+2 ec on pool 51 - 32 TB across 128 pgs ~= 21768 GB across 128 pgs ~=
256 GB per pg - with the 4+2 profile this should be spread across 4 parts
~= 64 GB per PG
in my experience - a lot of features in ceph work well with replicated
pools and is a little buggy with EC pools. I am speculating that is why you
haven't gotten a warning for it.
On Mon, Oct 24, 2022 at 10:41 AM Tim Bishop <tim-lists@xxxxxxxxxxx> wrote:
> Hi Joseph,
>
> Here's some of the larger pools. Notable the largest (pool 51, 32 TiB
> CephFS data) doesn't have the highest number of PGs.
>
> POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
> pool28 28 256 9.9 TiB 2.61M 30 TiB 43.28 13 TiB
> pool29 29 256 9.5 TiB 2.48M 28 TiB 42.13 13 TiB
> pool36 36 128 6.0 TiB 1.58M 18 TiB 31.67 13 TiB
> pool39 39 256 20 TiB 5.20M 60 TiB 60.37 13 TiB
> pool43 43 32 1.9 TiB 503.92k 5.7 TiB 12.79 13 TiB
> pool46 46 32 1.3 TiB 236.34k 3.9 TiB 9.14 13 TiB
> pool47 47 128 4.0 TiB 1.04M 12 TiB 23.35 13 TiB
> pool51 51 128 32 TiB 32.47M 55 TiB 58.30 26 TiB
> pool53 53 128 3.3 TiB 864.88k 9.9 TiB 20.21 13 TiB
> pool57 57 128 14 TiB 3.55M 21 TiB 34.80 26 TiB
>
> It does sounds like I need to increase that, but I had assume the
> autoscaler would have produced a warning if that was the case... it
> certainly has for some pools in the past, and I've adjusted as per its
> recommendation.
>
> Tim.
>
> On Mon, Oct 24, 2022 at 09:24:58AM -0400, Joseph Mundackal wrote:
> > Hi Tim,
> > You might want to check you pool utilization and see if there are
> > enough pg's in that pool. Higher GB per pg can result in this scenario.
> >
> > I am also assuming that you have the balancer module turn on (ceph
> balancer
> > status) should tell you that as well.
> >
> > If you have enough pgs in the bigger pools and the balancer module is on,
> > you shouldht have to manually reweight osd's.
> >
> > -Joseph
> >
> > On Mon, Oct 24, 2022 at 9:13 AM Tim Bishop <tim-lists@xxxxxxxxxxx>
> wrote:
> >
> > > Hi all,
> > >
> > > ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific
> > > (stable)
> > >
> > > We're having an issue with the spread of data across our OSDs. We have
> > > 108 OSDs in our cluster, all identical disk size, same number in each
> > > server, and the same number of servers in each rack. So I'd hoped we'd
> > > end up with a pretty balanced distribution of data across the disks.
> > > However, the fullest is at 85% full and the most empty is at 40% full.
> > >
> > > I've included the osd df output below, along with pool and crush rules.
> > >
> > > I've also looked at the reweight-by-utilization command which would
> > > apparently help:
> > >
> > > # ceph osd test-reweight-by-utilization
> > > moved 16 / 5715 (0.279965%)
> > > avg 52.9167
> > > stddev 7.20998 -> 7.15325 (expected baseline 7.24063)
> > > min osd.45 with 31 -> 31 pgs (0.585827 -> 0.585827 * mean)
> > > max osd.22 with 70 -> 68 pgs (1.32283 -> 1.28504 * mean)
> > >
> > > oload 120
> > > max_change 0.05
> > > max_change_osds 4
> > > average_utilization 0.6229
> > > overload_utilization 0.7474
> > > osd.22 weight 1.0000 -> 0.9500
> > > osd.23 weight 1.0000 -> 0.9500
> > > osd.53 weight 1.0000 -> 0.9500
> > > osd.78 weight 1.0000 -> 0.9500
> > > no change
> > >
> > > But I'd like to make sure I understand why the problem is occuring
> > > first so I can rule out a configuration issue, since it feels like the
> > > cluster shouldn't be getting in to this state in the first place.
> > >
> > > I have some suspicions that the number of PGs may be a bit low on some
> > > pools, but autoscale-status is set to "on" or "warn" for every pool, so
> > > it's happy with the current numbers. Does it play nice with CephFS?
> > >
> > > Thanks for any advice.
> > > Tim.
> > >
> > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
> > > AVAIL %USE VAR PGS STATUS
> > > 22 hdd 3.63199 1.00000 3.6 TiB 3.1 TiB 3.1 TiB 450 MiB 7.6
> > > GiB 557 GiB 85.04 1.37 70 up
> > > 23 hdd 3.63199 1.00000 3.6 TiB 2.9 TiB 2.9 TiB 459 MiB 7.5
> > > GiB 759 GiB 79.64 1.28 64 up
> > > 53 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 703 MiB 8.0
> > > GiB 823 GiB 77.91 1.25 66 up
> > > 78 hdd 3.63799 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 187 MiB 5.9
> > > GiB 851 GiB 77.15 1.24 61 up
> > > 26 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 432 MiB 7.7
> > > GiB 854 GiB 77.07 1.24 61 up
> > > 39 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.8 TiB 503 MiB 7.2
> > > GiB 874 GiB 76.55 1.23 65 up
> > > 42 hdd 3.63199 1.00000 3.6 TiB 2.8 TiB 2.7 TiB 439 MiB 6.6
> > > GiB 909 GiB 75.59 1.21 60 up
> > > 101 hdd 3.63820 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 306 MiB 7.1
> > > GiB 913 GiB 75.50 1.21 61 up
> > > 87 hdd 3.63820 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 539 MiB 7.5
> > > GiB 921 GiB 75.27 1.21 61 up
> > > 59 hdd 3.63799 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 721 MiB 7.9
> > > GiB 957 GiB 74.30 1.19 64 up
> > > 79 hdd 3.63799 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 950 MiB 9.0
> > > GiB 970 GiB 73.95 1.19 58 up
> > > 34 hdd 3.63199 1.00000 3.6 TiB 2.7 TiB 2.7 TiB 202 MiB 6.0
> > > GiB 974 GiB 73.85 1.19 57 up
> > > 60 hdd 3.63799 1.00000 3.6 TiB 2.7 TiB 2.6 TiB 668 MiB 7.2
> > > GiB 1009 GiB 72.91 1.17 59 up
> > > 18 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 453 MiB 6.5
> > > GiB 1021 GiB 72.59 1.17 60 up
> > > 74 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 693 MiB 7.5
> > > GiB 1.0 TiB 72.12 1.16 62 up
> > > 19 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 655 MiB 7.9
> > > GiB 1.0 TiB 71.71 1.15 63 up
> > > 69 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 445 MiB 6.2
> > > GiB 1.0 TiB 71.70 1.15 65 up
> > > 43 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 170 MiB 4.7
> > > GiB 1.0 TiB 71.62 1.15 63 up
> > > 97 hdd 3.63820 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 276 MiB 5.7
> > > GiB 1.0 TiB 71.33 1.15 66 up
> > > 67 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 430 MiB 6.3
> > > GiB 1.0 TiB 71.22 1.14 54 up
> > > 68 hdd 3.63799 1.00000 3.6 TiB 2.6 TiB 2.6 TiB 419 MiB 6.6
> > > GiB 1.1 TiB 70.68 1.13 58 up
> > > 31 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.5 TiB 419 MiB 5.2
> > > GiB 1.1 TiB 70.16 1.13 63 up
> > > 48 hdd 3.63199 1.00000 3.6 TiB 2.6 TiB 2.5 TiB 211 MiB 5.0
> > > GiB 1.1 TiB 70.13 1.13 56 up
> > > 73 hdd 3.63799 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 765 MiB 7.1
> > > GiB 1.1 TiB 69.52 1.12 57 up
> > > 98 hdd 3.63820 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 552 MiB 7.1
> > > GiB 1.1 TiB 68.72 1.10 60 up
> > > 58 hdd 3.63799 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 427 MiB 6.3
> > > GiB 1.2 TiB 68.39 1.10 53 up
> > > 14 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 409 MiB 6.0
> > > GiB 1.2 TiB 68.06 1.09 65 up
> > > 47 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 166 MiB 5.5
> > > GiB 1.2 TiB 67.84 1.09 55 up
> > > 9 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 419 MiB 5.9
> > > GiB 1.2 TiB 67.78 1.09 58 up
> > > 90 hdd 3.63820 1.00000 3.6 TiB 2.5 TiB 2.5 TiB 277 MiB 6.3
> > > GiB 1.2 TiB 67.56 1.08 57 up
> > > 12 hdd 3.63199 1.00000 3.6 TiB 2.5 TiB 2.4 TiB 649 MiB 6.6
> > > GiB 1.2 TiB 67.41 1.08 63 up
> > > 49 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 169 MiB 5.1
> > > GiB 1.2 TiB 66.95 1.07 54 up
> > > 41 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 203 MiB 5.5
> > > GiB 1.2 TiB 66.95 1.07 54 up
> > > 55 hdd 3.63799 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 179 MiB 5.1
> > > GiB 1.2 TiB 66.79 1.07 47 up
> > > 2 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 207 MiB 5.2
> > > GiB 1.2 TiB 66.79 1.07 59 up
> > > 36 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 194 MiB 5.4
> > > GiB 1.2 TiB 66.71 1.07 52 up
> > > 62 hdd 3.63799 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 167 MiB 5.1
> > > GiB 1.2 TiB 66.66 1.07 50 up
> > > 17 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 200 MiB 5.0
> > > GiB 1.2 TiB 66.53 1.07 53 up
> > > 52 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 404 MiB 5.0
> > > GiB 1.2 TiB 66.13 1.06 61 up
> > > 1 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 438 MiB 5.4
> > > GiB 1.2 TiB 66.12 1.06 64 up
> > > 96 hdd 3.63820 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 15 MiB 4.2
> > > GiB 1.2 TiB 65.92 1.06 53 up
> > > 54 hdd 3.63799 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 666 MiB 6.4
> > > GiB 1.3 TiB 65.47 1.05 63 up
> > > 37 hdd 3.63199 1.00000 3.6 TiB 2.4 TiB 2.4 TiB 412 MiB 6.6
> > > GiB 1.3 TiB 64.87 1.04 54 up
> > > 93 hdd 3.63820 1.00000 3.6 TiB 2.4 TiB 2.3 TiB 304 MiB 6.0
> > > GiB 1.3 TiB 64.63 1.04 57 up
> > > 15 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 200 MiB 5.8
> > > GiB 1.3 TiB 64.22 1.03 51 up
> > > 66 hdd 3.63799 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 427 MiB 5.2
> > > GiB 1.3 TiB 63.98 1.03 50 up
> > > 28 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 169 MiB 4.6
> > > GiB 1.3 TiB 63.75 1.02 54 up
> > > 30 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 400 MiB 5.6
> > > GiB 1.3 TiB 63.28 1.02 53 up
> > > 81 hdd 3.63820 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 55 MiB 5.6
> > > GiB 1.3 TiB 63.02 1.01 47 up
> > > 21 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 180 MiB 4.2
> > > GiB 1.3 TiB 63.00 1.01 51 up
> > > 64 hdd 3.63799 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 667 MiB 7.0
> > > GiB 1.4 TiB 62.73 1.01 56 up
> > > 57 hdd 3.63799 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 167 MiB 4.9
> > > GiB 1.4 TiB 62.49 1.00 48 up
> > > 8 hdd 3.63199 1.00000 3.6 TiB 2.3 TiB 2.3 TiB 682 MiB 6.9
> > > GiB 1.4 TiB 62.25 1.00 52 up
> > > 95 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 791 MiB 7.8
> > > GiB 1.4 TiB 61.74 0.99 57 up
> > > 75 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 1.1 GiB 8.1
> > > GiB 1.4 TiB 61.30 0.98 53 up
> > > 103 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 34 MiB 4.5
> > > GiB 1.4 TiB 61.29 0.98 50 up
> > > 89 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 25 MiB 4.3
> > > GiB 1.4 TiB 60.99 0.98 53 up
> > > 71 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 421 MiB 5.3
> > > GiB 1.4 TiB 60.95 0.98 56 up
> > > 5 hdd 3.63199 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 158 MiB 3.8
> > > GiB 1.4 TiB 60.69 0.97 50 up
> > > 61 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 166 MiB 4.6
> > > GiB 1.4 TiB 60.29 0.97 55 up
> > > 94 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 302 MiB 6.1
> > > GiB 1.4 TiB 60.26 0.97 55 up
> > > 72 hdd 3.63799 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 170 MiB 4.0
> > > GiB 1.5 TiB 59.91 0.96 51 up
> > > 91 hdd 3.63820 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 18 MiB 3.9
> > > GiB 1.5 TiB 59.75 0.96 51 up
> > > 38 hdd 3.63199 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 429 MiB 5.4
> > > GiB 1.5 TiB 59.42 0.95 49 up
> > > 33 hdd 3.63199 1.00000 3.6 TiB 2.2 TiB 2.2 TiB 980 MiB 6.9
> > > GiB 1.5 TiB 59.36 0.95 52 up
> > > 3 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 174 MiB 4.5
> > > GiB 1.5 TiB 58.79 0.94 49 up
> > > 13 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 177 MiB 3.9
> > > GiB 1.5 TiB 58.75 0.94 54 up
> > > 0 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 179 MiB 4.7
> > > GiB 1.5 TiB 58.63 0.94 50 up
> > > 77 hdd 3.63799 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 429 MiB 5.3
> > > GiB 1.5 TiB 58.59 0.94 50 up
> > > 46 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 406 MiB 5.8
> > > GiB 1.5 TiB 58.25 0.94 49 up
> > > 84 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 849 MiB 7.1
> > > GiB 1.5 TiB 57.97 0.93 52 up
> > > 63 hdd 3.63799 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 144 MiB 4.3
> > > GiB 1.5 TiB 57.94 0.93 55 up
> > > 25 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 153 MiB 4.8
> > > GiB 1.5 TiB 57.87 0.93 53 up
> > > 4 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 682 MiB 6.4
> > > GiB 1.5 TiB 57.76 0.93 47 up
> > > 102 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 49 MiB 4.9
> > > GiB 1.6 TiB 57.25 0.92 45 up
> > > 10 hdd 3.63199 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 446 MiB 4.5
> > > GiB 1.6 TiB 57.24 0.92 53 up
> > > 88 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 550 MiB 6.6
> > > GiB 1.6 TiB 56.57 0.91 50 up
> > > 92 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.1 TiB 39 MiB 4.5
> > > GiB 1.6 TiB 56.53 0.91 44 up
> > > 56 hdd 3.63799 1.00000 3.6 TiB 2.1 TiB 2.0 TiB 156 MiB 4.0
> > > GiB 1.6 TiB 56.40 0.91 54 up
> > > 85 hdd 3.63820 1.00000 3.6 TiB 2.1 TiB 2.0 TiB 35 MiB 4.3
> > > GiB 1.6 TiB 56.38 0.91 47 up
> > > 40 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 156 MiB 3.9
> > > GiB 1.6 TiB 56.22 0.90 47 up
> > > 76 hdd 3.63799 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 166 MiB 4.6
> > > GiB 1.6 TiB 56.18 0.90 46 up
> > > 83 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 35 MiB 4.3
> > > GiB 1.6 TiB 56.17 0.90 49 up
> > > 50 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 419 MiB 5.2
> > > GiB 1.6 TiB 55.29 0.89 51 up
> > > 35 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 415 MiB 5.5
> > > GiB 1.6 TiB 55.27 0.89 45 up
> > > 106 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 266 MiB 4.8
> > > GiB 1.6 TiB 55.19 0.89 47 up
> > > 44 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 378 MiB 5.2
> > > GiB 1.6 TiB 55.11 0.88 45 up
> > > 29 hdd 3.63199 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 416 MiB 4.8
> > > GiB 1.6 TiB 54.87 0.88 52 up
> > > 86 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 559 MiB 6.7
> > > GiB 1.6 TiB 54.80 0.88 47 up
> > > 82 hdd 3.63820 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 298 MiB 5.2
> > > GiB 1.6 TiB 54.67 0.88 49 up
> > > 65 hdd 3.63799 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 918 MiB 6.7
> > > GiB 1.7 TiB 54.15 0.87 51 up
> > > 70 hdd 3.63799 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 675 MiB 5.7
> > > GiB 1.7 TiB 53.49 0.86 50 up
> > > 11 hdd 3.63199 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 165 MiB 4.6
> > > GiB 1.7 TiB 52.97 0.85 46 up
> > > 24 hdd 3.63199 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 156 MiB 4.2
> > > GiB 1.7 TiB 52.80 0.85 44 up
> > > 105 hdd 3.63820 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 38 MiB 4.4
> > > GiB 1.7 TiB 52.45 0.84 47 up
> > > 104 hdd 3.63820 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 288 MiB 5.5
> > > GiB 1.7 TiB 52.09 0.84 38 up
> > > 107 hdd 3.63820 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 290 MiB 5.5
> > > GiB 1.8 TiB 51.57 0.83 41 up
> > > 20 hdd 3.63199 1.00000 3.6 TiB 1.9 TiB 1.9 TiB 381 MiB 5.2
> > > GiB 1.8 TiB 51.48 0.83 51 up
> > > 32 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 116 MiB 4.1
> > > GiB 1.8 TiB 50.37 0.81 48 up
> > > 100 hdd 3.63820 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 539 MiB 6.1
> > > GiB 1.8 TiB 50.21 0.81 47 up
> > > 7 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 173 MiB 3.9
> > > GiB 1.8 TiB 50.14 0.81 45 up
> > > 51 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 640 MiB 5.9
> > > GiB 1.8 TiB 49.97 0.80 46 up
> > > 16 hdd 3.63199 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 158 MiB 3.9
> > > GiB 1.8 TiB 49.87 0.80 41 up
> > > 99 hdd 3.63820 1.00000 3.6 TiB 1.8 TiB 1.8 TiB 258 MiB 4.4
> > > GiB 1.8 TiB 49.81 0.80 44 up
> > > 6 hdd 3.63199 1.00000 3.6 TiB 1.7 TiB 1.7 TiB 149 MiB 3.8
> > > GiB 1.9 TiB 47.96 0.77 42 up
> > > 80 hdd 3.63799 1.00000 3.6 TiB 1.7 TiB 1.7 TiB 640 MiB 5.1
> > > GiB 1.9 TiB 46.48 0.75 37 up
> > > 27 hdd 3.63199 1.00000 3.6 TiB 1.6 TiB 1.6 TiB 338 MiB 3.9
> > > GiB 2.1 TiB 42.72 0.69 36 up
> > > 45 hdd 3.63199 1.00000 3.6 TiB 1.5 TiB 1.5 TiB 150 MiB 3.2
> > > GiB 2.2 TiB 40.28 0.65 31 up
> > > TOTAL 393 TiB 245 TiB 244 TiB 38 GiB 605
> > > GiB 148 TiB 62.28
> > > MIN/MAX VAR: 0.65/1.37 STDDEV: 8.55
> > >
> > >
> > > --- RAW STORAGE ---
> > > CLASS SIZE AVAIL USED RAW USED %RAW USED
> > > hdd 393 TiB 148 TiB 245 TiB 245 TiB 62.29
> > > TOTAL 393 TiB 148 TiB 245 TiB 245 TiB 62.29
> > >
> > > --- POOLS ---
> > > POOL ID PGS STORED OBJECTS USED %USED MAX
> AVAIL
> > > pool28 28 256 9.9 TiB 2.61M 30 TiB 43.28
> 13 TiB
> > > pool29 29 256 9.5 TiB 2.48M 28 TiB 42.13
> 13 TiB
> > > pool36 36 128 6.0 TiB 1.58M 18 TiB 31.67
> 13 TiB
> > > pool39 39 256 20 TiB 5.20M 60 TiB 60.37
> 13 TiB
> > > pool43 43 32 1.9 TiB 503.92k 5.7 TiB 12.79
> 13 TiB
> > > pool44 44 16 385 GiB 133.10k 1.1 TiB 2.80
> 13 TiB
> > > pool45 45 32 9.1 KiB 0 27 KiB 0
> 13 TiB
> > > pool46 46 32 1.3 TiB 236.34k 3.9 TiB 9.14
> 13 TiB
> > > pool47 47 128 4.0 TiB 1.04M 12 TiB 23.35
> 13 TiB
> > > pool48 48 32 6.0 MiB 0 18 MiB 0
> 13 TiB
> > > pool49 49 32 8.3 GiB 1.49M 25 GiB 0.06
> 13 TiB
> > > pool50 50 32 3.0 MiB 3 9.2 MiB 0
> 13 TiB
> > > pool51 51 128 32 TiB 32.47M 55 TiB 58.30
> 26 TiB
> > > pool53 52 1 56 KiB 108 169 KiB 0
> 13 TiB
> > > pool53 53 128 3.3 TiB 864.88k 9.9 TiB 20.21
> 13 TiB
> > > pool54 54 32 2.2 MiB 4 7.0 MiB 0
> 13 TiB
> > > pool57 57 128 14 TiB 3.55M 21 TiB 34.80
> 26 TiB
> > >
> > >
> > > pool 28 'pool28' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 150301
> > > flags hashpspool,nearfull stripe_width 0 application rbd
> > > pool 29 'pool29' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 150301
> > > flags hashpspool,nearfull stripe_width 0 application rbd
> > > pool 36 'pool36' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 150301
> > > flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application
> rbd
> > > pool 39 'pool39' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 256 pgp_num 256 autoscale_mode warn last_change 150301
> > > flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application
> rbd
> > > pool 43 'pool43' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301
> lfor
> > > 0/140049/140047 flags hashpspool,nearfull,selfmanaged_snaps
> stripe_width 0
> > > application rbd
> > > pool 44 'pool44' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 16 pgp_num 16 autoscale_mode warn last_change 150301
> lfor
> > > 0/66919/66917 flags hashpspool,nearfull,selfmanaged_snaps stripe_width
> 0
> > > application rbd
> > > pool 45 'pool45' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301
> lfor
> > > 0/58489/67317 flags hashpspool,nearfull stripe_width 0 application rbd
> > > pool 46 'pool46' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301
> lfor
> > > 0/66943/66941 flags hashpspool,nearfull,selfmanaged_snaps stripe_width
> 0
> > > application rbd
> > > pool 47 'pool47' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 128 pgp_num 128 autoscale_mode warn last_change 150301
> > > flags hashpspool,nearfull,selfmanaged_snaps stripe_width 0 application
> rbd
> > > pool 48 'pool48' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301
> lfor
> > > 0/64382/67327 flags hashpspool,nearfull stripe_width 0 application
> cephfs
> > > pool 49 'pool49' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301
> lfor
> > > 0/66981/67337 flags hashpspool,nearfull stripe_width 0 application
> cephfs
> > > pool 50 'pool50' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 150301
> lfor
> > > 0/66733/67322 flags hashpspool,nearfull,selfmanaged_snaps stripe_width
> 0
> > > application benchmarks
> > > pool 51 'pool51' erasure profile ec42 size 6 min_size 4 crush_rule 1
> > > object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode warn
> last_change
> > > 150301 lfor 0/80117/81361 flags
> > > hashpspool,ec_overwrites,nearfull,selfmanaged_snaps stripe_width 16384
> > > application cephfs
> > > pool 52 'pool52' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 1 pgp_num 1 autoscale_mode warn last_change 150301
> flags
> > > hashpspool,nearfull stripe_width 0 pg_num_min 1 application
> mgr_devicehealth
> > > pool 53 'pool53' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 150301
> lfor
> > > 0/92746/101633 flags hashpspool,nearfull,selfmanaged_snaps
> stripe_width 0
> > > application rbd
> > > pool 54 'pool54' replicated size 3 min_size 2 crush_rule 0 object_hash
> > > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 150301 lfor
> > > 0/96348/96346 flags hashpspool,nearfull stripe_width 0 application rbd
> > > pool 57 'pool57' erasure profile ec42 size 6 min_size 4 crush_rule 1
> > > object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on
> last_change
> > > 150301 lfor 0/96618/101635 flags
> > > hashpspool,ec_overwrites,nearfull,selfmanaged_snaps stripe_width 16384
> > > application rbd
> > >
> > > [
> > > {
> > > "rule_id": 0,
> > > "rule_name": "replicated_ruleset",
> > > "ruleset": 0,
> > > "type": 1,
> > > "min_size": 1,
> > > "max_size": 10,
> > > "steps": [
> > > {
> > > "op": "take",
> > > "item": -1,
> > > "item_name": "default"
> > > },
> > > {
> > > "op": "chooseleaf_firstn",
> > > "num": 0,
> > > "type": "rack"
> > > },
> > > {
> > > "op": "emit"
> > > }
> > > ]
> > > },
> > > {
> > > "rule_id": 1,
> > > "rule_name": "ec42_ruleset",
> > > "ruleset": 1,
> > > "type": 3,
> > > "min_size": 3,
> > > "max_size": 6,
> > > "steps": [
> > > {
> > > "op": "set_chooseleaf_tries",
> > > "num": 5
> > > },
> > > {
> > > "op": "set_choose_tries",
> > > "num": 100
> > > },
> > > {
> > > "op": "take",
> > > "item": -1,
> > > "item_name": "default"
> > > },
> > > {
> > > "op": "choose_indep",
> > > "num": 3,
> > > "type": "rack"
> > > },
> > > {
> > > "op": "chooseleaf_indep",
> > > "num": 2,
> > > "type": "host"
> > > },
> > > {
> > > "op": "emit"
> > > }
> > > ]
> > > },
> > > {
> > > "rule_id": 2,
> > > "rule_name": "erasure-code",
> > > "ruleset": 2,
> > > "type": 3,
> > > "min_size": 3,
> > > "max_size": 3,
> > > "steps": [
> > > {
> > > "op": "set_chooseleaf_tries",
> > > "num": 5
> > > },
> > > {
> > > "op": "set_choose_tries",
> > > "num": 100
> > > },
> > > {
> > > "op": "take",
> > > "item": -1,
> > > "item_name": "default"
> > > },
> > > {
> > > "op": "chooseleaf_indep",
> > > "num": 0,
> > > "type": "host"
> > > },
> > > {
> > > "op": "emit"
> > > }
> > > ]
> > > }
> > > ]
> > >
> > >
> > > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> > > -1 392.58435 root default
> > > -8 130.86145 rack R1
> > > -4 43.62048 host C1
> > > 1 hdd 3.63199 osd.1 up 1.00000 1.00000
> > > 12 hdd 3.63199 osd.12 up 1.00000 1.00000
> > > 16 hdd 3.63199 osd.16 up 1.00000 1.00000
> > > 22 hdd 3.63199 osd.22 up 1.00000 1.00000
> > > 27 hdd 3.63199 osd.27 up 1.00000 1.00000
> > > 31 hdd 3.63199 osd.31 up 1.00000 1.00000
> > > 55 hdd 3.63799 osd.55 up 1.00000 1.00000
> > > 57 hdd 3.63799 osd.57 up 1.00000 1.00000
> > > 58 hdd 3.63799 osd.58 up 1.00000 1.00000
> > > 81 hdd 3.63820 osd.81 up 1.00000 1.00000
> > > 82 hdd 3.63820 osd.82 up 1.00000 1.00000
> > > 99 hdd 3.63820 osd.99 up 1.00000 1.00000
> > > -12 43.62048 host C2
> > > 37 hdd 3.63199 osd.37 up 1.00000 1.00000
> > > 40 hdd 3.63199 osd.40 up 1.00000 1.00000
> > > 44 hdd 3.63199 osd.44 up 1.00000 1.00000
> > > 47 hdd 3.63199 osd.47 up 1.00000 1.00000
> > > 50 hdd 3.63199 osd.50 up 1.00000 1.00000
> > > 53 hdd 3.63199 osd.53 up 1.00000 1.00000
> > > 56 hdd 3.63799 osd.56 up 1.00000 1.00000
> > > 59 hdd 3.63799 osd.59 up 1.00000 1.00000
> > > 60 hdd 3.63799 osd.60 up 1.00000 1.00000
> > > 83 hdd 3.63820 osd.83 up 1.00000 1.00000
> > > 84 hdd 3.63820 osd.84 up 1.00000 1.00000
> > > 100 hdd 3.63820 osd.100 up 1.00000 1.00000
> > > -2 43.62048 host C3
> > > 0 hdd 3.63199 osd.0 up 1.00000 1.00000
> > > 2 hdd 3.63199 osd.2 up 1.00000 1.00000
> > > 3 hdd 3.63199 osd.3 up 1.00000 1.00000
> > > 4 hdd 3.63199 osd.4 up 1.00000 1.00000
> > > 5 hdd 3.63199 osd.5 up 1.00000 1.00000
> > > 6 hdd 3.63199 osd.6 up 1.00000 1.00000
> > > 54 hdd 3.63799 osd.54 up 1.00000 1.00000
> > > 61 hdd 3.63799 osd.61 up 1.00000 1.00000
> > > 62 hdd 3.63799 osd.62 up 1.00000 1.00000
> > > 87 hdd 3.63820 osd.87 up 1.00000 1.00000
> > > 88 hdd 3.63820 osd.88 up 1.00000 1.00000
> > > 101 hdd 3.63820 osd.101 up 1.00000 1.00000
> > > -9 130.86145 rack R2
> > > -3 43.62048 host C4
> > > 13 hdd 3.63199 osd.13 up 1.00000 1.00000
> > > 20 hdd 3.63199 osd.20 up 1.00000 1.00000
> > > 23 hdd 3.63199 osd.23 up 1.00000 1.00000
> > > 28 hdd 3.63199 osd.28 up 1.00000 1.00000
> > > 33 hdd 3.63199 osd.33 up 1.00000 1.00000
> > > 35 hdd 3.63199 osd.35 up 1.00000 1.00000
> > > 63 hdd 3.63799 osd.63 up 1.00000 1.00000
> > > 65 hdd 3.63799 osd.65 up 1.00000 1.00000
> > > 67 hdd 3.63799 osd.67 up 1.00000 1.00000
> > > 93 hdd 3.63820 osd.93 up 1.00000 1.00000
> > > 94 hdd 3.63820 osd.94 up 1.00000 1.00000
> > > 103 hdd 3.63820 osd.103 up 1.00000 1.00000
> > > -11 43.62048 host C5
> > > 36 hdd 3.63199 osd.36 up 1.00000 1.00000
> > > 39 hdd 3.63199 osd.39 up 1.00000 1.00000
> > > 42 hdd 3.63199 osd.42 up 1.00000 1.00000
> > > 45 hdd 3.63199 osd.45 up 1.00000 1.00000
> > > 48 hdd 3.63199 osd.48 up 1.00000 1.00000
> > > 51 hdd 3.63199 osd.51 up 1.00000 1.00000
> > > 69 hdd 3.63799 osd.69 up 1.00000 1.00000
> > > 70 hdd 3.63799 osd.70 up 1.00000 1.00000
> > > 71 hdd 3.63799 osd.71 up 1.00000 1.00000
> > > 95 hdd 3.63820 osd.95 up 1.00000 1.00000
> > > 96 hdd 3.63820 osd.96 up 1.00000 1.00000
> > > 104 hdd 3.63820 osd.104 up 1.00000 1.00000
> > > -7 43.62048 host C6
> > > 10 hdd 3.63199 osd.10 up 1.00000 1.00000
> > > 15 hdd 3.63199 osd.15 up 1.00000 1.00000
> > > 19 hdd 3.63199 osd.19 up 1.00000 1.00000
> > > 25 hdd 3.63199 osd.25 up 1.00000 1.00000
> > > 30 hdd 3.63199 osd.30 up 1.00000 1.00000
> > > 34 hdd 3.63199 osd.34 up 1.00000 1.00000
> > > 64 hdd 3.63799 osd.64 up 1.00000 1.00000
> > > 66 hdd 3.63799 osd.66 up 1.00000 1.00000
> > > 68 hdd 3.63799 osd.68 up 1.00000 1.00000
> > > 91 hdd 3.63820 osd.91 up 1.00000 1.00000
> > > 92 hdd 3.63820 osd.92 up 1.00000 1.00000
> > > 102 hdd 3.63820 osd.102 up 1.00000 1.00000
> > > -10 130.86145 rack R3
> > > -5 43.62048 host C7
> > > 9 hdd 3.63199 osd.9 up 1.00000 1.00000
> > > 14 hdd 3.63199 osd.14 up 1.00000 1.00000
> > > 17 hdd 3.63199 osd.17 up 1.00000 1.00000
> > > 24 hdd 3.63199 osd.24 up 1.00000 1.00000
> > > 29 hdd 3.63199 osd.29 up 1.00000 1.00000
> > > 32 hdd 3.63199 osd.32 up 1.00000 1.00000
> > > 73 hdd 3.63799 osd.73 up 1.00000 1.00000
> > > 76 hdd 3.63799 osd.76 up 1.00000 1.00000
> > > 79 hdd 3.63799 osd.79 up 1.00000 1.00000
> > > 97 hdd 3.63820 osd.97 up 1.00000 1.00000
> > > 98 hdd 3.63820 osd.98 up 1.00000 1.00000
> > > 105 hdd 3.63820 osd.105 up 1.00000 1.00000
> > > -13 43.62048 host C8
> > > 38 hdd 3.63199 osd.38 up 1.00000 1.00000
> > > 41 hdd 3.63199 osd.41 up 1.00000 1.00000
> > > 43 hdd 3.63199 osd.43 up 1.00000 1.00000
> > > 46 hdd 3.63199 osd.46 up 1.00000 1.00000
> > > 49 hdd 3.63199 osd.49 up 1.00000 1.00000
> > > 52 hdd 3.63199 osd.52 up 1.00000 1.00000
> > > 72 hdd 3.63799 osd.72 up 1.00000 1.00000
> > > 75 hdd 3.63799 osd.75 up 1.00000 1.00000
> > > 78 hdd 3.63799 osd.78 up 1.00000 1.00000
> > > 85 hdd 3.63820 osd.85 up 1.00000 1.00000
> > > 86 hdd 3.63820 osd.86 up 1.00000 1.00000
> > > 107 hdd 3.63820 osd.107 up 1.00000 1.00000
> > > -6 43.62048 host C9
> > > 7 hdd 3.63199 osd.7 up 1.00000 1.00000
> > > 8 hdd 3.63199 osd.8 up 1.00000 1.00000
> > > 11 hdd 3.63199 osd.11 up 1.00000 1.00000
> > > 18 hdd 3.63199 osd.18 up 1.00000 1.00000
> > > 21 hdd 3.63199 osd.21 up 1.00000 1.00000
> > > 26 hdd 3.63199 osd.26 up 1.00000 1.00000
> > > 74 hdd 3.63799 osd.74 up 1.00000 1.00000
> > > 77 hdd 3.63799 osd.77 up 1.00000 1.00000
> > > 80 hdd 3.63799 osd.80 up 1.00000 1.00000
> > > 89 hdd 3.63820 osd.89 up 1.00000 1.00000
> > > 90 hdd 3.63820 osd.90 up 1.00000 1.00000
> > > 106 hdd 3.63820 osd.106 up 1.00000 1.00000
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
