Hi Andras,
In 14.2.8 (the latest) version of Nautilus, each pool is balanced
individually. Also, the latest version balances within 5 PGs by
default. That could be a spread of 10, 5 below the mean and 5 above. I
see a spread of 13 in your results. So either you are running an
earlier Nautilus version or the mgr hasn't finished balancing. If you
are running an earlier version, it would balance pools that use the same
crush rule together.
I'm working on a hybrid approach that will balance pools by rule, but
pools will be grouped with a configurable PG count limit.
Currently, we don't balance based on size.
I don't think we know how much space a given PG uses. Unless Bluestore
can or does efficiently track the space utilization, we can't reasonably
balance based on size.
David
On 3/16/20 1:29 PM, Andras Pataki wrote:
I've been trying the upmap balancer on a new Nautilus cluster. We
three main pools, a triple replicated pool (id:1) and two 6+3 erasure
coded pools (id: 4 and 5). The balancer does a very nice job on the
triple replicated pool, but does something strange on the EC pools.
Here is a sample of the PG counts on OSDs (each line) per pool (columns):
OSD 1 2 3 4 5 ALL
0 25 - - 80 34 139
1 26 - - 74 42 142
2 25 - - 74 42 141
3 26 - - 75 41 142
4 25 - - 83 31 139
5 26 - - 80 35 141
6 26 - - 79 36 141
7 26 - - 72 44 142
8 25 - - 74 42 141
9 26 - - 78 38 142
10 26 - - 70 46 142
11 25 - - 78 36 139
12 26 - - 74 41 141
13 26 - - 78 37 141
14 26 - - 76 40 142
15 25 - - 82 33 140
16 26 - - 77 37 140
17 26 - - 73 43 142
18 26 - - 77 37 140
19 26 - - 79 35 140
20 26 - - 74 42 142
21 26 - - 78 36 140
22 26 - - 79 35 140
23 26 - - 81 34 141
24 26 - - 77 38 141
25 26 - - 79 35 140
Pool #1 has 25 or 26 PGs on each OSD extremely consistently. But it
looks like pool #4 and #5 have a lot of variance, but their *sum*
seems to be well balanced (and hence the total PGs per OSD is also
very consistent). As if the balancer was balancing the two pools
together as one. Both pools are EC with 6+3 profile, and use the same
crush rule. Could that be the reason? If not - any ideas?
Also - a separate balancer related question: is there a way to have
the balancer balace the size on each OSD per pool as opposed to the
number of PGs? If not currently - is it something hard to implement?
Andras
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
