Hello everyone,
In my rather heterogeneous setup ...
-1 54.36 root default
-2 42.44 room 2.162
-4 6.09 host le09091
3 2.03 osd.3 up 1
1 2.03 osd.1 up 1
9 2.03 osd.9 up 1
-6 36.35 host le10970
4 7.27 osd.4 up 1
5 7.27 osd.5 up 1
6 7.27 osd.6 up 1
7 7.27 osd.7 up 1
8 7.27 osd.8 up 1
-3 11.92 room 2.166
-5 5.83 host le09086
2 2.03 osd.2 up 1
0 2.03 osd.0 up 1
10 1.77 osd.10 up 1
-7 6.09 host le08544
11 2.03 osd.11 up 1
12 2.03 osd.12 up 1
13 2.03 osd.13 up 1
... using size =3 for all pools,
the OSDs are not filled according to the weights
(which correspond to the disk sizes in TB).
le09086 (osd)
/dev/sdb1 1.8T 1.5T 323G 83% /var/lib/ceph/osd/ceph-10
/dev/sdc1 2.1T 1.7T 408G 81% /var/lib/ceph/osd/ceph-0
/dev/sdd1 2.1T 1.7T 344G 84% /var/lib/ceph/osd/ceph-2
le09091 (osd)
/dev/sda1 2.1T 1.6T 447G 79% /var/lib/ceph/osd/ceph-9
/dev/sdc1 2.1T 1.8T 317G 85% /var/lib/ceph/osd/ceph-3
/dev/sdb1 2.1T 1.7T 384G 82% /var/lib/ceph/osd/ceph-1
le10970 (osd)
/dev/sdd1 7.3T 1.4T 5.9T 19% /var/lib/ceph/osd/ceph-6
/dev/sdf1 7.3T 1.5T 5.9T 21% /var/lib/ceph/osd/ceph-8
/dev/sde1 7.3T 1.6T 5.7T 22% /var/lib/ceph/osd/ceph-7
/dev/sdc1 7.3T 1.4T 6.0T 19% /var/lib/ceph/osd/ceph-5
/dev/sdb1 7.3T 1.5T 5.8T 21% /var/lib/ceph/osd/ceph-4
le08544 (osd)
/dev/sdc1 2.1T 1.6T 443G 79% /var/lib/ceph/osd/ceph-13
/dev/sdb1 2.1T 1.7T 339G 84% /var/lib/ceph/osd/ceph-12
/dev/sda1 2.1T 1.7T 375G 82% /var/lib/ceph/osd/ceph-11
Clearly, I would like le10970 to be selected more often!
Increasing pg(p)_num from 256 to 512 of all the pools
didn't help.
Optimal tunables are used ...
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
... as well as the default algo (straw) and hash (0)
for all buckets. The ruleset is pretty much standard,
too ...
rule replicated_ruleset {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
... so I would assume it should select 3 hosts
(originating from root) according to the weights.
It is ceph version 0.87.
Is it the room bucket, which circumvents
a distribution according tot he weights?
Thanks and all the best,
Frank
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
