Hello Christian,
hello everyone,
> That would be my guess, as a similar 4 node cluster here with a flat, host
> only topology is having its data distributed as expected by the weights.
To verify this, I eliminated the room buckets.
Now all hosts are located within the root bucket.
(Same tree as the one below, just without room buckets)
However, after rebalancing, the data is still not distributed according to the weights.
(Almost the same as the one below, le10970 ~20%, all others ~80+-6%)
Any further ideas what I am doing wrong?
Thanks, Frank
> > In my rather heterogeneous setup ...
> >
> > -1 54.36 root default
> > -2 42.44 room 2.162
> > -4 6.09 host le09091
> > 3 2.03 osd.3 up 1
> > 1 2.03 osd.1 up 1
> > 9 2.03 osd.9 up 1
> > -6 36.35 host le10970
> > 4 7.27 osd.4 up 1
> > 5 7.27 osd.5 up 1
> > 6 7.27 osd.6 up 1
> > 7 7.27 osd.7 up 1
> > 8 7.27 osd.8 up 1
> > -3 11.92 room 2.166
> > -5 5.83 host le09086
> > 2 2.03 osd.2 up 1
> > 0 2.03 osd.0 up 1
> > 10 1.77 osd.10 up 1
> > -7 6.09 host le08544
> > 11 2.03 osd.11 up 1
> > 12 2.03 osd.12 up 1
> > 13 2.03 osd.13 up 1
> >
> > ... using size =3 for all pools,
> > the OSDs are not filled according to the weights
> > (which correspond to the disk sizes in TB).
> >
> > le09086 (osd)
> > /dev/sdb1 1.8T 1.5T 323G 83% /var/lib/ceph/osd/ceph-10
> > /dev/sdc1 2.1T 1.7T 408G 81% /var/lib/ceph/osd/ceph-0
> > /dev/sdd1 2.1T 1.7T 344G 84% /var/lib/ceph/osd/ceph-2
> > le09091 (osd)
> > /dev/sda1 2.1T 1.6T 447G 79% /var/lib/ceph/osd/ceph-9
> > /dev/sdc1 2.1T 1.8T 317G 85% /var/lib/ceph/osd/ceph-3
> > /dev/sdb1 2.1T 1.7T 384G 82% /var/lib/ceph/osd/ceph-1
> > le10970 (osd)
> > /dev/sdd1 7.3T 1.4T 5.9T 19% /var/lib/ceph/osd/ceph-6
> > /dev/sdf1 7.3T 1.5T 5.9T 21% /var/lib/ceph/osd/ceph-8
> > /dev/sde1 7.3T 1.6T 5.7T 22% /var/lib/ceph/osd/ceph-7
> > /dev/sdc1 7.3T 1.4T 6.0T 19% /var/lib/ceph/osd/ceph-5
> > /dev/sdb1 7.3T 1.5T 5.8T 21% /var/lib/ceph/osd/ceph-4
> > le08544 (osd)
> > /dev/sdc1 2.1T 1.6T 443G 79% /var/lib/ceph/osd/ceph-13
> > /dev/sdb1 2.1T 1.7T 339G 84% /var/lib/ceph/osd/ceph-12
> > /dev/sda1 2.1T 1.7T 375G 82% /var/lib/ceph/osd/ceph-11
> >
> > Clearly, I would like le10970 to be selected more often!
> >
> > Increasing pg(p)_num from 256 to 512 of all the pools
> > didn't help.
> >
> > Optimal tunables are used ...
> >
> > tunable choose_local_tries 0
> > tunable choose_local_fallback_tries 0
> > tunable choose_total_tries 50
> > tunable chooseleaf_descend_once 1
> > tunable chooseleaf_vary_r 1
> >
> > ... as well as the default algo (straw) and hash (0)
> > for all buckets. The ruleset is pretty much standard,
> > too ...
> >
> > rule replicated_ruleset {
> > ruleset 0
> > type replicated
> > min_size 1
> > max_size 10
> > step take default
> > step chooseleaf firstn 0 type host
> > step emit
> > }
> >
> > ... so I would assume it should select 3 hosts
> > (originating from root) according to the weights.
> >
> > It is ceph version 0.87.
> >
> > Is it the room bucket, which circumvents
> > a distribution according tot he weights?
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
