Hello,
On Tue, 24 Feb 2015 06:28:25 +0000 Frank.Zirkelbach@xxxxxxxxxxxxxxxxxx
wrote:
> Hello everyone,
>
> In my rather heterogeneous setup ...
>
> -1 54.36 root default
> -2 42.44 room 2.162
> -4 6.09 host le09091
> 3 2.03 osd.3 up 1
> 1 2.03 osd.1 up 1
> 9 2.03 osd.9 up 1
> -6 36.35 host le10970
> 4 7.27 osd.4 up 1
> 5 7.27 osd.5 up 1
> 6 7.27 osd.6 up 1
> 7 7.27 osd.7 up 1
> 8 7.27 osd.8 up 1
> -3 11.92 room 2.166
> -5 5.83 host le09086
> 2 2.03 osd.2 up 1
> 0 2.03 osd.0 up 1
> 10 1.77 osd.10 up 1
> -7 6.09 host le08544
> 11 2.03 osd.11 up 1
> 12 2.03 osd.12 up 1
> 13 2.03 osd.13 up 1
>
> ... using size =3 for all pools,
> the OSDs are not filled according to the weights
> (which correspond to the disk sizes in TB).
>
> le09086 (osd)
> /dev/sdb1 1.8T 1.5T 323G 83% /var/lib/ceph/osd/ceph-10
> /dev/sdc1 2.1T 1.7T 408G 81% /var/lib/ceph/osd/ceph-0
> /dev/sdd1 2.1T 1.7T 344G 84% /var/lib/ceph/osd/ceph-2
> le09091 (osd)
> /dev/sda1 2.1T 1.6T 447G 79% /var/lib/ceph/osd/ceph-9
> /dev/sdc1 2.1T 1.8T 317G 85% /var/lib/ceph/osd/ceph-3
> /dev/sdb1 2.1T 1.7T 384G 82% /var/lib/ceph/osd/ceph-1
> le10970 (osd)
> /dev/sdd1 7.3T 1.4T 5.9T 19% /var/lib/ceph/osd/ceph-6
> /dev/sdf1 7.3T 1.5T 5.9T 21% /var/lib/ceph/osd/ceph-8
> /dev/sde1 7.3T 1.6T 5.7T 22% /var/lib/ceph/osd/ceph-7
> /dev/sdc1 7.3T 1.4T 6.0T 19% /var/lib/ceph/osd/ceph-5
> /dev/sdb1 7.3T 1.5T 5.8T 21% /var/lib/ceph/osd/ceph-4
> le08544 (osd)
> /dev/sdc1 2.1T 1.6T 443G 79% /var/lib/ceph/osd/ceph-13
> /dev/sdb1 2.1T 1.7T 339G 84% /var/lib/ceph/osd/ceph-12
> /dev/sda1 2.1T 1.7T 375G 82% /var/lib/ceph/osd/ceph-11
>
> Clearly, I would like le10970 to be selected more often!
>
> Increasing pg(p)_num from 256 to 512 of all the pools
> didn't help.
>
> Optimal tunables are used ...
>
> tunable choose_local_tries 0
> tunable choose_local_fallback_tries 0
> tunable choose_total_tries 50
> tunable chooseleaf_descend_once 1
> tunable chooseleaf_vary_r 1
>
> ... as well as the default algo (straw) and hash (0)
> for all buckets. The ruleset is pretty much standard,
> too ...
>
> rule replicated_ruleset {
> ruleset 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
>
> ... so I would assume it should select 3 hosts
> (originating from root) according to the weights.
>
> It is ceph version 0.87.
>
> Is it the room bucket, which circumvents
> a distribution according tot he weights?
>
That would be my guess, as a similar 4 node cluster here with a flat, host
only topology is having its data distributed as expected by the weights.
Christian
> Thanks and all the best,
>
> Frank
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Christian Balzer Network/Systems Engineer
chibi@xxxxxxx Global OnLine Japan/Fusion Communications
http://www.gol.com/
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
