Probably chooseleaf also instead of choose.
Konrad Riedel <it@xxxxxxxxxxxxxx> skrev: (10 oktober 2017 17:05:52 CEST)
>Hello Ceph-users,
>
>after switching to luminous I was excited about the great
>crush-device-class feature - now we have 5 servers with 1x2TB NVMe
>based OSDs, 3 of them additionally with 4 HDDS per server. (we have
>only three 400G NVMe disks for block.wal and block.db and therefore
>can't distribute all HDDs evenly on all servers.)
>
>Output from "ceph pg dump" shows that some PGs end up on HDD OSDs on
>the same
>Host:
>
>ceph pg map 5.b
>osdmap e12912 pg 5.b (5.b) -> up [9,7,8] acting [9,7,8]
>
>(on rebooting this host I had 4 stale PGs)
>
>I've written a small perl script to add hostname after OSD number and
>got many PGs where
>ceph placed 2 replicas on the same host... :
>
>5.1e7: 8 - daniel 9 - daniel 11 - udo
>5.1eb: 10 - udo 7 - daniel 9 - daniel
>5.1ec: 10 - udo 11 - udo 7 - daniel
>5.1ed: 13 - felix 16 - felix 5 - udo
>
>
>Is there any way I can correct this?
>
>
>Please see crushmap below. Thanks for any help!
>
># begin crush map
>tunable choose_local_tries 0
>tunable choose_local_fallback_tries 0
>tunable choose_total_tries 50
>tunable chooseleaf_descend_once 1
>tunable chooseleaf_vary_r 1
>tunable chooseleaf_stable 1
>tunable straw_calc_version 1
>tunable allowed_bucket_algs 54
>
># devices
>device 0 osd.0 class hdd
>device 1 device1
>device 2 osd.2 class ssd
>device 3 device3
>device 4 device4
>device 5 osd.5 class hdd
>device 6 device6
>device 7 osd.7 class hdd
>device 8 osd.8 class hdd
>device 9 osd.9 class hdd
>device 10 osd.10 class hdd
>device 11 osd.11 class hdd
>device 12 osd.12 class hdd
>device 13 osd.13 class hdd
>device 14 osd.14 class hdd
>device 15 device15
>device 16 osd.16 class hdd
>device 17 device17
>device 18 device18
>device 19 device19
>device 20 device20
>device 21 device21
>device 22 device22
>device 23 device23
>device 24 osd.24 class hdd
>device 25 device25
>device 26 osd.26 class hdd
>device 27 osd.27 class hdd
>device 28 osd.28 class hdd
>device 29 osd.29 class hdd
>device 30 osd.30 class ssd
>device 31 osd.31 class ssd
>device 32 osd.32 class ssd
>device 33 osd.33 class ssd
>
># types
>type 0 osd
>type 1 host
>type 2 rack
>type 3 row
>type 4 room
>type 5 datacenter
>type 6 root
>
># buckets
>host daniel {
> id -4 # do not change unnecessarily
> id -2 class hdd # do not change unnecessarily
> id -9 class ssd # do not change unnecessarily
> # weight 3.459
> alg straw2
> hash 0 # rjenkins1
> item osd.31 weight 1.819
> item osd.7 weight 0.547
> item osd.8 weight 0.547
> item osd.9 weight 0.547
>}
>host felix {
> id -5 # do not change unnecessarily
> id -3 class hdd # do not change unnecessarily
> id -10 class ssd # do not change unnecessarily
> # weight 3.653
> alg straw2
> hash 0 # rjenkins1
> item osd.33 weight 1.819
> item osd.13 weight 0.547
> item osd.14 weight 0.467
> item osd.16 weight 0.547
> item osd.0 weight 0.274
>}
>host udo {
> id -6 # do not change unnecessarily
> id -7 class hdd # do not change unnecessarily
> id -11 class ssd # do not change unnecessarily
> # weight 4.006
> alg straw2
> hash 0 # rjenkins1
> item osd.32 weight 1.819
> item osd.5 weight 0.547
> item osd.10 weight 0.547
> item osd.11 weight 0.547
> item osd.12 weight 0.547
>}
>host moritz {
> id -13 # do not change unnecessarily
> id -14 class hdd # do not change unnecessarily
> id -15 class ssd # do not change unnecessarily
> # weight 1.819
> alg straw2
> hash 0 # rjenkins1
> item osd.30 weight 1.819
>}
>host bruno {
> id -16 # do not change unnecessarily
> id -17 class hdd # do not change unnecessarily
> id -18 class ssd # do not change unnecessarily
> # weight 3.183
> alg straw2
> hash 0 # rjenkins1
> item osd.24 weight 0.273
> item osd.26 weight 0.273
> item osd.27 weight 0.273
> item osd.28 weight 0.273
> item osd.29 weight 0.273
> item osd.2 weight 1.819
>}
>root default {
> id -1 # do not change unnecessarily
> id -8 class hdd # do not change unnecessarily
> id -12 class ssd # do not change unnecessarily
> # weight 16.121
> alg straw2
> hash 0 # rjenkins1
> item daniel weight 3.459
> item felix weight 3.653
> item udo weight 4.006
> item moritz weight 1.819
> item bruno weight 3.183
>}
>
># rules
>rule ssd {
> id 0
> type replicated
> min_size 1
> max_size 10
> step take default class ssd
> step choose firstn 0 type osd
> step emit
>}
>rule hdd {
> id 1
> type replicated
> min_size 1
> max_size 10
> step take default class hdd
> step choose firstn 0 type osd
> step emit
>}
>
># end crush map
>
>--
>
>Mit freundlichen Grüßen
>
>Konrad Riedel
>
>--
>
>Berufsförderungswerk Dresden gemeinnützige GmbH
>SG1
>IT/Infrastruktur
>Hellerhofstraße 35
>D-01129 Dresden
>Telefon (03 51) 85 48 - 115
>Telefax (03 51) 85 48 - 507
>E-Mail it@xxxxxxxxxxxxxx
>
>--------------------------------------------------------------------
>Vorsitzende des Verwaltungsrates: Dr. Ina Ueberschär
>Geschäftsführer: Henry Köhler
>Unternehmenssitz: Dresden
>Handelsregister: Amtsgericht Dresden HRB 2380
>Zertifiziert nach ISO 9001:2015 und AZAV
>--------------------------------------------------------------------
>
>_______________________________________________
>ceph-users mailing list
>ceph-users@xxxxxxxxxxxxxx
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
