They're still in the same root (default) and each host is member of
both device-classes, I guess you have a mixed setup (hosts c01/c02
have both HDDs and SSDs)? I don't think this separation is enough to
avoid remapping even if a different device-class is affected (your
report confirms that).
Dividing the crush tree into different subtrees might help here but
I'm not sure if that's really something you need. You might also just
deal with the remapping as long as it doesn't happen too often, I
guess. On the other hand, if your setup won't change (except adding
more OSDs) you might as well think about a different crush tree. It
really depends on your actual requirements.
We created two different subtrees when we got new hardware and it
helped us a lot moving the data only once to the new hardware avoiding
multiple remappings, now the older hardware is our EC environment
except for some SSDs on those old hosts that had to stay in the main
subtree. So our setup is also very individual but it works quite nice.
:-)
Zitat von Marc Roos <M.Roos@xxxxxxxxxxxxxxxxx>:
I have practically a default setup. If I do a 'ceph osd crush tree
--show-shadow' I have a listing like this[1]. I would assume from the
hosts being listed within the default~ssd and default~hdd, they are
separate (enough)?
[1]
root default~ssd
host c01~ssd
..
..
host c02~ssd
..
root default~hdd
host c01~hdd
..
host c02~hdd
..
root default
-----Original Message-----
To: ceph-users@xxxxxxx
Subject: Re: hdd pg's migrating when converting ssd class
osd's
Are all the OSDs in the same crush root? I would think that since the
crush weight of hosts change as soon as OSDs are out it impacts the
whole crush tree. If you separate the SSDs from the HDDs logically (e.g.
different bucket type in the crush tree) the ramapping wouldn't affect
the HDDs.
I have been converting ssd's osd's to dmcrypt, and I have noticed that
pg's of pools are migrated that should be (and are?) on hdd class.
On a healthy ok cluster I am getting, when I set the crush reweight to
0.0 of a ssd osd this:
17.35 10415 0 0 9907 0
36001743890 0 0 3045 3045
active+remapped+backfilling 2020-09-27 12:55:49.093054 83758'20725398
83758:100379720 [8,14,23] 8 [3,14,23] 3
83636'20718129 2020-09-27 00:58:07.098096 83300'20689151 2020-09-24
21:42:07.385360 0
However osds 3,14,23,8 are all hdd osd's
Since this is a cluster from Kraken/Luminous, I am not sure if the
device class of the replicated_ruleset[1] was set when the pool 17 was
created.
Weird thing is that all pg's of this pool seem to be on hdd osd[2]
Q. How can I display the definition of 'crush_rule 0' at the time of
the pool creation? (To be sure it had already this device class hdd
configured)
[1]
[@~]# ceph osd pool ls detail | grep 'pool 17'
pool 17 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 83712
flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
[@~]# ceph osd crush rule dump replicated_ruleset {
"rule_id": 0,
"rule_name": "replicated_ruleset",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -10,
"item_name": "default~hdd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host"
},
{
"op": "emit"
}
]
}
[2]
[@~]# for osd in `ceph pg dump pgs| grep '^17' | awk '{print $17"
"$19}'
| grep -oE '[0-9]{1,2}'| sort -u -n`; do ceph osd crush
| get-device-class
osd.$osd ; done | sort -u
dumped pgs
hdd
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
