Hi,
AFAIR the device types adds a bunch of shadow devices e.g. osd.1~hdd or
something like that... And those shadow devs have a different crush id than
the original, untyped device.
So, alas, I don't think your test is complete, and yes I expect that your
data would move if you change the rule properly.
Cheers, Dan
On Thu, 17 Sep 2020, 18:36 Mathias Lindberg, <mathlin@xxxxxxxxxxx> wrote:
> Hi,
>
> We have a 1.2PB Nautilus installation primarily using CephFS for our
> HPC-resources.
> Our OSD’s have spinning disks and NvME devices for WAL and DB in an
> LVM-setup.
>
> The CephFS metadata pool resides on spinning disks, and I wonder if there
> is any point from a performance perspective to put that on flash?
> Trying to google that does not provide for a single straight answer, some
> say its is heavily cached on the MDS and does not benefit that much from
> flash while others argue it is significantly faster putting it on flash.
>
> We would regardless like to introduce flash OSD’s.
> In order to do that and not have data moving on to the ssd’s OSD’s we add
> we need add crush map rules that place data on ssd and hdd exclusively .
>
> But from reading previous posts to the list adding rules like that would
> trigger PG’s to start migrating.
>
> But when I (reluctantly) manually edit the crush map, adding a new class
> ssd (using a previously unused id) and adding the new rules I need
> crushtool seems to think no data vill shuffle and that maps are equivalent.
> Is it that simple, or am i doing it wrong?
> I would have preferred to have used some sort of tool like the crushtool
> reclassify (if applicable) to make these changes, but I can’t get the hang
> of that at all.
> The plan is to then change the crush rule for the exiting pools to the new
> ones with device classes.
>
> Thank you for any pointers or advice.
>
> [root@cephyr-mon1 crushtest]# diff -u crush_comp crush_comp_corr
> --- crush_comp 2020-09-17 17:11:37.125310334 +0200
> +++ crush_comp_corr 2020-09-17 17:14:28.022704876 +0200
> @@ -574,6 +574,7 @@
> root default {
> id -1 # do not change unnecessarily
> id -2 class hdd # do not change unnecessarily
> + id -29 class ssd
> # weight 2087.180
> alg straw2
> hash 0 # rjenkins1
> @@ -710,5 +711,25 @@
> step chooseleaf firstn 0 type host
> step emit
> }
> +rule replicated_ssd {
> + id 15
> + type replicated
> + min_size 1
> + max_size 10
> + step take default class ssd
> + step chooseleaf firstn 0 type host
> + step emit
> +}
> +rule cephyrfs_data_hdd {
> + id 16
> + type erasure
> + min_size 3
> + max_size 12
> + step set_chooseleaf_tries 5
> + step set_choose_tries 100
> + step take default class hdd
> + step chooseleaf indep 0 type host
> + step emit
> +}
>
> # end crush map
>
> [root@cephyr-mon1 crushtest]# crushtool -i crush_comp.c --compare
> crush_comp_corr.c
> rule 0 had 0/10240 mismatched mappings (0)
> rule 1 had 0/6144 mismatched mappings (0)
> rule 6 had 0/10240 mismatched mappings (0)
> rule 7 had 0/4096 mismatched mappings (0)
> rule 8 had 0/4096 mismatched mappings (0)
> rule 9 had 0/4096 mismatched mappings (0)
> rule 10 had 0/4096 mismatched mappings (0)
> rule 11 had 0/4096 mismatched mappings (0)
> rule 12 had 0/4096 mismatched mappings (0)
> rule 13 had 0/4096 mismatched mappings (0)
> rule 14 had 0/10240 mismatched mappings (0)
> maps appear equivalent
>
> Regards,
> Mathias Lindberg
>
> Tel: +46 (0)31 7723059
> Mob: +46 (0)723 526107
> Mathias Lindberg
> mathlin@xxxxxxxxxxx
>
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
