... unless you use the reclassify tooling: https://docs.ceph.com/en/latest/rados/operations/crush-map-edits/#migrating-from-a-legacy-ssd-rule-to-device-classes On Thu, 17 Sep 2020, 19:30 Dan van der Ster, <dan@xxxxxxxxxxxxxx> wrote: > Hi, > > AFAIR the device types adds a bunch of shadow devices e.g. osd.1~hdd or > something like that... And those shadow devs have a different crush id than > the original, untyped device. > > So, alas, I don't think your test is complete, and yes I expect that your > data would move if you change the rule properly. > > Cheers, Dan > > > > On Thu, 17 Sep 2020, 18:36 Mathias Lindberg, <mathlin@xxxxxxxxxxx> wrote: > >> Hi, >> >> We have a 1.2PB Nautilus installation primarily using CephFS for our >> HPC-resources. >> Our OSD’s have spinning disks and NvME devices for WAL and DB in an >> LVM-setup. >> >> The CephFS metadata pool resides on spinning disks, and I wonder if there >> is any point from a performance perspective to put that on flash? >> Trying to google that does not provide for a single straight answer, some >> say its is heavily cached on the MDS and does not benefit that much from >> flash while others argue it is significantly faster putting it on flash. >> >> We would regardless like to introduce flash OSD’s. >> In order to do that and not have data moving on to the ssd’s OSD’s we add >> we need add crush map rules that place data on ssd and hdd exclusively . >> >> But from reading previous posts to the list adding rules like that would >> trigger PG’s to start migrating. >> >> But when I (reluctantly) manually edit the crush map, adding a new class >> ssd (using a previously unused id) and adding the new rules I need >> crushtool seems to think no data vill shuffle and that maps are equivalent. >> Is it that simple, or am i doing it wrong? >> I would have preferred to have used some sort of tool like the crushtool >> reclassify (if applicable) to make these changes, but I can’t get the hang >> of that at all. >> The plan is to then change the crush rule for the exiting pools to the >> new ones with device classes. >> >> Thank you for any pointers or advice. >> >> [root@cephyr-mon1 crushtest]# diff -u crush_comp crush_comp_corr >> --- crush_comp 2020-09-17 17:11:37.125310334 +0200 >> +++ crush_comp_corr 2020-09-17 17:14:28.022704876 +0200 >> @@ -574,6 +574,7 @@ >> root default { >> id -1 # do not change unnecessarily >> id -2 class hdd # do not change unnecessarily >> + id -29 class ssd >> # weight 2087.180 >> alg straw2 >> hash 0 # rjenkins1 >> @@ -710,5 +711,25 @@ >> step chooseleaf firstn 0 type host >> step emit >> } >> +rule replicated_ssd { >> + id 15 >> + type replicated >> + min_size 1 >> + max_size 10 >> + step take default class ssd >> + step chooseleaf firstn 0 type host >> + step emit >> +} >> +rule cephyrfs_data_hdd { >> + id 16 >> + type erasure >> + min_size 3 >> + max_size 12 >> + step set_chooseleaf_tries 5 >> + step set_choose_tries 100 >> + step take default class hdd >> + step chooseleaf indep 0 type host >> + step emit >> +} >> >> # end crush map >> >> [root@cephyr-mon1 crushtest]# crushtool -i crush_comp.c --compare >> crush_comp_corr.c >> rule 0 had 0/10240 mismatched mappings (0) >> rule 1 had 0/6144 mismatched mappings (0) >> rule 6 had 0/10240 mismatched mappings (0) >> rule 7 had 0/4096 mismatched mappings (0) >> rule 8 had 0/4096 mismatched mappings (0) >> rule 9 had 0/4096 mismatched mappings (0) >> rule 10 had 0/4096 mismatched mappings (0) >> rule 11 had 0/4096 mismatched mappings (0) >> rule 12 had 0/4096 mismatched mappings (0) >> rule 13 had 0/4096 mismatched mappings (0) >> rule 14 had 0/10240 mismatched mappings (0) >> maps appear equivalent >> >> Regards, >> Mathias Lindberg >> >> Tel: +46 (0)31 7723059 >> Mob: +46 (0)723 526107 >> Mathias Lindberg >> mathlin@xxxxxxxxxxx >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |