Crush device class switchover

Reed Dier <reed.dier@xxxxxxxxxxx> · Mon, 30 Sep 2019 11:28:50 -0500

I currently have two roots in my crush map, one for HDD devices and one for SSD devices, and have had it that way since Jewel.
I am currently on Nautilus, and have had my crush device classes for my OSD's set since Luminous.

ID  CLASS WEIGHT    TYPE NAME
-13       105.37599 root ssd
-11       105.37599     rack ssd.rack2
-14        17.61099         host ceph00
 24   ssd   1.76099             osd.24
 -1       398.92554 root default
-10       397.07343     rack default.rack2
-70        44.45032         chassis ceph05
-67        44.45032             host ceph05
 74   hdd   1.85210                 osd.74

I have crush rulesets that distribute based on the roots for each device class.

[
    {
        "rule_id": 0,
        "rule_name": "replicated_ruleset",
        "ruleset": 0,
        "type": 1,
        "min_size": 1,
        "max_size": 10,
        "steps": [
            {
                "op": "take",
                "item": -1,
                "item_name": "default"
            },
            {
                "op": "chooseleaf_firstn",
                "num": 0,
                "type": "chassis"
            },
            {
                "op": "emit"
            }
        ]
    },
    {
        "rule_id": 1,
        "rule_name": "ssd_ruleset",
        "ruleset": 1,
        "type": 1,
        "min_size": 1,
        "max_size": 10,
        "steps": [
            {
                "op": "take",
                "item": -13,
                "item_name": "ssd"
            },
            {
                "op": "chooseleaf_firstn",
                "num": 0,
                "type": "host"
            },
            {
                "op": "emit"
            }
        ]
    },
    {
        "rule_id": 2,
        "rule_name": "hybrid_ruleset",
        "ruleset": 2,
        "type": 1,
        "min_size": 1,
        "max_size": 10,
        "steps": [
            {
                "op": "take",
                "item": -13,
                "item_name": "ssd"
            },
            {
                "op": "chooseleaf_firstn",
                "num": 1,
                "type": "host"
            },
            {
                "op": "emit"
            },
            {
                "op": "take",
                "item": -1,
                "item_name": "default"
            },
            {
                "op": "chooseleaf_firstn",
                "num": -1,
                "type": "chassis"
            },
            {
                "op": "emit"
            }
        ]
    }

If I wanted to migrate to rulesets based on device class with minimal disruption, what are my options?

In my mind the way this would work would be to
1. Set the norebalance flag
2. Rework my crush rulesets to use takes based on class rather than root.
3. Merge my ssd hosts from the ssd root to the default root
4. Let things rebalance?

Would prefer minimal data movement, as that would be potentially disruptive, and I imagine provide minimal gain for me, but possibly better data distribution?

Maybe there are better steps to take?

Appreciate any help.

Reed

Attachment:
smime.p7s

Description: S/MIME cryptographic signature
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx