Re: Ceph Balancer per Pool/Crush Unit

[Reed Dier <reed.dier@xxxxxxxxxxx>]

I suppose I may have found the solution I was unaware existed.

balancer optimize <plan> {<pools> [<pools>...]} :  Run optimizer to create a new plan

So apparently you can create a plan specific to a pool(s).

So just to double check this, I created two plans, plan1 with the hdd pool (and not the ssd pool); plan2 with no arguments.

I then ran ceph balancer show planN and also ceph osd crush weight-set dump.

Then compared the values in the weight-set dump against the values in the two plans, and concluded that plan1 did not adjust the values for ssd osd’s, which is exactly what I was looking for:

ID  CLASS WEIGHT    TYPE NAME                        STATUS REWEIGHT PRI-AFF

-14        17.61093         host ceph00

 24   ssd   1.76109             osd.24                   up  1.00000 1.00000

 25   ssd   1.76109             osd.25                   up  1.00000 1.00000

 26   ssd   1.76109             osd.26                   up  1.00000 1.00000

 27   ssd   1.76109             osd.27                   up  1.00000 1.00000

 28   ssd   1.76109             osd.28                   up  1.00000 1.00000

 29   ssd   1.76109             osd.29                   up  1.00000 1.00000

 30   ssd   1.76109             osd.30                   up  1.00000 1.00000

 31   ssd   1.76109             osd.31                   up  1.00000 1.00000

 32   ssd   1.76109             osd.32                   up  1.00000 1.00000

 33   ssd   1.76109             osd.33                   up  1.00000 1.00000

ceph osd crush weight-set dump

       {

            "bucket_id": -14,

            "weight_set": [

                [

                    1.756317,

                    1.613647,

                    1.733200,

                    1.735825,

                    1.961304,

                    1.583069,

                    1.963791,

                    1.773041,

                    1.890228,

                    1.793457

                ]

            ]

        },

plan1 (no change)

ceph osd crush weight-set reweight-compat 24 1.756317

ceph osd crush weight-set reweight-compat 25 1.613647

ceph osd crush weight-set reweight-compat 26 1.733200

ceph osd crush weight-set reweight-compat 27 1.735825

ceph osd crush weight-set reweight-compat 28 1.961304

ceph osd crush weight-set reweight-compat 29 1.583069

ceph osd crush weight-set reweight-compat 30 1.963791

ceph osd crush weight-set reweight-compat 31 1.773041

ceph osd crush weight-set reweight-compat 32 1.890228

ceph osd crush weight-set reweight-compat 33 1.793457

plan2 (change)

ceph osd crush weight-set reweight-compat 24 1.742185

ceph osd crush weight-set reweight-compat 25 1.608330

ceph osd crush weight-set reweight-compat 26 1.753393

ceph osd crush weight-set reweight-compat 27 1.713531

ceph osd crush weight-set reweight-compat 28 1.964446

ceph osd crush weight-set reweight-compat 29 1.629001

ceph osd crush weight-set reweight-compat 30 1.961968

ceph osd crush weight-set reweight-compat 31 1.738253

ceph osd crush weight-set reweight-compat 32 1.884098

ceph osd crush weight-set reweight-compat 33 1.779180

Hopefully this will be helpful for someone else who overlooks this in the -h output.

Reed

On Aug 1, 2018, at 6:05 PM, Reed Dier <reed.dier@xxxxxxxxxxx> wrote:

Hi Cephers,

I’m starting to play with the Ceph Balancer plugin after moving to straw2 and running into something I’m surprised I haven’t seen posted here.

My cluster has two crush roots, one for HDD, one for SSD.

Right now, HDD’s are a single pool to themselves, SSD’s are a single pool to themselves.

Using Ceph Balancer Eval, I can see the eval score for the hdd’s (worse), and the ssd’s (better), and the blended score of the cluster overall.

pool “hdd" score 0.012529 (lower is better)

pool “ssd" score 0.004654 (lower is better)

current cluster score 0.008484 (lower is better)

My problem is that I need to get my hdd’s better, and stop touching my ssd's, because shuffling data wear’s the ssd's unnecessarily, and it has actually gotten the distribution worse over time. https://imgur.com/RVh0jfH

You can see that between 06:00 and 09:00 on the second day in the graph that the spread was very tight, and then it expanded back.

So my question is, how can I run the balancer on just my hdd’s without touching my ssd’s?

I removed about 15% of the PG’s living on the HDD’s because they were empty.

I also have two tiers of HDD’s 8TB’s and 2TB’s, but they are roughly equally weighted in crush at the chassis level where my failure domains are configured.

Hopefully this abbreviated ceph osd tree displays the hierarchy. Multipliers for that bucket on right.

ID  CLASS WEIGHT    TYPE NAME

 -1       218.49353 root default.hdd

-10       218.49353     rack default.rack-hdd

-70        43.66553         chassis hdd-2tb-chassis1 *1

-67        43.66553             host hdd-2tb-24-1       *1

 74   hdd   1.81940                 osd.74              *24

-55        43.70700         chassis hdd-8tb-chassis1    *4

 -2        21.85350             host hdd-8tb-3-1

  0   hdd   7.28450                 osd.0               *3

 -3        21.85350             host hdd-8tb-3-1

  1   hdd   7.28450                 osd.1               *3

I assume this doesn’t complicate totally, but figured I would mention it, as I assume it is more difficult to equally distribute across OSD’s that are 4:1 size delta.

If I create a plan plan1 with ceph balancer optimize plan1,

then do a show plan1, I see an entry:

ceph osd crush weight-set reweight-compat $OSD $ArbitraryNumberNearOsdSize

Could I then copy this output, remove entries for SSD OSD’s and then run the ceph osd crush weight-set reweight-compat commands in a script?

I and my SSD’s appreciate any insight.

Thanks,

Reed

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com