Re: The effect of changing an osd's class

[Roland Giesler <roland@xxxxxxxxxxxxxx>]

On 2024/11/17 18:12, Anthony D'Atri wrote:
> I see 5 OSDs with 0 CRUSH weight, is that intentional?

Yes, I  set the weight to 0 to ensure all the pg's are removed from them 
them since I'm removing them (worn out ssd's)

I think I found the problem.  I had created a CRUSH rule called old_ssd 
(and corresponding pool) into which I had earlier attempted to move the 
ssd's by changing their respective device class to a custom class I 
created.  Also I had removed the devices from that new class, the rule 
and pool remained.  Eventually I saw that there was actually ghost data 
assigned to that pool.  Since I was sure there the were no legitimate 
pg's in the pool, I deleted it and voila! all the "stuck" pg's 
vanished.  I have now removed those ssd's from their pool again and so 
far so good.

> Notably:
>
> > All the problem pg's are on osd.39.
> osd.39 has 0 CRUSH weight, so CRUSH shouldn’t be placing any PGs there.  Yet there appear to be PGs mapped to the 4x 0 weight OSDs that are up.  I had hoped that the health detail would show the PG IDs, and thus the pool(s) they are in.  7 pools is a lot for such a small cluster.  In case nobody has called it out yet, with size=2 you run a risk of data unavailability and eventual loss.
>
> You replied to Eugen asserting that rule 0 is the rule for the `ssd` class.  Is it possible that And later you showed a query for PG 28.42 — you must have a LOT of pools, or have created and removed a lot in the past.  So at least some and perhaps all of the problem PGs are in pool #28.
>
> Please send `ceph osd dump | grep pool` ; I don’t see that in the thread so far.  Similarly `ceph osd tree` which I don’t think we’ve seen yet.  Your cluster is … more complicated than I had expected.
>
> I see that a number of your OSDs have the class ‘old-ssd’, I’ll guess those are the smaller ones?
>
> Mind you it’s Sunday morning for me (yawwwn) but I see 4x OSDs with the device class ’ssd’ that appear to be directly under the CRUSH root, vs being under a host bucket.  That could be part of the problem as well.
>
>
>
>
>
> 25    ssd  0.28319   1.00000  290 GiB   152 GiB  151 GiB  2.8 MiB  1020 MiB  138 GiB  52.52  1.06   39      up
> 41    ssd  3.49309   1.00000  3.5 TiB   1.6 TiB  1.6 TiB   22 MiB   5.4 GiB  1.9 TiB  44.58  0.90  411      up
> 38    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0   41      up
>   0    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0    0    down
> 36    ssd        0   1.00000  290 GiB   1.3 GiB   23 MiB  2.8 MiB   1.2 GiB  289 GiB   0.44  0.01   13      up
> 37    ssd        0   1.00000  290 GiB   1.1 GiB   23 MiB  2.5 MiB   1.1 GiB  289 GiB   0.38  0.01   18      up
> 39    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0   86      up
>
> The SSD OSDs are really small, but you know that - just be aware that overhead is such that you won’t be able to fill them as much as you might think.
>   This is one of the wrinkles with having OSDs of considerably different sizes used for a common pool.  You didn’t ask about performance, but the 3.84TB SSDs are getting 12x the op workload of the 300GB SSDs.  Which may not be all bad, since a 300GB SSD may well be old, client-class, and/or have low performance/endurance.
>
>
> > > Disabling mclock as described here https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/ might help
> > I cannot see any option that allows me to disable mclock...
> The option to enable override is what I meant, also cf later replies to this thread.
>
>
>          "choose_total_tries": 50,
>
> Upping this to say 100 may help.  I’ve seen placement snags on very small clusters.
>
> > > Also, you have a small cluster with a bunch of small OSDs.  Please send `ceph health detail`
> > # ceph health detail
> > HEALTH_WARN 1 pool(s) have no replicas configured
> > [WRN] POOL_NO_REDUNDANCY: 1 pool(s) have no replicas configured
> >      pool 'backups' has no replicas configured
> >
> > The backups pool is on spinners and 1 copy is sufficient since we're also replicating this offsite in real time.  Otherwise too many copies take too much space.
> >
> > > Please also send `ceph osd df` and `ceph osd crush dump`
> > NodeC:~# ceph osd df
> > ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE   DATA     OMAP     META      AVAIL    %USE   VAR   PGS  STATUS
> >   2    hdd  1.86029   1.00000  1.9 TiB   1.1 TiB  1.1 TiB   16 MiB   2.7 GiB  746 GiB  60.97  1.23   68      up
> >   3    hdd  1.86029   1.00000  1.9 TiB   949 GiB  902 GiB   12 MiB   2.6 GiB  961 GiB  49.68  1.00   57      up
> >   4    hdd  1.86029   1.00000  1.9 TiB   848 GiB  800 GiB   10 MiB   2.3 GiB  1.0 TiB  44.37  0.89   51      up
> >   5    hdd  1.86589   1.00000  1.9 TiB   1.0 TiB  982 GiB   15 MiB   2.4 GiB  881 GiB  53.87  1.08   66      up
> >   0    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0    0    down
> >   1    ssd  0.28319   1.00000  290 GiB   123 GiB  122 GiB  2.3 MiB   1.2 GiB  167 GiB  42.38  0.85   46      up
> > 36    ssd        0   1.00000  290 GiB   1.3 GiB   23 MiB  2.8 MiB   1.2 GiB  289 GiB   0.44  0.01   13      up
> > 37    ssd        0   1.00000  290 GiB   1.1 GiB   23 MiB  2.5 MiB   1.1 GiB  289 GiB   0.38  0.01   18      up
> > 38    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0   41      up
> > 39    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0   86      up
> > 40    ssd  3.30690   1.00000  3.3 TiB   1.7 TiB  1.7 TiB   23 MiB   4.8 GiB  1.6 TiB  50.49  1.02  424      up
> > 10    hdd  1.86029   1.00000  1.9 TiB   1.0 TiB  990 GiB   13 MiB   2.8 GiB  873 GiB  54.19  1.09   64      up
> > 11    hdd  1.86029   1.00000  1.9 TiB   1.3 TiB  1.2 TiB   15 MiB   3.2 GiB  625 GiB  67.20  1.35   72      up
> > 26    hdd  1.86029   1.00000  1.9 TiB   843 GiB  801 GiB   12 MiB   2.3 GiB  1.0 TiB  44.25  0.89   50      up
> > 27    hdd  1.86029   1.00000  1.9 TiB   1.1 TiB  1.0 TiB   16 MiB   2.7 GiB  793 GiB  58.38  1.18   70      up
> >   6    ssd  0.28319   1.00000  290 GiB   115 GiB  115 GiB  2.3 MiB   743 MiB  175 GiB  39.82  0.80   28      up
> >   7    ssd  0.28319   1.00000  290 GiB    94 GiB   93 GiB  2.0 MiB   921 MiB  196 GiB  32.51  0.65   23      up
> >   8    ssd  0.28319   1.00000  290 GiB   103 GiB  102 GiB  2.1 MiB   963 MiB  187 GiB  35.46  0.71   27      up
> >   9    ssd  0.28319   1.00000  290 GiB   131 GiB  130 GiB  2.5 MiB   1.1 GiB  159 GiB  45.28  0.91   33      up
> > 24    ssd  0.28319   1.00000  290 GiB   102 GiB  101 GiB  2.1 MiB   719 MiB  188 GiB  35.10  0.71   26      up
> > 25    ssd  0.28319   1.00000  290 GiB   152 GiB  151 GiB  2.8 MiB  1020 MiB  138 GiB  52.52  1.06   39      up
> > 41    ssd  3.49309   1.00000  3.5 TiB   1.6 TiB  1.6 TiB   22 MiB   5.4 GiB  1.9 TiB  44.58  0.90  411      up
> > 14    hdd  1.59999   1.00000  1.9 TiB   1.0 TiB  1.0 TiB   13 MiB   2.4 GiB  837 GiB  56.04  1.13   63      up
> > 15    hdd  1.86029   1.00000  1.9 TiB  1017 GiB  975 GiB   13 MiB   2.1 GiB  888 GiB  53.36  1.07   60      up
> > 16    hdd  1.86029   1.00000  1.9 TiB   990 GiB  948 GiB   13 MiB   2.2 GiB  915 GiB  51.97  1.05   59      up
> > 17    hdd  1.86029   1.00000  1.9 TiB   1.0 TiB  992 GiB   14 MiB   2.5 GiB  871 GiB  54.28  1.09   64      up
> > 12    ssd  0.28319   1.00000  290 GiB   122 GiB  121 GiB  2.3 MiB   1.1 GiB  168 GiB  41.96  0.84   31      up
> > 13    ssd  0.28319   1.00000  290 GiB   145 GiB  144 GiB  2.6 MiB   1.2 GiB  145 GiB  49.95  1.01   37      up
> > 28    ssd  0.28319   1.00000  290 GiB   142 GiB  141 GiB  2.6 MiB   1.1 GiB  148 GiB  48.89  0.98   35      up
> > 29    ssd  0.28319   1.00000  290 GiB   109 GiB  108 GiB  2.2 MiB   1.3 GiB  181 GiB  37.58  0.76   29      up
> > 30    ssd  0.28319   1.00000  290 GiB   146 GiB  145 GiB  2.7 MiB   1.3 GiB  144 GiB  50.37  1.01   37      up
> > 31    ssd  0.28319   1.00000  290 GiB   131 GiB  130 GiB  2.5 MiB   1.3 GiB  159 GiB  45.24  0.91   34      up
> > 43    ssd  3.30690   1.00000  3.3 TiB   1.4 TiB  1.4 TiB   19 MiB   4.0 GiB  1.9 TiB  42.50  0.86  353      up
> > 20    hdd  1.86029   1.00000  1.9 TiB   1.2 TiB  1.2 TiB   12 MiB   2.8 GiB  659 GiB  65.38  1.32   67      up
> > 21    hdd  1.86029   1.00000  1.9 TiB   1.1 TiB  1.0 TiB   15 MiB   2.8 GiB  815 GiB  57.20  1.15   68      up
> > 22    hdd  1.86029   1.00000  1.9 TiB   878 GiB  836 GiB   13 MiB   2.4 GiB  1.0 TiB  46.10  0.93   54      up
> > 23    hdd  1.86029   1.00000  1.9 TiB  1018 GiB  977 GiB   14 MiB   2.8 GiB  886 GiB  53.46  1.08   59      up
> > 18    ssd  0.28319   1.00000  290 GiB   115 GiB  114 GiB  2.3 MiB   1.3 GiB  175 GiB  39.74  0.80   29      up
> > 19    ssd  0.28319   1.00000  290 GiB   115 GiB  114 GiB  2.3 MiB   961 MiB  175 GiB  39.59  0.80   28      up
> > 32    ssd  0.28319   1.00000  290 GiB   103 GiB  102 GiB  2.1 MiB   1.0 GiB  187 GiB  35.65  0.72   26      up
> > 33    ssd  0.28319   1.00000  290 GiB    98 GiB   97 GiB  2.1 MiB   1.4 GiB  192 GiB  33.92  0.68   26      up
> > 34    ssd  0.28319   1.00000  290 GiB   141 GiB  140 GiB  2.6 MiB   987 MiB  149 GiB  48.49  0.98   35      up
> > 35    ssd  0.28319   1.00000  290 GiB   116 GiB  114 GiB  2.3 MiB   1.3 GiB  174 GiB  39.92  0.80   31      up
> > 42    ssd  3.49309   1.00000  3.5 TiB   1.4 TiB  1.4 TiB   20 MiB   4.5 GiB  2.1 TiB  40.45  0.81  372      up
> >                         TOTAL   49 TiB    25 TiB   24 TiB  351 MiB    83 GiB   25 TiB  49.67
> > MIN/MAX VAR: 0.01/1.35  STDDEV: 13.87
> >
> > NodeC:~# ceph osd crush dump
> > {
> >      "devices": [
> >          {
> >              "id": 0,
> >              "name": "osd.0",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 1,
> >              "name": "osd.1",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 2,
> >              "name": "osd.2",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 3,
> >              "name": "osd.3",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 4,
> >              "name": "osd.4",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 5,
> >              "name": "osd.5",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 6,
> >              "name": "osd.6",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 7,
> >              "name": "osd.7",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 8,
> >              "name": "osd.8",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 9,
> >              "name": "osd.9",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 10,
> >              "name": "osd.10",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 11,
> >              "name": "osd.11",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 12,
> >              "name": "osd.12",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 13,
> >              "name": "osd.13",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 14,
> >              "name": "osd.14",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 15,
> >              "name": "osd.15",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 16,
> >              "name": "osd.16",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 17,
> >              "name": "osd.17",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 18,
> >              "name": "osd.18",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 19,
> >              "name": "osd.19",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 20,
> >              "name": "osd.20",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 21,
> >              "name": "osd.21",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 22,
> >              "name": "osd.22",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 23,
> >              "name": "osd.23",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 24,
> >              "name": "osd.24",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 25,
> >              "name": "osd.25",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 26,
> >              "name": "osd.26",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 27,
> >              "name": "osd.27",
> >              "class": "hdd"
> >          },
> >          {
> >              "id": 28,
> >              "name": "osd.28",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 29,
> >              "name": "osd.29",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 30,
> >              "name": "osd.30",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 31,
> >              "name": "osd.31",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 32,
> >              "name": "osd.32",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 33,
> >              "name": "osd.33",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 34,
> >              "name": "osd.34",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 35,
> >              "name": "osd.35",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 36,
> >              "name": "osd.36",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 37,
> >              "name": "osd.37",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 38,
> >              "name": "osd.38",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 39,
> >              "name": "osd.39",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 40,
> >              "name": "osd.40",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 41,
> >              "name": "osd.41",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 42,
> >              "name": "osd.42",
> >              "class": "ssd"
> >          },
> >          {
> >              "id": 43,
> >              "name": "osd.43",
> >              "class": "ssd"
> >          }
> >      ],
> >      "types": [
> >          {
> >              "type_id": 0,
> >              "name": "osd"
> >          },
> >          {
> >              "type_id": 1,
> >              "name": "host"
> >          },
> >          {
> >              "type_id": 2,
> >              "name": "chassis"
> >          },
> >          {
> >              "type_id": 3,
> >              "name": "rack"
> >          },
> >          {
> >              "type_id": 4,
> >              "name": "row"
> >          },
> >          {
> >              "type_id": 5,
> >              "name": "pdu"
> >          },
> >          {
> >              "type_id": 6,
> >              "name": "pod"
> >          },
> >          {
> >              "type_id": 7,
> >              "name": "room"
> >          },
> >          {
> >              "type_id": 8,
> >              "name": "datacenter"
> >          },
> >          {
> >              "type_id": 9,
> >              "name": "zone"
> >          },
> >          {
> >              "type_id": 10,
> >              "name": "region"
> >          },
> >          {
> >              "type_id": 11,
> >              "name": "root"
> >          }
> >      ],
> >      "buckets": [
> >          {
> >              "id": -1,
> >              "name": "default",
> >              "type_id": 11,
> >              "type_name": "root",
> >              "weight": 3177873,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": -3,
> >                      "weight": 723311,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": -7,
> >                      "weight": 827941,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": -10,
> >                      "weight": 798680,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": -13,
> >                      "weight": 827941,
> >                      "pos": 3
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -2,
> >              "name": "default~ssd",
> >              "type_id": 11,
> >              "type_name": "root",
> >              "weight": 1243909,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": -4,
> >                      "weight": 235280,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": -8,
> >                      "weight": 340277,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": -11,
> >                      "weight": 328075,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": -14,
> >                      "weight": 340277,
> >                      "pos": 3
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -3,
> >              "name": "FT1-NodeA",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 723311,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 1,
> >                      "weight": 18559,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 36,
> >                      "weight": 0,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 37,
> >                      "weight": 0,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 0,
> >                      "weight": 0,
> >                      "pos": 3
> >                  },
> >                  {
> >                      "id": 38,
> >                      "weight": 0,
> >                      "pos": 4
> >                  },
> >                  {
> >                      "id": 39,
> >                      "weight": 0,
> >                      "pos": 5
> >                  },
> >                  {
> >                      "id": 2,
> >                      "weight": 121916,
> >                      "pos": 6
> >                  },
> >                  {
> >                      "id": 3,
> >                      "weight": 121916,
> >                      "pos": 7
> >                  },
> >                  {
> >                      "id": 4,
> >                      "weight": 121916,
> >                      "pos": 8
> >                  },
> >                  {
> >                      "id": 40,
> >                      "weight": 216721,
> >                      "pos": 9
> >                  },
> >                  {
> >                      "id": 5,
> >                      "weight": 122283,
> >                      "pos": 10
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -4,
> >              "name": "FT1-NodeA~ssd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 235280,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 1,
> >                      "weight": 18559,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 36,
> >                      "weight": 0,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 37,
> >                      "weight": 0,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 0,
> >                      "weight": 0,
> >                      "pos": 3
> >                  },
> >                  {
> >                      "id": 38,
> >                      "weight": 0,
> >                      "pos": 4
> >                  },
> >                  {
> >                      "id": 39,
> >                      "weight": 0,
> >                      "pos": 5
> >                  },
> >                  {
> >                      "id": 40,
> >                      "weight": 216721,
> >                      "pos": 6
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -5,
> >              "name": "FT1-NodeA~hdd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 488031,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 2,
> >                      "weight": 121916,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 3,
> >                      "weight": 121916,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 4,
> >                      "weight": 121916,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 5,
> >                      "weight": 122283,
> >                      "pos": 3
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -6,
> >              "name": "default~hdd",
> >              "type_id": 11,
> >              "type_name": "root",
> >              "weight": 1933964,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": -5,
> >                      "weight": 488031,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": -9,
> >                      "weight": 487664,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": -12,
> >                      "weight": 470605,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": -15,
> >                      "weight": 487664,
> >                      "pos": 3
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -7,
> >              "name": "FT1-NodeB",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 827941,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 6,
> >                      "weight": 18559,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 24,
> >                      "weight": 18559,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 25,
> >                      "weight": 18559,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 11,
> >                      "weight": 121916,
> >                      "pos": 3
> >                  },
> >                  {
> >                      "id": 10,
> >                      "weight": 121916,
> >                      "pos": 4
> >                  },
> >                  {
> >                      "id": 7,
> >                      "weight": 18559,
> >                      "pos": 5
> >                  },
> >                  {
> >                      "id": 8,
> >                      "weight": 18559,
> >                      "pos": 6
> >                  },
> >                  {
> >                      "id": 9,
> >                      "weight": 18559,
> >                      "pos": 7
> >                  },
> >                  {
> >                      "id": 26,
> >                      "weight": 121916,
> >                      "pos": 8
> >                  },
> >                  {
> >                      "id": 27,
> >                      "weight": 121916,
> >                      "pos": 9
> >                  },
> >                  {
> >                      "id": 41,
> >                      "weight": 228923,
> >                      "pos": 10
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -8,
> >              "name": "FT1-NodeB~ssd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 340277,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 6,
> >                      "weight": 18559,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 24,
> >                      "weight": 18559,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 25,
> >                      "weight": 18559,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 7,
> >                      "weight": 18559,
> >                      "pos": 3
> >                  },
> >                  {
> >                      "id": 8,
> >                      "weight": 18559,
> >                      "pos": 4
> >                  },
> >                  {
> >                      "id": 9,
> >                      "weight": 18559,
> >                      "pos": 5
> >                  },
> >                  {
> >                      "id": 41,
> >                      "weight": 228923,
> >                      "pos": 6
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -9,
> >              "name": "FT1-NodeB~hdd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 487664,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 11,
> >                      "weight": 121916,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 10,
> >                      "weight": 121916,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 26,
> >                      "weight": 121916,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 27,
> >                      "weight": 121916,
> >                      "pos": 3
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -10,
> >              "name": "FT1-NodeC",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 798680,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 13,
> >                      "weight": 18559,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 28,
> >                      "weight": 18559,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 29,
> >                      "weight": 18559,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 14,
> >                      "weight": 104857,
> >                      "pos": 3
> >                  },
> >                  {
> >                      "id": 12,
> >                      "weight": 18559,
> >                      "pos": 4
> >                  },
> >                  {
> >                      "id": 30,
> >                      "weight": 18559,
> >                      "pos": 5
> >                  },
> >                  {
> >                      "id": 31,
> >                      "weight": 18559,
> >                      "pos": 6
> >                  },
> >                  {
> >                      "id": 15,
> >                      "weight": 121916,
> >                      "pos": 7
> >                  },
> >                  {
> >                      "id": 16,
> >                      "weight": 121916,
> >                      "pos": 8
> >                  },
> >                  {
> >                      "id": 17,
> >                      "weight": 121916,
> >                      "pos": 9
> >                  },
> >                  {
> >                      "id": 43,
> >                      "weight": 216721,
> >                      "pos": 10
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -11,
> >              "name": "FT1-NodeC~ssd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 328075,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 13,
> >                      "weight": 18559,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 28,
> >                      "weight": 18559,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 29,
> >                      "weight": 18559,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 12,
> >                      "weight": 18559,
> >                      "pos": 3
> >                  },
> >                  {
> >                      "id": 30,
> >                      "weight": 18559,
> >                      "pos": 4
> >                  },
> >                  {
> >                      "id": 31,
> >                      "weight": 18559,
> >                      "pos": 5
> >                  },
> >                  {
> >                      "id": 43,
> >                      "weight": 216721,
> >                      "pos": 6
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -12,
> >              "name": "FT1-NodeC~hdd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 470605,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 14,
> >                      "weight": 104857,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 15,
> >                      "weight": 121916,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 16,
> >                      "weight": 121916,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 17,
> >                      "weight": 121916,
> >                      "pos": 3
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -13,
> >              "name": "FT1-NodeD",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 827941,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 18,
> >                      "weight": 18559,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 32,
> >                      "weight": 18559,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 33,
> >                      "weight": 18559,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 22,
> >                      "weight": 121916,
> >                      "pos": 3
> >                  },
> >                  {
> >                      "id": 19,
> >                      "weight": 18559,
> >                      "pos": 4
> >                  },
> >                  {
> >                      "id": 34,
> >                      "weight": 18559,
> >                      "pos": 5
> >                  },
> >                  {
> >                      "id": 35,
> >                      "weight": 18559,
> >                      "pos": 6
> >                  },
> >                  {
> >                      "id": 23,
> >                      "weight": 121916,
> >                      "pos": 7
> >                  },
> >                  {
> >                      "id": 21,
> >                      "weight": 121916,
> >                      "pos": 8
> >                  },
> >                  {
> >                      "id": 20,
> >                      "weight": 121916,
> >                      "pos": 9
> >                  },
> >                  {
> >                      "id": 42,
> >                      "weight": 228923,
> >                      "pos": 10
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -14,
> >              "name": "FT1-NodeD~ssd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 340277,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 18,
> >                      "weight": 18559,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 32,
> >                      "weight": 18559,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 33,
> >                      "weight": 18559,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 19,
> >                      "weight": 18559,
> >                      "pos": 3
> >                  },
> >                  {
> >                      "id": 34,
> >                      "weight": 18559,
> >                      "pos": 4
> >                  },
> >                  {
> >                      "id": 35,
> >                      "weight": 18559,
> >                      "pos": 5
> >                  },
> >                  {
> >                      "id": 42,
> >                      "weight": 228923,
> >                      "pos": 6
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -15,
> >              "name": "FT1-NodeD~hdd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 487664,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": 22,
> >                      "weight": 121916,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": 23,
> >                      "weight": 121916,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": 21,
> >                      "weight": 121916,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": 20,
> >                      "weight": 121916,
> >                      "pos": 3
> >                  }
> >              ]
> >          },
> >          {
> >              "id": -16,
> >              "name": "FT1-NodeA~old-ssd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 0,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": []
> >          },
> >          {
> >              "id": -17,
> >              "name": "FT1-NodeB~old-ssd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 0,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": []
> >          },
> >          {
> >              "id": -18,
> >              "name": "FT1-NodeC~old-ssd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 0,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": []
> >          },
> >          {
> >              "id": -19,
> >              "name": "FT1-NodeD~old-ssd",
> >              "type_id": 1,
> >              "type_name": "host",
> >              "weight": 0,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": []
> >          },
> >          {
> >              "id": -20,
> >              "name": "default~old-ssd",
> >              "type_id": 11,
> >              "type_name": "root",
> >              "weight": 0,
> >              "alg": "straw2",
> >              "hash": "rjenkins1",
> >              "items": [
> >                  {
> >                      "id": -16,
> >                      "weight": 0,
> >                      "pos": 0
> >                  },
> >                  {
> >                      "id": -17,
> >                      "weight": 0,
> >                      "pos": 1
> >                  },
> >                  {
> >                      "id": -18,
> >                      "weight": 0,
> >                      "pos": 2
> >                  },
> >                  {
> >                      "id": -19,
> >                      "weight": 0,
> >                      "pos": 3
> >                  }
> >              ]
> >          }
> >      ],
> >      "rules": [
> >          {
> >              "rule_id": 0,
> >              "rule_name": "ssd_rule",
> >              "type": 1,
> >              "steps": [
> >                  {
> >                      "op": "take",
> >                      "item": -2,
> >                      "item_name": "default~ssd"
> >                  },
> >                  {
> >                      "op": "chooseleaf_firstn",
> >                      "num": 0,
> >                      "type": "host"
> >                  },
> >                  {
> >                      "op": "emit"
> >                  }
> >              ]
> >          },
> >          {
> >              "rule_id": 1,
> >              "rule_name": "hdd_rule",
> >              "type": 1,
> >              "steps": [
> >                  {
> >                      "op": "take",
> >                      "item": -6,
> >                      "item_name": "default~hdd"
> >                  },
> >                  {
> >                      "op": "chooseleaf_firstn",
> >                      "num": 0,
> >                      "type": "host"
> >                  },
> >                  {
> >                      "op": "emit"
> >                  }
> >              ]
> >          },
> >          {
> >              "rule_id": 2,
> >              "rule_name": "old_nvme",
> >              "type": 1,
> >              "steps": [
> >                  {
> >                      "op": "take",
> >                      "item": -20,
> >                      "item_name": "default~old-ssd"
> >                  },
> >                  {
> >                      "op": "chooseleaf_firstn",
> >                      "num": 0,
> >                      "type": "host"
> >                  },
> >                  {
> >                      "op": "emit"
> >                  }
> >              ]
> >          }
> >      ],
> >      "tunables": {
> >          "choose_local_tries": 0,
> >          "choose_local_fallback_tries": 0,
> >          "choose_total_tries": 50,
> >          "chooseleaf_descend_once": 1,
> >          "chooseleaf_vary_r": 1,
> >          "chooseleaf_stable": 1,
> >          "straw_calc_version": 1,
> >          "allowed_bucket_algs": 54,
> >          "profile": "jewel",
> >          "optimal_tunables": 1,
> >          "legacy_tunables": 0,
> >          "minimum_required_version": "jewel",
> >          "require_feature_tunables": 1,
> >          "require_feature_tunables2": 1,
> >          "has_v2_rules": 0,
> >          "require_feature_tunables3": 1,
> >          "has_v3_rules": 0,
> >          "has_v4_buckets": 1,
> >          "require_feature_tunables5": 1,
> >          "has_v5_rules": 0
> >      },
> >      "choose_args": {}
> > }
> >
> >
> >
> > > It may help to manually edit the CRUSH map as detailed here:
> > >
> > > https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/
> > >
> > >
> > > At the end of the decompiled CRUSH map is a “tunables” section.  Look for the “choose_total_tries”
> > > line, which should have the default value of 50.  Change that to 100.
> > >
> > >
> > >
> > > > On Nov 16, 2024, at 11:13 AM, Roland Giesler <roland@xxxxxxxxxxxxxx> <mailto:roland@xxxxxxxxxxxxxx> wrote:
> > > >
> > > > On 2024/11/15 16:37, Anthony D'Atri wrote:
> > > > > Only 2 OSDs in the acting set.  This isn’t a size=2 pool is it?  Did you restart osd.39?
> > > > Restarted both repeatedly.  Has no effect.
> > > >
> > > >
> > > >
> > > > > > On Nov 15, 2024, at 9:32 AM, Roland Giesler <roland@xxxxxxxxxxxxxx> <mailto:roland@xxxxxxxxxxxxxx> wrote:
> > > > > >
> > > > > >      "acting": [
> > > > > >          39,
> > > > > >          1
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx