Re: The effect of changing an osd's class

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 2024/11/16 18:38, Anthony D'Atri wrote:
Disabling mclock as described here https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/ might help

I cannot see any option that allows me to disable mclock...


Also, you have a small cluster with a bunch of small OSDs.  Please send `ceph health detail`

# ceph health detail
HEALTH_WARN 1 pool(s) have no replicas configured
[WRN] POOL_NO_REDUNDANCY: 1 pool(s) have no replicas configured
    pool 'backups' has no replicas configured

The backups pool is on spinners and 1 copy is sufficient since we're also replicating this offsite in real time.  Otherwise too many copies take too much space.


Please also send `ceph osd df` and `ceph osd crush dump`
NodeC:~# ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE   DATA     OMAP META      AVAIL    %USE   VAR   PGS  STATUS  2    hdd  1.86029   1.00000  1.9 TiB   1.1 TiB  1.1 TiB   16 MiB   2.7 GiB  746 GiB  60.97  1.23   68      up  3    hdd  1.86029   1.00000  1.9 TiB   949 GiB  902 GiB   12 MiB   2.6 GiB  961 GiB  49.68  1.00   57      up  4    hdd  1.86029   1.00000  1.9 TiB   848 GiB  800 GiB   10 MiB   2.3 GiB  1.0 TiB  44.37  0.89   51      up  5    hdd  1.86589   1.00000  1.9 TiB   1.0 TiB  982 GiB   15 MiB   2.4 GiB  881 GiB  53.87  1.08   66      up  0    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0    0    down  1    ssd  0.28319   1.00000  290 GiB   123 GiB  122 GiB  2.3 MiB   1.2 GiB  167 GiB  42.38  0.85   46      up 36    ssd        0   1.00000  290 GiB   1.3 GiB   23 MiB  2.8 MiB   1.2 GiB  289 GiB   0.44  0.01   13      up 37    ssd        0   1.00000  290 GiB   1.1 GiB   23 MiB  2.5 MiB   1.1 GiB  289 GiB   0.38  0.01   18      up 38    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0   41      up 39    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0   86      up 40    ssd  3.30690   1.00000  3.3 TiB   1.7 TiB  1.7 TiB   23 MiB   4.8 GiB  1.6 TiB  50.49  1.02  424      up 10    hdd  1.86029   1.00000  1.9 TiB   1.0 TiB  990 GiB   13 MiB   2.8 GiB  873 GiB  54.19  1.09   64      up 11    hdd  1.86029   1.00000  1.9 TiB   1.3 TiB  1.2 TiB   15 MiB   3.2 GiB  625 GiB  67.20  1.35   72      up 26    hdd  1.86029   1.00000  1.9 TiB   843 GiB  801 GiB   12 MiB   2.3 GiB  1.0 TiB  44.25  0.89   50      up 27    hdd  1.86029   1.00000  1.9 TiB   1.1 TiB  1.0 TiB   16 MiB   2.7 GiB  793 GiB  58.38  1.18   70      up  6    ssd  0.28319   1.00000  290 GiB   115 GiB  115 GiB  2.3 MiB   743 MiB  175 GiB  39.82  0.80   28      up  7    ssd  0.28319   1.00000  290 GiB    94 GiB   93 GiB  2.0 MiB   921 MiB  196 GiB  32.51  0.65   23      up  8    ssd  0.28319   1.00000  290 GiB   103 GiB  102 GiB  2.1 MiB   963 MiB  187 GiB  35.46  0.71   27      up  9    ssd  0.28319   1.00000  290 GiB   131 GiB  130 GiB  2.5 MiB   1.1 GiB  159 GiB  45.28  0.91   33      up 24    ssd  0.28319   1.00000  290 GiB   102 GiB  101 GiB  2.1 MiB   719 MiB  188 GiB  35.10  0.71   26      up 25    ssd  0.28319   1.00000  290 GiB   152 GiB  151 GiB  2.8 MiB 1020 MiB  138 GiB  52.52  1.06   39      up 41    ssd  3.49309   1.00000  3.5 TiB   1.6 TiB  1.6 TiB   22 MiB   5.4 GiB  1.9 TiB  44.58  0.90  411      up 14    hdd  1.59999   1.00000  1.9 TiB   1.0 TiB  1.0 TiB   13 MiB   2.4 GiB  837 GiB  56.04  1.13   63      up 15    hdd  1.86029   1.00000  1.9 TiB  1017 GiB  975 GiB   13 MiB   2.1 GiB  888 GiB  53.36  1.07   60      up 16    hdd  1.86029   1.00000  1.9 TiB   990 GiB  948 GiB   13 MiB   2.2 GiB  915 GiB  51.97  1.05   59      up 17    hdd  1.86029   1.00000  1.9 TiB   1.0 TiB  992 GiB   14 MiB   2.5 GiB  871 GiB  54.28  1.09   64      up 12    ssd  0.28319   1.00000  290 GiB   122 GiB  121 GiB  2.3 MiB   1.1 GiB  168 GiB  41.96  0.84   31      up 13    ssd  0.28319   1.00000  290 GiB   145 GiB  144 GiB  2.6 MiB   1.2 GiB  145 GiB  49.95  1.01   37      up 28    ssd  0.28319   1.00000  290 GiB   142 GiB  141 GiB  2.6 MiB   1.1 GiB  148 GiB  48.89  0.98   35      up 29    ssd  0.28319   1.00000  290 GiB   109 GiB  108 GiB  2.2 MiB   1.3 GiB  181 GiB  37.58  0.76   29      up 30    ssd  0.28319   1.00000  290 GiB   146 GiB  145 GiB  2.7 MiB   1.3 GiB  144 GiB  50.37  1.01   37      up 31    ssd  0.28319   1.00000  290 GiB   131 GiB  130 GiB  2.5 MiB   1.3 GiB  159 GiB  45.24  0.91   34      up 43    ssd  3.30690   1.00000  3.3 TiB   1.4 TiB  1.4 TiB   19 MiB   4.0 GiB  1.9 TiB  42.50  0.86  353      up 20    hdd  1.86029   1.00000  1.9 TiB   1.2 TiB  1.2 TiB   12 MiB   2.8 GiB  659 GiB  65.38  1.32   67      up 21    hdd  1.86029   1.00000  1.9 TiB   1.1 TiB  1.0 TiB   15 MiB   2.8 GiB  815 GiB  57.20  1.15   68      up 22    hdd  1.86029   1.00000  1.9 TiB   878 GiB  836 GiB   13 MiB   2.4 GiB  1.0 TiB  46.10  0.93   54      up 23    hdd  1.86029   1.00000  1.9 TiB  1018 GiB  977 GiB   14 MiB   2.8 GiB  886 GiB  53.46  1.08   59      up 18    ssd  0.28319   1.00000  290 GiB   115 GiB  114 GiB  2.3 MiB   1.3 GiB  175 GiB  39.74  0.80   29      up 19    ssd  0.28319   1.00000  290 GiB   115 GiB  114 GiB  2.3 MiB   961 MiB  175 GiB  39.59  0.80   28      up 32    ssd  0.28319   1.00000  290 GiB   103 GiB  102 GiB  2.1 MiB   1.0 GiB  187 GiB  35.65  0.72   26      up 33    ssd  0.28319   1.00000  290 GiB    98 GiB   97 GiB  2.1 MiB   1.4 GiB  192 GiB  33.92  0.68   26      up 34    ssd  0.28319   1.00000  290 GiB   141 GiB  140 GiB  2.6 MiB   987 MiB  149 GiB  48.49  0.98   35      up 35    ssd  0.28319   1.00000  290 GiB   116 GiB  114 GiB  2.3 MiB   1.3 GiB  174 GiB  39.92  0.80   31      up 42    ssd  3.49309   1.00000  3.5 TiB   1.4 TiB  1.4 TiB   20 MiB   4.5 GiB  2.1 TiB  40.45  0.81  372      up                        TOTAL   49 TiB    25 TiB   24 TiB  351 MiB    83 GiB   25 TiB  49.67

MIN/MAX VAR: 0.01/1.35  STDDEV: 13.87

NodeC:~# ceph osd crush dump
{
    "devices": [
        {
            "id": 0,
            "name": "osd.0",
            "class": "ssd"
        },
        {
            "id": 1,
            "name": "osd.1",
            "class": "ssd"
        },
        {
            "id": 2,
            "name": "osd.2",
            "class": "hdd"
        },
        {
            "id": 3,
            "name": "osd.3",
            "class": "hdd"
        },
        {
            "id": 4,
            "name": "osd.4",
            "class": "hdd"
        },
        {
            "id": 5,
            "name": "osd.5",
            "class": "hdd"
        },
        {
            "id": 6,
            "name": "osd.6",
            "class": "ssd"
        },
        {
            "id": 7,
            "name": "osd.7",
            "class": "ssd"
        },
        {
            "id": 8,
            "name": "osd.8",
            "class": "ssd"
        },
        {
            "id": 9,
            "name": "osd.9",
            "class": "ssd"
        },
        {
            "id": 10,
            "name": "osd.10",
            "class": "hdd"
        },
        {
            "id": 11,
            "name": "osd.11",
            "class": "hdd"
        },
        {
            "id": 12,
            "name": "osd.12",
            "class": "ssd"
        },
        {
            "id": 13,
            "name": "osd.13",
            "class": "ssd"
        },
        {
            "id": 14,
            "name": "osd.14",
            "class": "hdd"
        },
        {
            "id": 15,
            "name": "osd.15",
            "class": "hdd"
        },
        {
            "id": 16,
            "name": "osd.16",
            "class": "hdd"
        },
        {
            "id": 17,
            "name": "osd.17",
            "class": "hdd"
        },
        {
            "id": 18,
            "name": "osd.18",
            "class": "ssd"
        },
        {
            "id": 19,
            "name": "osd.19",
            "class": "ssd"
        },
        {
            "id": 20,
            "name": "osd.20",
            "class": "hdd"
        },
        {
            "id": 21,
            "name": "osd.21",
            "class": "hdd"
        },
        {
            "id": 22,
            "name": "osd.22",
            "class": "hdd"
        },
        {
            "id": 23,
            "name": "osd.23",
            "class": "hdd"
        },
        {
            "id": 24,
            "name": "osd.24",
            "class": "ssd"
        },
        {
            "id": 25,
            "name": "osd.25",
            "class": "ssd"
        },
        {
            "id": 26,
            "name": "osd.26",
            "class": "hdd"
        },
        {
            "id": 27,
            "name": "osd.27",
            "class": "hdd"
        },
        {
            "id": 28,
            "name": "osd.28",
            "class": "ssd"
        },
        {
            "id": 29,
            "name": "osd.29",
            "class": "ssd"
        },
        {
            "id": 30,
            "name": "osd.30",
            "class": "ssd"
        },
        {
            "id": 31,
            "name": "osd.31",
            "class": "ssd"
        },
        {
            "id": 32,
            "name": "osd.32",
            "class": "ssd"
        },
        {
            "id": 33,
            "name": "osd.33",
            "class": "ssd"
        },
        {
            "id": 34,
            "name": "osd.34",
            "class": "ssd"
        },
        {
            "id": 35,
            "name": "osd.35",
            "class": "ssd"
        },
        {
            "id": 36,
            "name": "osd.36",
            "class": "ssd"
        },
        {
            "id": 37,
            "name": "osd.37",
            "class": "ssd"
        },
        {
            "id": 38,
            "name": "osd.38",
            "class": "ssd"
        },
        {
            "id": 39,
            "name": "osd.39",
            "class": "ssd"
        },
        {
            "id": 40,
            "name": "osd.40",
            "class": "ssd"
        },
        {
            "id": 41,
            "name": "osd.41",
            "class": "ssd"
        },
        {
            "id": 42,
            "name": "osd.42",
            "class": "ssd"
        },
        {
            "id": 43,
            "name": "osd.43",
            "class": "ssd"
        }
    ],
    "types": [
        {
            "type_id": 0,
            "name": "osd"
        },
        {
            "type_id": 1,
            "name": "host"
        },
        {
            "type_id": 2,
            "name": "chassis"
        },
        {
            "type_id": 3,
            "name": "rack"
        },
        {
            "type_id": 4,
            "name": "row"
        },
        {
            "type_id": 5,
            "name": "pdu"
        },
        {
            "type_id": 6,
            "name": "pod"
        },
        {
            "type_id": 7,
            "name": "room"
        },
        {
            "type_id": 8,
            "name": "datacenter"
        },
        {
            "type_id": 9,
            "name": "zone"
        },
        {
            "type_id": 10,
            "name": "region"
        },
        {
            "type_id": 11,
            "name": "root"
        }
    ],
    "buckets": [
        {
            "id": -1,
            "name": "default",
            "type_id": 11,
            "type_name": "root",
            "weight": 3177873,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -3,
                    "weight": 723311,
                    "pos": 0
                },
                {
                    "id": -7,
                    "weight": 827941,
                    "pos": 1
                },
                {
                    "id": -10,
                    "weight": 798680,
                    "pos": 2
                },
                {
                    "id": -13,
                    "weight": 827941,
                    "pos": 3
                }
            ]
        },
        {
            "id": -2,
            "name": "default~ssd",
            "type_id": 11,
            "type_name": "root",
            "weight": 1243909,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -4,
                    "weight": 235280,
                    "pos": 0
                },
                {
                    "id": -8,
                    "weight": 340277,
                    "pos": 1
                },
                {
                    "id": -11,
                    "weight": 328075,
                    "pos": 2
                },
                {
                    "id": -14,
                    "weight": 340277,
                    "pos": 3
                }
            ]
        },
        {
            "id": -3,
            "name": "FT1-NodeA",
            "type_id": 1,
            "type_name": "host",
            "weight": 723311,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 1,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 36,
                    "weight": 0,
                    "pos": 1
                },
                {
                    "id": 37,
                    "weight": 0,
                    "pos": 2
                },
                {
                    "id": 0,
                    "weight": 0,
                    "pos": 3
                },
                {
                    "id": 38,
                    "weight": 0,
                    "pos": 4
                },
                {
                    "id": 39,
                    "weight": 0,
                    "pos": 5
                },
                {
                    "id": 2,
                    "weight": 121916,
                    "pos": 6
                },
                {
                    "id": 3,
                    "weight": 121916,
                    "pos": 7
                },
                {
                    "id": 4,
                    "weight": 121916,
                    "pos": 8
                },
                {
                    "id": 40,
                    "weight": 216721,
                    "pos": 9
                },
                {
                    "id": 5,
                    "weight": 122283,
                    "pos": 10
                }
            ]
        },
        {
            "id": -4,
            "name": "FT1-NodeA~ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 235280,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 1,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 36,
                    "weight": 0,
                    "pos": 1
                },
                {
                    "id": 37,
                    "weight": 0,
                    "pos": 2
                },
                {
                    "id": 0,
                    "weight": 0,
                    "pos": 3
                },
                {
                    "id": 38,
                    "weight": 0,
                    "pos": 4
                },
                {
                    "id": 39,
                    "weight": 0,
                    "pos": 5
                },
                {
                    "id": 40,
                    "weight": 216721,
                    "pos": 6
                }
            ]
        },
        {
            "id": -5,
            "name": "FT1-NodeA~hdd",
            "type_id": 1,
            "type_name": "host",
            "weight": 488031,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 2,
                    "weight": 121916,
                    "pos": 0
                },
                {
                    "id": 3,
                    "weight": 121916,
                    "pos": 1
                },
                {
                    "id": 4,
                    "weight": 121916,
                    "pos": 2
                },
                {
                    "id": 5,
                    "weight": 122283,
                    "pos": 3
                }
            ]
        },
        {
            "id": -6,
            "name": "default~hdd",
            "type_id": 11,
            "type_name": "root",
            "weight": 1933964,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -5,
                    "weight": 488031,
                    "pos": 0
                },
                {
                    "id": -9,
                    "weight": 487664,
                    "pos": 1
                },
                {
                    "id": -12,
                    "weight": 470605,
                    "pos": 2
                },
                {
                    "id": -15,
                    "weight": 487664,
                    "pos": 3
                }
            ]
        },
        {
            "id": -7,
            "name": "FT1-NodeB",
            "type_id": 1,
            "type_name": "host",
            "weight": 827941,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 6,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 24,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 25,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 11,
                    "weight": 121916,
                    "pos": 3
                },
                {
                    "id": 10,
                    "weight": 121916,
                    "pos": 4
                },
                {
                    "id": 7,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 8,
                    "weight": 18559,
                    "pos": 6
                },
                {
                    "id": 9,
                    "weight": 18559,
                    "pos": 7
                },
                {
                    "id": 26,
                    "weight": 121916,
                    "pos": 8
                },
                {
                    "id": 27,
                    "weight": 121916,
                    "pos": 9
                },
                {
                    "id": 41,
                    "weight": 228923,
                    "pos": 10
                }
            ]
        },
        {
            "id": -8,
            "name": "FT1-NodeB~ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 340277,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 6,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 24,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 25,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 7,
                    "weight": 18559,
                    "pos": 3
                },
                {
                    "id": 8,
                    "weight": 18559,
                    "pos": 4
                },
                {
                    "id": 9,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 41,
                    "weight": 228923,
                    "pos": 6
                }
            ]
        },
        {
            "id": -9,
            "name": "FT1-NodeB~hdd",
            "type_id": 1,
            "type_name": "host",
            "weight": 487664,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 11,
                    "weight": 121916,
                    "pos": 0
                },
                {
                    "id": 10,
                    "weight": 121916,
                    "pos": 1
                },
                {
                    "id": 26,
                    "weight": 121916,
                    "pos": 2
                },
                {
                    "id": 27,
                    "weight": 121916,
                    "pos": 3
                }
            ]
        },
        {
            "id": -10,
            "name": "FT1-NodeC",
            "type_id": 1,
            "type_name": "host",
            "weight": 798680,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 13,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 28,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 29,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 14,
                    "weight": 104857,
                    "pos": 3
                },
                {
                    "id": 12,
                    "weight": 18559,
                    "pos": 4
                },
                {
                    "id": 30,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 31,
                    "weight": 18559,
                    "pos": 6
                },
                {
                    "id": 15,
                    "weight": 121916,
                    "pos": 7
                },
                {
                    "id": 16,
                    "weight": 121916,
                    "pos": 8
                },
                {
                    "id": 17,
                    "weight": 121916,
                    "pos": 9
                },
                {
                    "id": 43,
                    "weight": 216721,
                    "pos": 10
                }
            ]
        },
        {
            "id": -11,
            "name": "FT1-NodeC~ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 328075,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 13,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 28,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 29,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 12,
                    "weight": 18559,
                    "pos": 3
                },
                {
                    "id": 30,
                    "weight": 18559,
                    "pos": 4
                },
                {
                    "id": 31,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 43,
                    "weight": 216721,
                    "pos": 6
                }
            ]
        },
        {
            "id": -12,
            "name": "FT1-NodeC~hdd",
            "type_id": 1,
            "type_name": "host",
            "weight": 470605,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 14,
                    "weight": 104857,
                    "pos": 0
                },
                {
                    "id": 15,
                    "weight": 121916,
                    "pos": 1
                },
                {
                    "id": 16,
                    "weight": 121916,
                    "pos": 2
                },
                {
                    "id": 17,
                    "weight": 121916,
                    "pos": 3
                }
            ]
        },
        {
            "id": -13,
            "name": "FT1-NodeD",
            "type_id": 1,
            "type_name": "host",
            "weight": 827941,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 18,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 32,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 33,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 22,
                    "weight": 121916,
                    "pos": 3
                },
                {
                    "id": 19,
                    "weight": 18559,
                    "pos": 4
                },
                {
                    "id": 34,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 35,
                    "weight": 18559,
                    "pos": 6
                },
                {
                    "id": 23,
                    "weight": 121916,
                    "pos": 7
                },
                {
                    "id": 21,
                    "weight": 121916,
                    "pos": 8
                },
                {
                    "id": 20,
                    "weight": 121916,
                    "pos": 9
                },
                {
                    "id": 42,
                    "weight": 228923,
                    "pos": 10
                }
            ]
        },
        {
            "id": -14,
            "name": "FT1-NodeD~ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 340277,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 18,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 32,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 33,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 19,
                    "weight": 18559,
                    "pos": 3
                },
                {
                    "id": 34,
                    "weight": 18559,
                    "pos": 4
                },
                {
                    "id": 35,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 42,
                    "weight": 228923,
                    "pos": 6
                }
            ]
        },
        {
            "id": -15,
            "name": "FT1-NodeD~hdd",
            "type_id": 1,
            "type_name": "host",
            "weight": 487664,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 22,
                    "weight": 121916,
                    "pos": 0
                },
                {
                    "id": 23,
                    "weight": 121916,
                    "pos": 1
                },
                {
                    "id": 21,
                    "weight": 121916,
                    "pos": 2
                },
                {
                    "id": 20,
                    "weight": 121916,
                    "pos": 3
                }
            ]
        },
        {
            "id": -16,
            "name": "FT1-NodeA~old-ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 0,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": []
        },
        {
            "id": -17,
            "name": "FT1-NodeB~old-ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 0,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": []
        },
        {
            "id": -18,
            "name": "FT1-NodeC~old-ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 0,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": []
        },
        {
            "id": -19,
            "name": "FT1-NodeD~old-ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 0,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": []
        },
        {
            "id": -20,
            "name": "default~old-ssd",
            "type_id": 11,
            "type_name": "root",
            "weight": 0,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -16,
                    "weight": 0,
                    "pos": 0
                },
                {
                    "id": -17,
                    "weight": 0,
                    "pos": 1
                },
                {
                    "id": -18,
                    "weight": 0,
                    "pos": 2
                },
                {
                    "id": -19,
                    "weight": 0,
                    "pos": 3
                }
            ]
        }
    ],
    "rules": [
        {
            "rule_id": 0,
            "rule_name": "ssd_rule",
            "type": 1,
            "steps": [
                {
                    "op": "take",
                    "item": -2,
                    "item_name": "default~ssd"
                },
                {
                    "op": "chooseleaf_firstn",
                    "num": 0,
                    "type": "host"
                },
                {
                    "op": "emit"
                }
            ]
        },
        {
            "rule_id": 1,
            "rule_name": "hdd_rule",
            "type": 1,
            "steps": [
                {
                    "op": "take",
                    "item": -6,
                    "item_name": "default~hdd"
                },
                {
                    "op": "chooseleaf_firstn",
                    "num": 0,
                    "type": "host"
                },
                {
                    "op": "emit"
                }
            ]
        },
        {
            "rule_id": 2,
            "rule_name": "old_nvme",
            "type": 1,
            "steps": [
                {
                    "op": "take",
                    "item": -20,
                    "item_name": "default~old-ssd"
                },
                {
                    "op": "chooseleaf_firstn",
                    "num": 0,
                    "type": "host"
                },
                {
                    "op": "emit"
                }
            ]
        }
    ],
    "tunables": {
        "choose_local_tries": 0,
        "choose_local_fallback_tries": 0,
        "choose_total_tries": 50,
        "chooseleaf_descend_once": 1,
        "chooseleaf_vary_r": 1,
        "chooseleaf_stable": 1,
        "straw_calc_version": 1,
        "allowed_bucket_algs": 54,
        "profile": "jewel",
        "optimal_tunables": 1,
        "legacy_tunables": 0,
        "minimum_required_version": "jewel",
        "require_feature_tunables": 1,
        "require_feature_tunables2": 1,
        "has_v2_rules": 0,
        "require_feature_tunables3": 1,
        "has_v3_rules": 0,
        "has_v4_buckets": 1,
        "require_feature_tunables5": 1,
        "has_v5_rules": 0
    },
    "choose_args": {}
}



It may help to manually edit the CRUSH map as detailed here:

https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/


At the end of the decompiled CRUSH map is a “tunables” section.  Look for the “c|hoose_total_tries|”
line, which should have the default value of 50.  Change that to 100.



On Nov 16, 2024, at 11:13 AM, Roland Giesler <roland@xxxxxxxxxxxxxx> wrote:

On 2024/11/15 16:37, Anthony D'Atri wrote:
Only 2 OSDs in the acting set.  This isn’t a size=2 pool is it?  Did you restart osd.39?

Restarted both repeatedly.  Has no effect.



On Nov 15, 2024, at 9:32 AM, Roland Giesler <roland@xxxxxxxxxxxxxx> wrote:

"acting": [
39,
1


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux