Re: no recovery running

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I was running into that as well. Setting
`osd_mclock_override_recovery_settings` [1] to true allowed me to manage
osd_max_backfills again and get recovery to start happening again. It's on
my todo list to understand mclock profiles, but resizing PGs was a
nightmare with it. Changing to override the recovery settings saved me.

[1]
https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/#confval-osd_mclock_override_recovery_settings

On Fri, Oct 25, 2024 at 11:13 AM Joffrey <joff.au@xxxxxxxxx> wrote:

> HI,
>
>
> This is my cluster:
>
>   cluster:
>     id:     c300532c-51fa-11ec-9a41-0050569c3b55
>     health: HEALTH_WARN
>             Degraded data redundancy: 2062374/1331064781 objects degraded
> (0.155%), 278 pgs degraded, 40 pgs undersized
>             2497 pgs not deep-scrubbed in time
>             2497 pgs not scrubbed in time
>
>   services:
>     mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3
> (age 9d)
>     mgr: hbgt-ceph1-mon3.gmfzqm(active, since 10d), standbys:
> hbgt-ceph1-mon2.nteihj, hbgt-ceph1-mon1.thrnnu
>     osd: 96 osds: 96 up (since 9d), 96 in (since 45h); 1588 remapped pgs
>     rgw: 3 daemons active (3 hosts, 2 zones)
>
>   data:
>     pools:   16 pools, 2497 pgs
>     objects: 266.22M objects, 518 TiB
>     usage:   976 TiB used, 808 TiB / 1.7 PiB avail
>     pgs:     2062374/1331064781 objects degraded (0.155%)
>              349917519/1331064781 objects misplaced (26.289%)
>              1312 active+remapped+backfill_wait
>              864  active+clean
>              199  active+recovery_wait+degraded+remapped
>              38   active+recovery_wait+degraded
>              33   active+undersized+degraded+remapped+backfill_wait
>              33   active+recovery_wait+remapped
>              7    active+recovery_wait
>              6    active+undersized+degraded+remapped+backfilling
>              2    active+recovering+remapped
>              1    active+remapped+backfilling
>              1    active+recovering+degraded+remapped
>              1    active+recovery_wait+undersized+degraded+remapped
>
>   io:
>     client:   683 KiB/s rd, 2.2 KiB/s wr, 51 op/s rd, 2 op/s wr
>
>
> No recovery is running and I don't understand why.
> I have free space:
>
> ID   CLASS  WEIGHT      REWEIGHT  SIZE     RAW USE   DATA     OMAP     META
>      AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
>  -1         1784.12231         -  1.7 PiB   976 TiB  895 TiB  298 GiB   4.1
> TiB  808 TiB  54.72  1.00    -          root default
>  -5          208.09680         -  208 TiB   142 TiB  130 TiB   51 GiB   605
> GiB   66 TiB  68.14  1.25    -              host hbgt-ceph1-osd01
>   1    hdd    17.34140   1.00000   17 TiB    11 TiB   11 TiB   33 KiB    49
> GiB  5.9 TiB  66.16  1.21  136      up          osd.1
>   3    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB   23 GiB    49
> GiB  6.3 TiB  63.80  1.17  139      up          osd.3
>   5    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  139 MiB    53
> GiB  4.8 TiB  72.31  1.32  142      up          osd.5
>   7    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   11 GiB    51
> GiB  5.6 TiB  67.97  1.24  145      up          osd.7
>   9    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB  2.2 GiB    49
> GiB  6.0 TiB  65.67  1.20  140      up          osd.9
>  11    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  329 MiB    50
> GiB  5.5 TiB  68.42  1.25  145      up          osd.11
>  13    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.5 GiB    52
> GiB  5.1 TiB  70.45  1.29  153      up          osd.13
>  15    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   61 KiB    48
> GiB  5.7 TiB  66.85  1.22  144      up          osd.15
>  17    hdd    17.34140   1.00000   17 TiB    11 TiB  9.5 TiB  272 MiB    45
> GiB  6.8 TiB  60.63  1.11  120      up          osd.17
>  19    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB   12 GiB    50
> GiB  5.9 TiB  65.90  1.20  134      up          osd.19
>  21    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  1.6 GiB    57
> GiB  4.1 TiB  76.49  1.40  152      up          osd.21
>  23    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   31 KiB    54
> GiB  4.7 TiB  73.10  1.34  124      up          osd.23
>  -3          208.09680         -  208 TiB   146 TiB  134 TiB   64 GiB   629
> GiB   62 TiB  70.05  1.28    -              host hbgt-ceph1-osd02
>   0    hdd    17.34140   1.00000   17 TiB    11 TiB  9.8 TiB   22 GiB    49
> GiB  6.6 TiB  62.07  1.13  124      up          osd.0
>   2    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.7 GiB    52
> GiB  5.2 TiB  70.14  1.28  150      up          osd.2
>   4    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.8 GiB    48
> GiB  5.8 TiB  66.83  1.22  152      up          osd.4
>   6    hdd    17.34140   0.85004   17 TiB    13 TiB   12 TiB   11 GiB    58
> GiB  4.0 TiB  76.85  1.40  153      up          osd.6
>   8    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   11 GiB    54
> GiB  4.9 TiB  71.58  1.31  152      up          osd.8
>  10    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB  6.3 MiB    47
> GiB  6.1 TiB  64.91  1.19  133      up          osd.10
>  12    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  109 MiB    51
> GiB  5.6 TiB  67.72  1.24  137      up          osd.12
>  14    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   12 GiB    53
> GiB  5.1 TiB  70.37  1.29  148      up          osd.14
>  16    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  2.9 GiB    54
> GiB  4.9 TiB  71.65  1.31  145      up          osd.16
>  18    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   88 MiB    55
> GiB  4.8 TiB  72.45  1.32  154      up          osd.18
>  20    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  108 MiB    55
> GiB  4.6 TiB  73.39  1.34  166      up          osd.20
>  22    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  209 MiB    54
> GiB  4.7 TiB  72.68  1.33  138      up          osd.22
>  -7          208.09680         -  208 TiB   148 TiB  136 TiB   29 GiB   635
> GiB   60 TiB  71.03  1.30    -              host hbgt-ceph1-osd03
>  24    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  1.4 GiB    56
> GiB  4.5 TiB  74.27  1.36  150      up          osd.24
>  25    hdd    17.34140   0.95001   17 TiB    13 TiB   12 TiB   32 KiB    57
> GiB  4.1 TiB  76.27  1.39  162      up          osd.25
>  26    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB  1.4 GiB    49
> GiB  6.0 TiB  65.30  1.19  134      up          osd.26
>  27    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  1.7 GiB    53
> GiB  4.7 TiB  73.16  1.34  152      up          osd.27
>  28    hdd    17.34140   1.00000   17 TiB    12 TiB   12 TiB  2.3 MiB    52
> GiB  4.9 TiB  71.97  1.32  158      up          osd.28
>  29    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   10 GiB    52
> GiB  5.3 TiB  69.72  1.27  141      up          osd.29
>  30    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  266 MiB    54
> GiB  4.6 TiB  73.38  1.34  142      up          osd.30
>  31    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   93 MiB    50
> GiB  5.7 TiB  67.36  1.23  145      up          osd.31
>  32    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB   13 GiB    50
> GiB  6.1 TiB  64.77  1.18  131      up          osd.32
>  33    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   12 MiB    53
> GiB  5.2 TiB  69.77  1.28  132      up          osd.33
>  34    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  242 MiB    50
> GiB  5.5 TiB  68.17  1.25  139      up          osd.34
>  35    hdd    17.34140   1.00000   17 TiB    14 TiB   13 TiB  135 MiB    58
> GiB  3.8 TiB  78.26  1.43  157      up          osd.35
>  -9          208.09680         -  208 TiB   146 TiB  134 TiB   38 GiB   627
> GiB   62 TiB  70.19  1.28    -              host hbgt-ceph1-osd04
>  36    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB  210 MiB    47
> GiB  6.4 TiB  63.16  1.15  139      up          osd.36
>  37    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  222 MiB    53
> GiB  5.0 TiB  71.34  1.30  147      up          osd.37
>  38    hdd    17.34140   0.95001   17 TiB    13 TiB   12 TiB   13 MiB    55
> GiB  4.5 TiB  74.33  1.36  155      up          osd.38
>  39    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  135 MiB    52
> GiB  5.2 TiB  70.22  1.28  140      up          osd.39
>  40    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   25 MiB    53
> GiB  4.9 TiB  71.46  1.31  150      up          osd.40
>  41    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.8 GiB    54
> GiB  5.0 TiB  71.36  1.30  150      up          osd.41
>  42    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  823 MiB    50
> GiB  5.8 TiB  66.49  1.22  156      up          osd.42
>  43    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.5 GiB    52
> GiB  5.2 TiB  69.97  1.28  160      up          osd.43
>  44    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   11 GiB    52
> GiB  5.3 TiB  69.35  1.27  133      up          osd.44
>  45    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   12 GiB    55
> GiB  4.8 TiB  72.12  1.32  148      up          osd.45
>  46    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   11 GiB    52
> GiB  5.1 TiB  70.74  1.29  148      up          osd.46
>  47    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  188 MiB    54
> GiB  4.9 TiB  71.74  1.31  151      up          osd.47
> -11          208.09680         -  208 TiB   147 TiB  135 TiB   27 GiB   627
> GiB   62 TiB  70.45  1.29    -              host hbgt-ceph1-osd05
>  48    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  150 MiB    49
> GiB  5.7 TiB  67.11  1.23  152      up          osd.48
>  49    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  150 MiB    52
> GiB  5.1 TiB  70.87  1.30  155      up          osd.49
>  50    hdd    17.34140   0.90002   17 TiB    12 TiB   11 TiB   11 GiB    54
> GiB  5.2 TiB  70.01  1.28  147      up          osd.50
>  51    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   84 MiB    54
> GiB  4.6 TiB  73.24  1.34  147      up          osd.51
>  52    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  337 MiB    53
> GiB  4.8 TiB  72.26  1.32  146      up          osd.52
>  53    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.6 GiB    54
> GiB  5.1 TiB  70.56  1.29  132      up          osd.53
>  54    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  1.3 GiB    56
> GiB  4.5 TiB  73.77  1.35  138      up          osd.54
>  55    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   10 GiB    53
> GiB  4.8 TiB  72.59  1.33  148      up          osd.55
>  56    hdd    17.34140   1.00000   17 TiB    11 TiB  9.6 TiB  1.8 GiB    46
> GiB  6.7 TiB  61.16  1.12  127      up          osd.56
>  57    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  3.1 MiB    50
> GiB  5.6 TiB  67.96  1.24  148      up          osd.57
>  58    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  687 KiB    52
> GiB  5.0 TiB  70.93  1.30  152      up          osd.58
>  59    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   22 KiB    55
> GiB  4.4 TiB  74.88  1.37  147      up          osd.59
> -13          245.93866         -  246 TiB   112 TiB  106 TiB   28 GiB   498
> GiB  134 TiB  45.56  0.83    -              host hbgt-ceph1-osd06
>  60    hdd    20.49489   1.00000   20 TiB    11 TiB   11 TiB   10 GiB    51
> GiB  9.3 TiB  54.43  0.99  134      up          osd.60
>  61    hdd    20.49489   1.00000   20 TiB   9.6 TiB  9.2 TiB  216 MiB    43
> GiB   11 TiB  47.06  0.86  120      up          osd.61
>  62    hdd    20.49489   1.00000   20 TiB   9.1 TiB  8.6 TiB  330 MiB    41
> GiB   11 TiB  44.53  0.81  116      up          osd.62
>  63    hdd    20.49489   1.00000   20 TiB   8.9 TiB  8.4 TiB  354 MiB    39
> GiB   12 TiB  43.29  0.79  113      up          osd.63
>  64    hdd    20.49489   1.00000   20 TiB   8.1 TiB  7.7 TiB  148 MiB    36
> GiB   12 TiB  39.72  0.73  117      up          osd.64
>  65    hdd    20.49489   1.00000   20 TiB    12 TiB   12 TiB   14 KiB    53
> GiB  8.4 TiB  58.90  1.08  159      up          osd.65
>  66    hdd    20.49489   1.00000   20 TiB    11 TiB   10 TiB   18 MiB    49
> GiB  9.8 TiB  52.33  0.96  129      up          osd.66
>  67    hdd    20.49489   1.00000   20 TiB   498 GiB  1.5 GiB    1 KiB   745
> MiB   20 TiB   2.37  0.04    6      up          osd.67
>  68    hdd    20.49489   1.00000   20 TiB    10 TiB  9.8 TiB   13 GiB    47
> GiB   10 TiB  50.12  0.92  138      up          osd.68
>  69    hdd    20.49489   1.00000   20 TiB    11 TiB   10 TiB  2.0 GiB    48
> GiB  9.7 TiB  52.77  0.96  135      up          osd.69
>  70    hdd    20.49489   1.00000   20 TiB    11 TiB   11 TiB  1.6 GiB    50
> GiB  9.2 TiB  55.24  1.01  150      up          osd.70
>  71    hdd    20.49489   1.00000   20 TiB   9.4 TiB  8.9 TiB   86 MiB    42
> GiB   11 TiB  45.96  0.84  130      up          osd.71
> -15          245.93866         -  246 TiB   114 TiB  108 TiB   62 GiB   505
> GiB  132 TiB  46.18  0.84    -              host hbgt-ceph1-osd07
>  72    hdd    20.49489   1.00000   20 TiB    11 TiB   10 TiB   12 GiB    49
> GiB  9.6 TiB  53.39  0.98  146      up          osd.72
>  73    hdd    20.49489   1.00000   20 TiB   8.5 TiB  8.0 TiB  194 MiB    38
> GiB   12 TiB  41.31  0.75  112      up          osd.73
>  74    hdd    20.49489   1.00000   20 TiB   9.5 TiB  9.0 TiB   10 GiB    42
> GiB   11 TiB  46.12  0.84  124      up          osd.74
>  75    hdd    20.49489   1.00000   20 TiB   9.1 TiB  8.6 TiB   12 GiB    41
> GiB   11 TiB  44.16  0.81  125      up          osd.75
>  76    hdd    20.49489   1.00000   20 TiB   9.4 TiB  8.9 TiB  114 MiB    41
> GiB   11 TiB  45.91  0.84  111      up          osd.76
>  77    hdd    20.49489   1.00000   20 TiB   9.6 TiB  9.2 TiB  1.5 GiB    42
> GiB   11 TiB  47.03  0.86  127      up          osd.77
>  78    hdd    20.49489   1.00000   20 TiB   7.7 TiB  7.3 TiB   12 GiB    36
> GiB   13 TiB  37.77  0.69  105      up          osd.78
>  79    hdd    20.49489   1.00000   20 TiB   8.2 TiB  7.7 TiB    1 KiB    36
> GiB   12 TiB  39.92  0.73   91      up          osd.79
>  80    hdd    20.49489   1.00000   20 TiB   9.5 TiB  9.0 TiB  371 MiB    41
> GiB   11 TiB  46.13  0.84  132      up          osd.80
>  81    hdd    20.49489   1.00000   20 TiB   9.8 TiB  9.3 TiB  353 MiB    45
> GiB   11 TiB  47.95  0.88  121      up          osd.81
>  82    hdd    20.49489   1.00000   20 TiB    11 TiB   11 TiB   12 GiB    51
> GiB  9.2 TiB  55.23  1.01  142      up          osd.82
>  83    hdd    20.49489   1.00000   20 TiB    10 TiB  9.6 TiB  1.9 GiB    44
> GiB   10 TiB  49.28  0.90  125      up          osd.83
> -17          251.76105         -  252 TiB    23 TiB   11 TiB   43 MiB    64
> GiB  229 TiB   8.99  0.16    -              host hbgt-ceph1-osd08
>  84    hdd    20.98009   1.00000   21 TiB   1.0 TiB   65 GiB    1 KiB   1.4
> GiB   20 TiB   4.93  0.09    5      up          osd.84
>  85    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB   813
> MiB   20 TiB   4.63  0.08    7      up          osd.85
>  86    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB   1.0
> GiB   20 TiB   4.63  0.08    2      up          osd.86
>  87    hdd    20.98009   1.00000   21 TiB  1008 GiB   14 GiB    1 KiB   1.0
> GiB   20 TiB   4.69  0.09    7      up          osd.87
>  88    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB   1.6
> GiB   20 TiB   4.63  0.08    6      up          osd.88
>  89    hdd    20.98009   1.00000   21 TiB   4.9 TiB  4.0 TiB   17 MiB    20
> GiB   16 TiB  23.56  0.43   51      up          osd.89
>  90    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB   1.5
> GiB   20 TiB   4.63  0.08    5      up          osd.90
>  91    hdd    20.98009   1.00000   21 TiB   3.3 TiB  2.3 TiB    1 KiB    11
> GiB   18 TiB  15.70  0.29   40      up          osd.91
>  92    hdd    20.98009   1.00000   21 TiB   5.1 TiB  4.1 TiB   26 MiB    20
> GiB   16 TiB  24.34  0.44   60      up          osd.92
>  93    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB   1.3
> GiB   20 TiB   4.63  0.08    9      up          osd.93
>  94    hdd    20.98009   1.00000   21 TiB   1.4 TiB  489 GiB    1 KiB   3.4
> GiB   20 TiB   6.90  0.13   14      up          osd.94
>  95    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB  1016
> MiB   20 TiB   4.63  0.08    4      up          osd.95
>                            TOTAL  1.7 PiB   976 TiB  895 TiB  298 GiB   4.1
> TiB  808 TiB  54.72
>
>
> I tried:
> - update backfill and recovery settings
> - change osd mclock profile
> - change pg/pgp_num
>
> The cluster does not recovery. Clients are working well.
>
> What can I do ?
>
> Thanks for your help
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux