Re: strange OSD status when rebooting one server

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



The hosts with no OSDs now previously had OSDs, but they no longer do. Those hosts don't exist any more.  I guess they could be removed from the crushmap, but I assume hosts with no OSD doesn't hurt anything?

The OSDs that used to be on these servers were deleted, and then new OSDs were created on new servers.  The osd.id was automatically reused, so the id of the OSD is not actually reflective of the order the disks were added into the cluster.

Extract of the crushmap:

host ceph-osd13 {
       id -5           # do not change unnecessarily
       id -6 class hdd         # do not change unnecessarily
       id -2 class ssd         # do not change unnecessarily
       # weight 0.00000
       alg straw2
       hash 0  # rjenkins1
}
host ceph-osd11 {
       id -7           # do not change unnecessarily
       id -8 class hdd         # do not change unnecessarily
       id -3 class ssd         # do not change unnecessarily
       # weight 0.00000
       alg straw2
       hash 0  # rjenkins1
}
host ceph-osd12 {
       id -9           # do not change unnecessarily
       id -10 class hdd                # do not change unnecessarily
       id -11 class ssd                # do not change unnecessarily
       # weight 0.00000
       alg straw2
       hash 0  # rjenkins1
}

etc...

On 2022-10-14 13:27, Frank Schilder wrote:
You have hosts in the crush map with no OSDs. They are out+down and will be counted while other hosts are also down. It will go back to normal when you start the host with disks again. If you delete the hosts with no disks, you will probably see misplaced objects. Why are they there in the first place? Are you planning to add hosts or are these replaced ones?

Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14

________________________________________
From: Matthew Darwin<bugs@xxxxxxxxxx>
Sent: 14 October 2022 18:57:37
To:ceph@xxxxxxxxxx;ceph-users@xxxxxxx
Subject:  Re: strange OSD status when rebooting one server

https://gist.githubusercontent.com/matthewdarwin/aec3c2b16ba5e74beb4af1d49e8cfb1a/raw/d8d8f34d989823b9f708608bb2773c7d4093c648/ceph-osd-tree.txt

ID   CLASS  WEIGHT      REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
   -1         1331.88013         -  1.3 PiB  889 TiB  887 TiB   48 GiB  1.9 TiB  443 TiB  66.76  1.00    -          root default
   -7                  0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host ceph-osd11
   -9                  0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host ceph-osd12
   -5                  0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host ceph-osd13
-16          180.80972         -  181 TiB  122 TiB  122 TiB  6.5 GiB  262 GiB   59 TiB  67.55  1.01    -              host ceph-osd16
    0    hdd    16.37109   1.00000   16 TiB   10 TiB   10 TiB    1 KiB   23 GiB  6.2 TiB  61.91  0.93   70      up          osd.0
    1    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   23 GiB  5.2 TiB  68.08  1.02   77      up          osd.1
    8    hdd    16.37109   1.00000   16 TiB   13 TiB   13 TiB   16 KiB   27 GiB  3.3 TiB  79.58  1.19   90      up          osd.8
   15    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB      0 B   24 GiB  5.1 TiB  68.95  1.03   78      up          osd.15
   17    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   23 GiB  5.4 TiB  67.14  1.01   76      up          osd.17
   18    hdd    16.37109   1.00000   16 TiB   13 TiB   13 TiB      0 B   28 GiB  3.2 TiB  80.43  1.20   91      up          osd.18
   22    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB      0 B   24 GiB  5.2 TiB  68.05  1.02   77      up          osd.22
   27    hdd    16.37109   1.00000   16 TiB  8.5 TiB  8.5 TiB      0 B   18 GiB  7.8 TiB  52.10  0.78   59      up          osd.27
   42    hdd    16.37109   1.00000   16 TiB   10 TiB   10 TiB    1 KiB   23 GiB  6.2 TiB  61.85  0.93   70      up          osd.42
   46    hdd    16.37109   1.00000   16 TiB  9.6 TiB  9.5 TiB    1 KiB   21 GiB  6.8 TiB  58.35  0.87   66      up          osd.46
   49    hdd    16.37109   1.00000   16 TiB   13 TiB   13 TiB    1 KiB   28 GiB  3.4 TiB  79.49  1.19   90      up          osd.49
   53    ssd     0.72769   1.00000  745 GiB   22 GiB   14 GiB  6.5 GiB  1.1 GiB  723 GiB   2.94  0.04  102      up          osd.53
-19          164.43863         -  164 TiB  101 TiB  101 TiB  4.3 GiB  219 GiB   64 TiB  61.32  0.92    -              host ceph-osd17
    2    hdd    16.37109   1.00000   16 TiB  9.5 TiB  9.5 TiB    1 KiB   20 GiB  6.8 TiB  58.26  0.87   66      up          osd.2
    5    hdd    16.37109   1.00000   16 TiB  8.7 TiB  8.7 TiB    1 KiB   21 GiB  7.7 TiB  53.07  0.79   60      up          osd.5
   14    hdd    16.37109   0.85004   16 TiB  8.7 TiB  8.7 TiB      0 B   19 GiB  7.7 TiB  53.02  0.79   60      up          osd.14
   16    hdd    16.37109   1.00000   16 TiB  9.7 TiB  9.7 TiB      0 B   20 GiB  6.7 TiB  59.14  0.89   67      up          osd.16
   21    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB      0 B   24 GiB  5.2 TiB  68.06  1.02   77      up          osd.21
   24    hdd    16.37109   1.00000   16 TiB   13 TiB   13 TiB    2 KiB   26 GiB  3.8 TiB  76.89  1.15   87      up          osd.24
   28    hdd    16.37109   1.00000   16 TiB  9.7 TiB  9.7 TiB      0 B   21 GiB  6.7 TiB  59.25  0.89   67      up          osd.28
   34    hdd    16.37109   0.85004   16 TiB  7.7 TiB  7.7 TiB      0 B   16 GiB  8.7 TiB  46.86  0.70   53      up          osd.34
   44    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   25 GiB  5.7 TiB  65.40  0.98   74      up          osd.44
   45    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB    1 KiB   26 GiB  3.9 TiB  75.90  1.14   86      up          osd.45
   54    ssd     0.72769   1.00000  745 GiB   14 GiB  9.3 GiB  4.3 GiB  701 MiB  731 GiB   1.92  0.03  100      up          osd.54
-22          164.43863         -  164 TiB  108 TiB  108 TiB  6.3 GiB  235 GiB   56 TiB  65.76  0.99    -              host ceph-osd18
    3    hdd    16.37109   1.00000   16 TiB   13 TiB   13 TiB    1 KiB   26 GiB  3.8 TiB  76.87  1.15   87      up          osd.3
   19    hdd    16.37109   1.00000   16 TiB   10 TiB   10 TiB      0 B   22 GiB  6.2 TiB  61.89  0.93   70      up          osd.19
   23    hdd    16.37109   1.00000   16 TiB  9.4 TiB  9.4 TiB      0 B   21 GiB  7.0 TiB  57.45  0.86   65      up          osd.23
   26    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB      0 B   24 GiB  5.4 TiB  67.19  1.01   76      up          osd.26
   29    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB      0 B   22 GiB  5.7 TiB  65.41  0.98   74      up          osd.29
   32    hdd    16.37109   1.00000   16 TiB  8.8 TiB  8.8 TiB   17 KiB   19 GiB  7.5 TiB  53.97  0.81   61      up          osd.32
   35    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB      0 B   25 GiB  4.8 TiB  70.72  1.06   80      up          osd.35
   37    hdd    16.37109   1.00000   16 TiB   10 TiB   10 TiB    1 KiB   24 GiB  5.9 TiB  63.70  0.95   72      up          osd.37
   47    hdd    16.37109   1.00000   16 TiB   10 TiB   10 TiB    1 KiB   23 GiB  6.4 TiB  60.98  0.91   69      up          osd.47
   50    hdd    16.37109   1.00000   16 TiB   13 TiB   13 TiB    1 KiB   29 GiB  2.9 TiB  82.21  1.23   93      up          osd.50
   55    ssd     0.72769   1.00000  745 GiB   20 GiB   13 GiB  6.3 GiB  1.0 GiB  725 GiB   2.70  0.04  116      up          osd.55
-13          164.43863         -  164 TiB  110 TiB  110 TiB  3.9 GiB  239 GiB   54 TiB  66.97  1.00    -              host ceph-osd19
    4    hdd    16.37109   1.00000   16 TiB   10 TiB   10 TiB    1 KiB   22 GiB  6.0 TiB  63.62  0.95   72      up          osd.4
   20    hdd    16.37109   1.00000   16 TiB   13 TiB   13 TiB      0 B   27 GiB  3.3 TiB  79.55  1.19   90      up          osd.20
   25    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB      0 B   22 GiB  5.7 TiB  65.37  0.98   74      up          osd.25
   30    hdd    16.37109   1.00000   16 TiB  7.8 TiB  7.8 TiB      0 B   17 GiB  8.6 TiB  47.76  0.72   54      up          osd.30
   31    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB      0 B   25 GiB  4.5 TiB  72.45  1.09   82      up          osd.31
   33    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB      0 B   25 GiB  4.5 TiB  72.48  1.09   82      up          osd.33
   36    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB      0 B   25 GiB  4.8 TiB  70.68  1.06   80      up          osd.36
   38    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB    1 KiB   26 GiB  4.7 TiB  71.55  1.07   81      up          osd.38
   48    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   25 GiB  4.9 TiB  69.88  1.05   79      up          osd.48
   51    hdd    16.37109   1.00000   16 TiB  9.7 TiB  9.7 TiB    1 KiB   23 GiB  6.7 TiB  59.24  0.89   67      up          osd.51
   56    ssd     0.72769   1.00000  745 GiB   14 GiB  9.3 GiB  3.9 GiB  1.1 GiB  731 GiB   1.92  0.03   97      up          osd.56
-25                  0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host ceph-osd41
-28                  0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host ceph-osd42
-31                  0         -      0 B      0 B      0 B      0 B      0 B      0 B      0     0    -              host ceph-osd43
-34          164.43863         -  164 TiB  109 TiB  109 TiB  6.7 GiB  234 GiB   55 TiB  66.46  1.00    -              host ceph-osd57
    6    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   22 GiB  5.5 TiB  66.29  0.99   75      up          osd.6
    7    hdd    16.37109   1.00000   16 TiB   13 TiB   13 TiB    1 KiB   28 GiB  2.9 TiB  82.24  1.23   93      up          osd.7
    9    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   24 GiB  5.7 TiB  65.41  0.98   74      up          osd.9
   12    hdd    16.37109   1.00000   16 TiB  9.4 TiB  9.4 TiB    1 KiB   19 GiB  7.0 TiB  57.45  0.86   65      up          osd.12
   62    hdd    16.37109   1.00000   16 TiB  9.8 TiB  9.8 TiB    1 KiB   23 GiB  6.5 TiB  60.10  0.90   68      up          osd.62
   66    hdd    16.37109   1.00000   16 TiB  9.3 TiB  9.2 TiB    1 KiB   20 GiB  7.1 TiB  56.56  0.85   64      up          osd.66
   70    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   24 GiB  5.4 TiB  67.16  1.01   76      up          osd.70
   74    hdd    16.37109   1.00000   16 TiB   13 TiB   13 TiB    1 KiB   26 GiB  3.6 TiB  77.80  1.17   88      up          osd.74
   78    hdd    16.37109   1.00000   16 TiB   10 TiB   10 TiB    1 KiB   22 GiB  6.1 TiB  62.80  0.94   71      up          osd.78
   82    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB    1 KiB   24 GiB  4.7 TiB  71.59  1.07   81      up          osd.82
   57    ssd     0.72769   1.00000  745 GiB   18 GiB   10 GiB  6.7 GiB  934 MiB  728 GiB   2.36  0.04   84      up          osd.57
-37          164.43863         -  164 TiB  114 TiB  114 TiB  2.2 GiB  245 GiB   50 TiB  69.54  1.04    -              host ceph-osd58
   10    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   25 GiB  5.5 TiB  66.34  0.99   75      up          osd.10
   13    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   23 GiB  5.4 TiB  67.24  1.01   76      up          osd.13
   39    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB    1 KiB   25 GiB  4.4 TiB  73.33  1.10   83      up          osd.39
   40    hdd    16.37109   1.00000   16 TiB   13 TiB   13 TiB    1 KiB   26 GiB  3.6 TiB  77.78  1.17   88      up          osd.40
   63    hdd    16.37109   1.00000   16 TiB   10 TiB   10 TiB    1 KiB   24 GiB  5.9 TiB  63.68  0.95   72      up          osd.63
   67    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   25 GiB  5.5 TiB  66.35  0.99   75      up          osd.67
   71    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB    1 KiB   26 GiB  3.9 TiB  76.00  1.14   86      up          osd.71
   75    hdd    16.37109   1.00000   16 TiB   10 TiB   10 TiB    1 KiB   22 GiB  6.0 TiB  63.60  0.95   72      up          osd.75
   79    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB    1 KiB   26 GiB  3.9 TiB  76.01  1.14   86      up          osd.79
   83    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   23 GiB  5.2 TiB  68.07  1.02   77      up          osd.83
   58    ssd     0.72769   1.00000  745 GiB   14 GiB   11 GiB  2.2 GiB  166 MiB  731 GiB   1.85  0.03   87      up          osd.58
-40          164.43863         -  164 TiB  112 TiB  112 TiB  4.5 GiB  240 GiB   52 TiB  68.12  1.02    -              host ceph-osd59
   11    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   26 GiB  5.1 TiB  68.93  1.03   78      up          osd.11
   41    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB    1 KiB   25 GiB  4.4 TiB  73.33  1.10   83      up          osd.41
   52    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB    1 KiB   25 GiB  4.2 TiB  74.15  1.11   84      up          osd.52
   64    hdd    16.37109   1.00000   16 TiB   14 TiB   14 TiB    1 KiB   30 GiB  2.5 TiB  84.86  1.27   96      up          osd.64
   68    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   26 GiB  5.1 TiB  69.01  1.03   78      up          osd.68
   72    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   23 GiB  5.8 TiB  64.49  0.97   73      up          osd.72
   76    hdd    16.37109   1.00000   16 TiB  9.7 TiB  9.7 TiB    1 KiB   21 GiB  6.7 TiB  59.23  0.89   67      up          osd.76
   80    hdd    16.37109   1.00000   16 TiB  9.7 TiB  9.7 TiB    1 KiB   21 GiB  6.7 TiB  59.23  0.89   67      up          osd.80
   84    hdd    16.37109   1.00000   16 TiB   10 TiB   10 TiB    1 KiB   21 GiB  6.4 TiB  60.98  0.91   69      up          osd.84
   87    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   23 GiB  4.9 TiB  69.86  1.05   79      up          osd.87
   59    ssd     0.72769   1.00000  745 GiB   17 GiB   12 GiB  4.5 GiB  557 MiB  728 GiB   2.28  0.03  106      up          osd.59
-43          164.43863         -  164 TiB  112 TiB  112 TiB   13 GiB  235 GiB   52 TiB  68.29  1.02    -              host ceph-osd60
   43    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   22 GiB  5.4 TiB  67.06  1.00   76      up          osd.43
   61    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   25 GiB  5.1 TiB  68.95  1.03   78      up          osd.61
   65    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   23 GiB  5.5 TiB  66.27  0.99   75      up          osd.65
   69    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   25 GiB  4.9 TiB  69.85  1.05   79      up          osd.69
   73    hdd    16.37109   1.00000   16 TiB   12 TiB   12 TiB    1 KiB   24 GiB  4.6 TiB  71.63  1.07   81      up          osd.73
   77    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   24 GiB  4.9 TiB  69.77  1.05   79      up          osd.77
   81    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   23 GiB  5.4 TiB  67.19  1.01   76      up          osd.81
   85    hdd    16.37109   1.00000   16 TiB   11 TiB   11 TiB    1 KiB   23 GiB  5.1 TiB  69.00  1.03   78      up          osd.85
   86    hdd    16.37109   1.00000   16 TiB   14 TiB   14 TiB    1 KiB   29 GiB  2.8 TiB  83.04  1.24   94      up          osd.86
   88    hdd    16.37109   1.00000   16 TiB  8.7 TiB  8.7 TiB    1 KiB   17 GiB  7.7 TiB  52.98  0.79   60      up          osd.88
   60    ssd     0.72769   1.00000  745 GiB   27 GiB   13 GiB   13 GiB  1.0 GiB  718 GiB   3.61  0.05  103      up          osd.60
                             TOTAL  1.3 PiB  889 TiB  887 TiB   48 GiB  1.9 TiB  443 TiB  66.76
MIN/MAX VAR: 0.03/1.27  STDDEV: 20.83

On 2022-10-14 12:52,ceph@xxxxxxxxxx  wrote:
Could you please share output of

Ceph osd df tree

There could be an hint...
Hth

Am 14. Oktober 2022 18:45:40 MESZ schrieb Matthew Darwin
<bugs@xxxxxxxxxx>:

     Hi,

     I am hoping someone can help explain this strange message.  I took 1 physical server offline which contains 11 OSDs.  "ceph -s" reports 11 osd down.  Great.

     But on the next line it says "4 hosts" are impacted.  It should only be 1 single host?  When I look the manager dashboard all the OSDs that are down belong to a single host.

     Why does it say 4 hosts here?

     $ ceph -s

        cluster:
          id:     xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
          health: HEALTH_WARN
                  11 osds down
                  4 hosts (11 osds) down
                  Reduced data availability: 2 pgs inactive, 3 pgs peering
                  Degraded data redundancy: 44341491/351041478 objects degraded (12.631%), 834 pgs degraded, 782 pgs undersized
                  2 pgs not deep-scrubbed in time
                  1 pgs not scrubbed in time
     ----------------------------------------------------------------------
     ceph-users mailing list --ceph-users@xxxxxxx
     To unsubscribe send an emailtoceph-users-leave@xxxxxxx

_______________________________________________
ceph-users mailing list --ceph-users@xxxxxxx
To unsubscribe send an email toceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux