Hosts buckets with no disks don't hurt, but they do influence the crush map.
To avoid unnecessary data movement, you might want to change your server replacement method. Since mimic, instead of purging OSDs, you can destroy OSDs, which is the same as purge but keeps the ID intact. This way, you can decommission your old servers and replace them with new ones, assigning the same host name and adding OSDs with the IDs of previous OSDs on that host back. In other words, you replace a host and OSDs with a new host with new disks, but use the original crush locations. This will reduce the amount of re-mapped objects dramatically.
I guess if you add a new host with a new name and move the old OSD IDs there, you see at least >50% misplaced objects after adding the host. With the above procedure, which preserves the crush map as much as possible, you will see significantly fewer objects misplaced after replacement.
Its more work, but also faster recovery.
If you continue to replace hosts and give them new host names, you should remove the old ones. At some point these buckets might interfere with mappings in unexpected ways.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Matthew Darwin <bugs@xxxxxxxxxx>
Sent: 14 October 2022 19:41:53
To: Frank Schilder; ceph@xxxxxxxxxx; ceph-users@xxxxxxx
Subject: Re: Re: strange OSD status when rebooting one server
The hosts with no OSDs now previously had OSDs, but they no longer do. Those hosts don't exist any more. I guess they could be removed from the crushmap, but I assume hosts with no OSD doesn't hurt anything?
The OSDs that used to be on these servers were deleted, and then new OSDs were created on new servers. The osd.id was automatically reused, so the id of the OSD is not actually reflective of the order the disks were added into the cluster.
Extract of the crushmap:
host ceph-osd13 {
id -5 # do not change unnecessarily
id -6 class hdd # do not change unnecessarily
id -2 class ssd # do not change unnecessarily
# weight 0.00000
alg straw2
hash 0 # rjenkins1
}
host ceph-osd11 {
id -7 # do not change unnecessarily
id -8 class hdd # do not change unnecessarily
id -3 class ssd # do not change unnecessarily
# weight 0.00000
alg straw2
hash 0 # rjenkins1
}
host ceph-osd12 {
id -9 # do not change unnecessarily
id -10 class hdd # do not change unnecessarily
id -11 class ssd # do not change unnecessarily
# weight 0.00000
alg straw2
hash 0 # rjenkins1
}
etc...
On 2022-10-14 13:27, Frank Schilder wrote:
You have hosts in the crush map with no OSDs. They are out+down and will be counted while other hosts are also down. It will go back to normal when you start the host with disks again. If you delete the hosts with no disks, you will probably see misplaced objects. Why are they there in the first place? Are you planning to add hosts or are these replaced ones?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Matthew Darwin <bugs@xxxxxxxxxx><mailto:bugs@xxxxxxxxxx>
Sent: 14 October 2022 18:57:37
To: ceph@xxxxxxxxxx<mailto:ceph@xxxxxxxxxx>; ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
Subject: Re: strange OSD status when rebooting one server
https://gist.githubusercontent.com/matthewdarwin/aec3c2b16ba5e74beb4af1d49e8cfb1a/raw/d8d8f34d989823b9f708608bb2773c7d4093c648/ceph-osd-tree.txt
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 1331.88013 - 1.3 PiB 889 TiB 887 TiB 48 GiB 1.9 TiB 443 TiB 66.76 1.00 - root default
-7 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd11
-9 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd12
-5 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd13
-16 180.80972 - 181 TiB 122 TiB 122 TiB 6.5 GiB 262 GiB 59 TiB 67.55 1.01 - host ceph-osd16
0 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 23 GiB 6.2 TiB 61.91 0.93 70 up osd.0
1 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.2 TiB 68.08 1.02 77 up osd.1
8 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 16 KiB 27 GiB 3.3 TiB 79.58 1.19 90 up osd.8
15 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 24 GiB 5.1 TiB 68.95 1.03 78 up osd.15
17 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.4 TiB 67.14 1.01 76 up osd.17
18 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 0 B 28 GiB 3.2 TiB 80.43 1.20 91 up osd.18
22 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 24 GiB 5.2 TiB 68.05 1.02 77 up osd.22
27 hdd 16.37109 1.00000 16 TiB 8.5 TiB 8.5 TiB 0 B 18 GiB 7.8 TiB 52.10 0.78 59 up osd.27
42 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 23 GiB 6.2 TiB 61.85 0.93 70 up osd.42
46 hdd 16.37109 1.00000 16 TiB 9.6 TiB 9.5 TiB 1 KiB 21 GiB 6.8 TiB 58.35 0.87 66 up osd.46
49 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 28 GiB 3.4 TiB 79.49 1.19 90 up osd.49
53 ssd 0.72769 1.00000 745 GiB 22 GiB 14 GiB 6.5 GiB 1.1 GiB 723 GiB 2.94 0.04 102 up osd.53
-19 164.43863 - 164 TiB 101 TiB 101 TiB 4.3 GiB 219 GiB 64 TiB 61.32 0.92 - host ceph-osd17
2 hdd 16.37109 1.00000 16 TiB 9.5 TiB 9.5 TiB 1 KiB 20 GiB 6.8 TiB 58.26 0.87 66 up osd.2
5 hdd 16.37109 1.00000 16 TiB 8.7 TiB 8.7 TiB 1 KiB 21 GiB 7.7 TiB 53.07 0.79 60 up osd.5
14 hdd 16.37109 0.85004 16 TiB 8.7 TiB 8.7 TiB 0 B 19 GiB 7.7 TiB 53.02 0.79 60 up osd.14
16 hdd 16.37109 1.00000 16 TiB 9.7 TiB 9.7 TiB 0 B 20 GiB 6.7 TiB 59.14 0.89 67 up osd.16
21 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 24 GiB 5.2 TiB 68.06 1.02 77 up osd.21
24 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 2 KiB 26 GiB 3.8 TiB 76.89 1.15 87 up osd.24
28 hdd 16.37109 1.00000 16 TiB 9.7 TiB 9.7 TiB 0 B 21 GiB 6.7 TiB 59.25 0.89 67 up osd.28
34 hdd 16.37109 0.85004 16 TiB 7.7 TiB 7.7 TiB 0 B 16 GiB 8.7 TiB 46.86 0.70 53 up osd.34
44 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 5.7 TiB 65.40 0.98 74 up osd.44
45 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 26 GiB 3.9 TiB 75.90 1.14 86 up osd.45
54 ssd 0.72769 1.00000 745 GiB 14 GiB 9.3 GiB 4.3 GiB 701 MiB 731 GiB 1.92 0.03 100 up osd.54
-22 164.43863 - 164 TiB 108 TiB 108 TiB 6.3 GiB 235 GiB 56 TiB 65.76 0.99 - host ceph-osd18
3 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 26 GiB 3.8 TiB 76.87 1.15 87 up osd.3
19 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 0 B 22 GiB 6.2 TiB 61.89 0.93 70 up osd.19
23 hdd 16.37109 1.00000 16 TiB 9.4 TiB 9.4 TiB 0 B 21 GiB 7.0 TiB 57.45 0.86 65 up osd.23
26 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 24 GiB 5.4 TiB 67.19 1.01 76 up osd.26
29 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 22 GiB 5.7 TiB 65.41 0.98 74 up osd.29
32 hdd 16.37109 1.00000 16 TiB 8.8 TiB 8.8 TiB 17 KiB 19 GiB 7.5 TiB 53.97 0.81 61 up osd.32
35 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 0 B 25 GiB 4.8 TiB 70.72 1.06 80 up osd.35
37 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 24 GiB 5.9 TiB 63.70 0.95 72 up osd.37
47 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 23 GiB 6.4 TiB 60.98 0.91 69 up osd.47
50 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 29 GiB 2.9 TiB 82.21 1.23 93 up osd.50
55 ssd 0.72769 1.00000 745 GiB 20 GiB 13 GiB 6.3 GiB 1.0 GiB 725 GiB 2.70 0.04 116 up osd.55
-13 164.43863 - 164 TiB 110 TiB 110 TiB 3.9 GiB 239 GiB 54 TiB 66.97 1.00 - host ceph-osd19
4 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 22 GiB 6.0 TiB 63.62 0.95 72 up osd.4
20 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 0 B 27 GiB 3.3 TiB 79.55 1.19 90 up osd.20
25 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 22 GiB 5.7 TiB 65.37 0.98 74 up osd.25
30 hdd 16.37109 1.00000 16 TiB 7.8 TiB 7.8 TiB 0 B 17 GiB 8.6 TiB 47.76 0.72 54 up osd.30
31 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 0 B 25 GiB 4.5 TiB 72.45 1.09 82 up osd.31
33 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 0 B 25 GiB 4.5 TiB 72.48 1.09 82 up osd.33
36 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 0 B 25 GiB 4.8 TiB 70.68 1.06 80 up osd.36
38 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 26 GiB 4.7 TiB 71.55 1.07 81 up osd.38
48 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 4.9 TiB 69.88 1.05 79 up osd.48
51 hdd 16.37109 1.00000 16 TiB 9.7 TiB 9.7 TiB 1 KiB 23 GiB 6.7 TiB 59.24 0.89 67 up osd.51
56 ssd 0.72769 1.00000 745 GiB 14 GiB 9.3 GiB 3.9 GiB 1.1 GiB 731 GiB 1.92 0.03 97 up osd.56
-25 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd41
-28 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd42
-31 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd43
-34 164.43863 - 164 TiB 109 TiB 109 TiB 6.7 GiB 234 GiB 55 TiB 66.46 1.00 - host ceph-osd57
6 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 22 GiB 5.5 TiB 66.29 0.99 75 up osd.6
7 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 28 GiB 2.9 TiB 82.24 1.23 93 up osd.7
9 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 24 GiB 5.7 TiB 65.41 0.98 74 up osd.9
12 hdd 16.37109 1.00000 16 TiB 9.4 TiB 9.4 TiB 1 KiB 19 GiB 7.0 TiB 57.45 0.86 65 up osd.12
62 hdd 16.37109 1.00000 16 TiB 9.8 TiB 9.8 TiB 1 KiB 23 GiB 6.5 TiB 60.10 0.90 68 up osd.62
66 hdd 16.37109 1.00000 16 TiB 9.3 TiB 9.2 TiB 1 KiB 20 GiB 7.1 TiB 56.56 0.85 64 up osd.66
70 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 24 GiB 5.4 TiB 67.16 1.01 76 up osd.70
74 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 26 GiB 3.6 TiB 77.80 1.17 88 up osd.74
78 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 22 GiB 6.1 TiB 62.80 0.94 71 up osd.78
82 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 24 GiB 4.7 TiB 71.59 1.07 81 up osd.82
57 ssd 0.72769 1.00000 745 GiB 18 GiB 10 GiB 6.7 GiB 934 MiB 728 GiB 2.36 0.04 84 up osd.57
-37 164.43863 - 164 TiB 114 TiB 114 TiB 2.2 GiB 245 GiB 50 TiB 69.54 1.04 - host ceph-osd58
10 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 5.5 TiB 66.34 0.99 75 up osd.10
13 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.4 TiB 67.24 1.01 76 up osd.13
39 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 25 GiB 4.4 TiB 73.33 1.10 83 up osd.39
40 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 26 GiB 3.6 TiB 77.78 1.17 88 up osd.40
63 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 24 GiB 5.9 TiB 63.68 0.95 72 up osd.63
67 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 5.5 TiB 66.35 0.99 75 up osd.67
71 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 26 GiB 3.9 TiB 76.00 1.14 86 up osd.71
75 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 22 GiB 6.0 TiB 63.60 0.95 72 up osd.75
79 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 26 GiB 3.9 TiB 76.01 1.14 86 up osd.79
83 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.2 TiB 68.07 1.02 77 up osd.83
58 ssd 0.72769 1.00000 745 GiB 14 GiB 11 GiB 2.2 GiB 166 MiB 731 GiB 1.85 0.03 87 up osd.58
-40 164.43863 - 164 TiB 112 TiB 112 TiB 4.5 GiB 240 GiB 52 TiB 68.12 1.02 - host ceph-osd59
11 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 26 GiB 5.1 TiB 68.93 1.03 78 up osd.11
41 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 25 GiB 4.4 TiB 73.33 1.10 83 up osd.41
52 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 25 GiB 4.2 TiB 74.15 1.11 84 up osd.52
64 hdd 16.37109 1.00000 16 TiB 14 TiB 14 TiB 1 KiB 30 GiB 2.5 TiB 84.86 1.27 96 up osd.64
68 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 26 GiB 5.1 TiB 69.01 1.03 78 up osd.68
72 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.8 TiB 64.49 0.97 73 up osd.72
76 hdd 16.37109 1.00000 16 TiB 9.7 TiB 9.7 TiB 1 KiB 21 GiB 6.7 TiB 59.23 0.89 67 up osd.76
80 hdd 16.37109 1.00000 16 TiB 9.7 TiB 9.7 TiB 1 KiB 21 GiB 6.7 TiB 59.23 0.89 67 up osd.80
84 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 21 GiB 6.4 TiB 60.98 0.91 69 up osd.84
87 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 4.9 TiB 69.86 1.05 79 up osd.87
59 ssd 0.72769 1.00000 745 GiB 17 GiB 12 GiB 4.5 GiB 557 MiB 728 GiB 2.28 0.03 106 up osd.59
-43 164.43863 - 164 TiB 112 TiB 112 TiB 13 GiB 235 GiB 52 TiB 68.29 1.02 - host ceph-osd60
43 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 22 GiB 5.4 TiB 67.06 1.00 76 up osd.43
61 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 5.1 TiB 68.95 1.03 78 up osd.61
65 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.5 TiB 66.27 0.99 75 up osd.65
69 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 4.9 TiB 69.85 1.05 79 up osd.69
73 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 24 GiB 4.6 TiB 71.63 1.07 81 up osd.73
77 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 24 GiB 4.9 TiB 69.77 1.05 79 up osd.77
81 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.4 TiB 67.19 1.01 76 up osd.81
85 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.1 TiB 69.00 1.03 78 up osd.85
86 hdd 16.37109 1.00000 16 TiB 14 TiB 14 TiB 1 KiB 29 GiB 2.8 TiB 83.04 1.24 94 up osd.86
88 hdd 16.37109 1.00000 16 TiB 8.7 TiB 8.7 TiB 1 KiB 17 GiB 7.7 TiB 52.98 0.79 60 up osd.88
60 ssd 0.72769 1.00000 745 GiB 27 GiB 13 GiB 13 GiB 1.0 GiB 718 GiB 3.61 0.05 103 up osd.60
TOTAL 1.3 PiB 889 TiB 887 TiB 48 GiB 1.9 TiB 443 TiB 66.76
MIN/MAX VAR: 0.03/1.27 STDDEV: 20.83
On 2022-10-14 12:52, ceph@xxxxxxxxxx<mailto:ceph@xxxxxxxxxx> wrote:
Could you please share output of
Ceph osd df tree
There could be an hint...
Hth
Am 14. Oktober 2022 18:45:40 MESZ schrieb Matthew Darwin
<bugs@xxxxxxxxxx><mailto:bugs@xxxxxxxxxx>:
Hi,
I am hoping someone can help explain this strange message. I took 1 physical server offline which contains 11 OSDs. "ceph -s" reports 11 osd down. Great.
But on the next line it says "4 hosts" are impacted. It should only be 1 single host? When I look the manager dashboard all the OSDs that are down belong to a single host.
Why does it say 4 hosts here?
$ ceph -s
cluster:
id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
health: HEALTH_WARN
11 osds down
4 hosts (11 osds) down
Reduced data availability: 2 pgs inactive, 3 pgs peering
Degraded data redundancy: 44341491/351041478 objects degraded (12.631%), 834 pgs degraded, 782 pgs undersized
2 pgs not deep-scrubbed in time
1 pgs not scrubbed in time
----------------------------------------------------------------------
ceph-users mailing list --ceph-users@xxxxxxx
To unsubscribe send an email toceph-users-leave@xxxxxxx<mailto:toceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
