You have hosts in the crush map with no OSDs. They are out+down and will be counted while other hosts are also down. It will go back to normal when you start the host with disks again. If you delete the hosts with no disks, you will probably see misplaced objects. Why are they there in the first place? Are you planning to add hosts or are these replaced ones? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Matthew Darwin <bugs@xxxxxxxxxx> Sent: 14 October 2022 18:57:37 To: ceph@xxxxxxxxxx; ceph-users@xxxxxxx Subject: Re: strange OSD status when rebooting one server https://gist.githubusercontent.com/matthewdarwin/aec3c2b16ba5e74beb4af1d49e8cfb1a/raw/d8d8f34d989823b9f708608bb2773c7d4093c648/ceph-osd-tree.txt ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 1331.88013 - 1.3 PiB 889 TiB 887 TiB 48 GiB 1.9 TiB 443 TiB 66.76 1.00 - root default -7 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd11 -9 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd12 -5 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd13 -16 180.80972 - 181 TiB 122 TiB 122 TiB 6.5 GiB 262 GiB 59 TiB 67.55 1.01 - host ceph-osd16 0 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 23 GiB 6.2 TiB 61.91 0.93 70 up osd.0 1 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.2 TiB 68.08 1.02 77 up osd.1 8 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 16 KiB 27 GiB 3.3 TiB 79.58 1.19 90 up osd.8 15 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 24 GiB 5.1 TiB 68.95 1.03 78 up osd.15 17 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.4 TiB 67.14 1.01 76 up osd.17 18 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 0 B 28 GiB 3.2 TiB 80.43 1.20 91 up osd.18 22 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 24 GiB 5.2 TiB 68.05 1.02 77 up osd.22 27 hdd 16.37109 1.00000 16 TiB 8.5 TiB 8.5 TiB 0 B 18 GiB 7.8 TiB 52.10 0.78 59 up osd.27 42 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 23 GiB 6.2 TiB 61.85 0.93 70 up osd.42 46 hdd 16.37109 1.00000 16 TiB 9.6 TiB 9.5 TiB 1 KiB 21 GiB 6.8 TiB 58.35 0.87 66 up osd.46 49 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 28 GiB 3.4 TiB 79.49 1.19 90 up osd.49 53 ssd 0.72769 1.00000 745 GiB 22 GiB 14 GiB 6.5 GiB 1.1 GiB 723 GiB 2.94 0.04 102 up osd.53 -19 164.43863 - 164 TiB 101 TiB 101 TiB 4.3 GiB 219 GiB 64 TiB 61.32 0.92 - host ceph-osd17 2 hdd 16.37109 1.00000 16 TiB 9.5 TiB 9.5 TiB 1 KiB 20 GiB 6.8 TiB 58.26 0.87 66 up osd.2 5 hdd 16.37109 1.00000 16 TiB 8.7 TiB 8.7 TiB 1 KiB 21 GiB 7.7 TiB 53.07 0.79 60 up osd.5 14 hdd 16.37109 0.85004 16 TiB 8.7 TiB 8.7 TiB 0 B 19 GiB 7.7 TiB 53.02 0.79 60 up osd.14 16 hdd 16.37109 1.00000 16 TiB 9.7 TiB 9.7 TiB 0 B 20 GiB 6.7 TiB 59.14 0.89 67 up osd.16 21 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 24 GiB 5.2 TiB 68.06 1.02 77 up osd.21 24 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 2 KiB 26 GiB 3.8 TiB 76.89 1.15 87 up osd.24 28 hdd 16.37109 1.00000 16 TiB 9.7 TiB 9.7 TiB 0 B 21 GiB 6.7 TiB 59.25 0.89 67 up osd.28 34 hdd 16.37109 0.85004 16 TiB 7.7 TiB 7.7 TiB 0 B 16 GiB 8.7 TiB 46.86 0.70 53 up osd.34 44 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 5.7 TiB 65.40 0.98 74 up osd.44 45 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 26 GiB 3.9 TiB 75.90 1.14 86 up osd.45 54 ssd 0.72769 1.00000 745 GiB 14 GiB 9.3 GiB 4.3 GiB 701 MiB 731 GiB 1.92 0.03 100 up osd.54 -22 164.43863 - 164 TiB 108 TiB 108 TiB 6.3 GiB 235 GiB 56 TiB 65.76 0.99 - host ceph-osd18 3 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 26 GiB 3.8 TiB 76.87 1.15 87 up osd.3 19 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 0 B 22 GiB 6.2 TiB 61.89 0.93 70 up osd.19 23 hdd 16.37109 1.00000 16 TiB 9.4 TiB 9.4 TiB 0 B 21 GiB 7.0 TiB 57.45 0.86 65 up osd.23 26 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 24 GiB 5.4 TiB 67.19 1.01 76 up osd.26 29 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 22 GiB 5.7 TiB 65.41 0.98 74 up osd.29 32 hdd 16.37109 1.00000 16 TiB 8.8 TiB 8.8 TiB 17 KiB 19 GiB 7.5 TiB 53.97 0.81 61 up osd.32 35 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 0 B 25 GiB 4.8 TiB 70.72 1.06 80 up osd.35 37 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 24 GiB 5.9 TiB 63.70 0.95 72 up osd.37 47 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 23 GiB 6.4 TiB 60.98 0.91 69 up osd.47 50 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 29 GiB 2.9 TiB 82.21 1.23 93 up osd.50 55 ssd 0.72769 1.00000 745 GiB 20 GiB 13 GiB 6.3 GiB 1.0 GiB 725 GiB 2.70 0.04 116 up osd.55 -13 164.43863 - 164 TiB 110 TiB 110 TiB 3.9 GiB 239 GiB 54 TiB 66.97 1.00 - host ceph-osd19 4 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 22 GiB 6.0 TiB 63.62 0.95 72 up osd.4 20 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 0 B 27 GiB 3.3 TiB 79.55 1.19 90 up osd.20 25 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 0 B 22 GiB 5.7 TiB 65.37 0.98 74 up osd.25 30 hdd 16.37109 1.00000 16 TiB 7.8 TiB 7.8 TiB 0 B 17 GiB 8.6 TiB 47.76 0.72 54 up osd.30 31 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 0 B 25 GiB 4.5 TiB 72.45 1.09 82 up osd.31 33 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 0 B 25 GiB 4.5 TiB 72.48 1.09 82 up osd.33 36 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 0 B 25 GiB 4.8 TiB 70.68 1.06 80 up osd.36 38 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 26 GiB 4.7 TiB 71.55 1.07 81 up osd.38 48 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 4.9 TiB 69.88 1.05 79 up osd.48 51 hdd 16.37109 1.00000 16 TiB 9.7 TiB 9.7 TiB 1 KiB 23 GiB 6.7 TiB 59.24 0.89 67 up osd.51 56 ssd 0.72769 1.00000 745 GiB 14 GiB 9.3 GiB 3.9 GiB 1.1 GiB 731 GiB 1.92 0.03 97 up osd.56 -25 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd41 -28 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd42 -31 0 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host ceph-osd43 -34 164.43863 - 164 TiB 109 TiB 109 TiB 6.7 GiB 234 GiB 55 TiB 66.46 1.00 - host ceph-osd57 6 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 22 GiB 5.5 TiB 66.29 0.99 75 up osd.6 7 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 28 GiB 2.9 TiB 82.24 1.23 93 up osd.7 9 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 24 GiB 5.7 TiB 65.41 0.98 74 up osd.9 12 hdd 16.37109 1.00000 16 TiB 9.4 TiB 9.4 TiB 1 KiB 19 GiB 7.0 TiB 57.45 0.86 65 up osd.12 62 hdd 16.37109 1.00000 16 TiB 9.8 TiB 9.8 TiB 1 KiB 23 GiB 6.5 TiB 60.10 0.90 68 up osd.62 66 hdd 16.37109 1.00000 16 TiB 9.3 TiB 9.2 TiB 1 KiB 20 GiB 7.1 TiB 56.56 0.85 64 up osd.66 70 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 24 GiB 5.4 TiB 67.16 1.01 76 up osd.70 74 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 26 GiB 3.6 TiB 77.80 1.17 88 up osd.74 78 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 22 GiB 6.1 TiB 62.80 0.94 71 up osd.78 82 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 24 GiB 4.7 TiB 71.59 1.07 81 up osd.82 57 ssd 0.72769 1.00000 745 GiB 18 GiB 10 GiB 6.7 GiB 934 MiB 728 GiB 2.36 0.04 84 up osd.57 -37 164.43863 - 164 TiB 114 TiB 114 TiB 2.2 GiB 245 GiB 50 TiB 69.54 1.04 - host ceph-osd58 10 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 5.5 TiB 66.34 0.99 75 up osd.10 13 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.4 TiB 67.24 1.01 76 up osd.13 39 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 25 GiB 4.4 TiB 73.33 1.10 83 up osd.39 40 hdd 16.37109 1.00000 16 TiB 13 TiB 13 TiB 1 KiB 26 GiB 3.6 TiB 77.78 1.17 88 up osd.40 63 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 24 GiB 5.9 TiB 63.68 0.95 72 up osd.63 67 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 5.5 TiB 66.35 0.99 75 up osd.67 71 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 26 GiB 3.9 TiB 76.00 1.14 86 up osd.71 75 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 22 GiB 6.0 TiB 63.60 0.95 72 up osd.75 79 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 26 GiB 3.9 TiB 76.01 1.14 86 up osd.79 83 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.2 TiB 68.07 1.02 77 up osd.83 58 ssd 0.72769 1.00000 745 GiB 14 GiB 11 GiB 2.2 GiB 166 MiB 731 GiB 1.85 0.03 87 up osd.58 -40 164.43863 - 164 TiB 112 TiB 112 TiB 4.5 GiB 240 GiB 52 TiB 68.12 1.02 - host ceph-osd59 11 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 26 GiB 5.1 TiB 68.93 1.03 78 up osd.11 41 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 25 GiB 4.4 TiB 73.33 1.10 83 up osd.41 52 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 25 GiB 4.2 TiB 74.15 1.11 84 up osd.52 64 hdd 16.37109 1.00000 16 TiB 14 TiB 14 TiB 1 KiB 30 GiB 2.5 TiB 84.86 1.27 96 up osd.64 68 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 26 GiB 5.1 TiB 69.01 1.03 78 up osd.68 72 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.8 TiB 64.49 0.97 73 up osd.72 76 hdd 16.37109 1.00000 16 TiB 9.7 TiB 9.7 TiB 1 KiB 21 GiB 6.7 TiB 59.23 0.89 67 up osd.76 80 hdd 16.37109 1.00000 16 TiB 9.7 TiB 9.7 TiB 1 KiB 21 GiB 6.7 TiB 59.23 0.89 67 up osd.80 84 hdd 16.37109 1.00000 16 TiB 10 TiB 10 TiB 1 KiB 21 GiB 6.4 TiB 60.98 0.91 69 up osd.84 87 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 4.9 TiB 69.86 1.05 79 up osd.87 59 ssd 0.72769 1.00000 745 GiB 17 GiB 12 GiB 4.5 GiB 557 MiB 728 GiB 2.28 0.03 106 up osd.59 -43 164.43863 - 164 TiB 112 TiB 112 TiB 13 GiB 235 GiB 52 TiB 68.29 1.02 - host ceph-osd60 43 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 22 GiB 5.4 TiB 67.06 1.00 76 up osd.43 61 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 5.1 TiB 68.95 1.03 78 up osd.61 65 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.5 TiB 66.27 0.99 75 up osd.65 69 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 25 GiB 4.9 TiB 69.85 1.05 79 up osd.69 73 hdd 16.37109 1.00000 16 TiB 12 TiB 12 TiB 1 KiB 24 GiB 4.6 TiB 71.63 1.07 81 up osd.73 77 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 24 GiB 4.9 TiB 69.77 1.05 79 up osd.77 81 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.4 TiB 67.19 1.01 76 up osd.81 85 hdd 16.37109 1.00000 16 TiB 11 TiB 11 TiB 1 KiB 23 GiB 5.1 TiB 69.00 1.03 78 up osd.85 86 hdd 16.37109 1.00000 16 TiB 14 TiB 14 TiB 1 KiB 29 GiB 2.8 TiB 83.04 1.24 94 up osd.86 88 hdd 16.37109 1.00000 16 TiB 8.7 TiB 8.7 TiB 1 KiB 17 GiB 7.7 TiB 52.98 0.79 60 up osd.88 60 ssd 0.72769 1.00000 745 GiB 27 GiB 13 GiB 13 GiB 1.0 GiB 718 GiB 3.61 0.05 103 up osd.60 TOTAL 1.3 PiB 889 TiB 887 TiB 48 GiB 1.9 TiB 443 TiB 66.76 MIN/MAX VAR: 0.03/1.27 STDDEV: 20.83 On 2022-10-14 12:52, ceph@xxxxxxxxxx wrote: > Could you please share output of > > Ceph osd df tree > > There could be an hint... > Hth > > Am 14. Oktober 2022 18:45:40 MESZ schrieb Matthew Darwin > <bugs@xxxxxxxxxx>: > > Hi, > > I am hoping someone can help explain this strange message. I took 1 physical server offline which contains 11 OSDs. "ceph -s" reports 11 osd down. Great. > > But on the next line it says "4 hosts" are impacted. It should only be 1 single host? When I look the manager dashboard all the OSDs that are down belong to a single host. > > Why does it say 4 hosts here? > > $ ceph -s > > cluster: > id: xxxxxxxxxxxxxxxxxxxxxxxxxxxxx > health: HEALTH_WARN > 11 osds down > 4 hosts (11 osds) down > Reduced data availability: 2 pgs inactive, 3 pgs peering > Degraded data redundancy: 44341491/351041478 objects degraded (12.631%), 834 pgs degraded, 782 pgs undersized > 2 pgs not deep-scrubbed in time > 1 pgs not scrubbed in time > ---------------------------------------------------------------------- > ceph-users mailing list --ceph-users@xxxxxxx > To unsubscribe send an email toceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |