> When you say "don't use ceph adm", can you explain why this is bad?
It means I can't help you. The problems you are in seem to be due to maintenance without telling ceph adm. It also seems that more hosts were moved than your failure domain would allow. Hence the mess.
I don't know if anyone has been in the same situation before. Ceph-adm is a youg (meaning: not very matured) orchestrator. Possible that you will have to figure it out mostly by yourself. A start could be to unmanage the cluster and start to bring services back by hand. I would start with the monitors and then try to get the OSDs to show up. You can start any daemon by hand (after stopping its systemd unit) and see what it complains about during startup. Chances are, its network related.
Good luck!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Jeremy Hansen <farnsworth.mcfadden@xxxxxxxxx>
Sent: 25 July 2022 23:00:27
To: Frank Schilder
Cc: ceph-users@xxxxxxx
Subject: Re: [Warning Possible spam] Re: Issues after a shutdown
I noticed this on the initial run of ceph health, but I no longer see it. When you say "don't use ceph adm", can you explain why this is bad?
This is ceph health outside of cephadm shell:
HEALTH_WARN 1 filesystem is degraded; 2 MDSs report slow metadata IOs; 2/5 mons down, quorum cn02,cn03,cn01; 10 osds down; 3 hosts (17 osds) down; Reduced data ava
ilability: 13 pgs inactive, 9 pgs down; Degraded data redundancy: 8515690/30862245 objects degraded (27.593%), 326 pgs degraded, 447 pgs undersized
[WRN] FS_DEGRADED: 1 filesystem is degraded
fs coldlogix is degraded
[WRN] MDS_SLOW_METADATA_IO: 2 MDSs report slow metadata IOs
mds.coldlogix.cn01.uriofo(mds.0): 2 slow metadata IOs are blocked > 30 secs, oldest blocked for 3701 secs
mds.btc.cn02.ouvaus(mds.0): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 382 secs
[WRN] MON_DOWN: 2/5 mons down, quorum cn02,cn03,cn01
mon.cn05 (rank 0) addr [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0<http://192.168.30.15:3300/0,v1:192.168.30.15:6789/0>] is down (out of quorum)
mon.cn04 (rank 3) addr [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0<http://192.168.30.14:3300/0,v1:192.168.30.14:6789/0>] is down (out of quorum)
[WRN] OSD_DOWN: 10 osds down
osd.0 (root=default,host=cn05) is down
osd.1 (root=default,host=cn06) is down
osd.7 (root=default,host=cn04) is down
osd.13 (root=default,host=cn06) is down
osd.15 (root=default,host=cn05) is down
osd.18 (root=default,host=cn04) is down
osd.20 (root=default,host=cn04) is down
osd.33 (root=default,host=cn06) is down
osd.34 (root=default,host=cn06) is down
osd.36 (root=default,host=cn05) is down
[WRN] OSD_HOST_DOWN: 3 hosts (17 osds) down
host cn04 (root=default) (6 osds) is down
host cn05 (root=default) (5 osds) is down
host cn06 (root=default) (6 osds) is down
[WRN] PG_AVAILABILITY: Reduced data availability: 13 pgs inactive, 9 pgs down
pg 9.3a is down, acting [8]
pg 9.7a is down, acting [8]
pg 9.ba<http://9.ba> is down, acting [8]
pg 9.fa is down, acting [8]
pg 11.3 is stuck inactive for 39h, current state undersized+degraded+peered, last acting [11]
pg 11.11 is down, acting [19,9]
pg 11.1f is stuck inactive for 13h, current state undersized+degraded+peered, last acting [10]
pg 12.36 is down, acting [21,16]
pg 12.59 is down, acting [26,5]
pg 12.66 is down, acting [5]
pg 19.4 is stuck inactive for 39h, current state undersized+degraded+peered, last acting [6]
pg 19.1c is down, acting [21,16,11]
pg 21.1 is stuck inactive for 2m, current state unknown, last acting []
[WRN] PG_DEGRADED: Degraded data redundancy: 8515690/30862245 objects degraded (27.593%), 326 pgs degraded, 447 pgs undersized
pg 9.75 is stuck undersized for 61m, current state active+undersized+remapped, last acting [4,8,35]
pg 9.76 is stuck undersized for 62m, current state active+undersized+degraded, last acting [35,10,21]
pg 9.77 is stuck undersized for 61m, current state active+undersized+remapped, last acting [32,35,4]
pg 9.78 is stuck undersized for 62m, current state active+undersized+degraded, last acting [14,10]
pg 9.79 is stuck undersized for 62m, current state active+undersized+degraded, last acting [21,32]
pg 9.7b is stuck undersized for 61m, current state active+undersized+degraded, last acting [8,12,5]
pg 9.7c is stuck undersized for 61m, current state active+undersized+degraded, last acting [4,35,10]
pg 9.7d is stuck undersized for 62m, current state active+undersized+degraded, last acting [5,19,10]
pg 9.7e is stuck undersized for 62m, current state active+undersized+remapped, last acting [21,10,17]
pg 9.80 is stuck undersized for 61m, current state active+undersized+degraded, last acting [8,4,17]
pg 9.81 is stuck undersized for 62m, current state active+undersized+degraded, last acting [14,26]
pg 9.82 is stuck undersized for 62m, current state active+undersized+degraded, last acting [26,16]
pg 9.83 is stuck undersized for 61m, current state active+undersized+degraded, last acting [8,4]
pg 9.84 is stuck undersized for 61m, current state active+undersized+degraded, last acting [4,35,6]
pg 9.85 is stuck undersized for 61m, current state active+undersized+degraded, last acting [32,12,9]
pg 9.86 is stuck undersized for 61m, current state active+undersized+degraded, last acting [35,5,8]
pg 9.87 is stuck undersized for 61m, current state active+undersized+degraded, last acting [9,12]
pg 9.88 is stuck undersized for 62m, current state active+undersized+remapped, last acting [19,32,35]
pg 9.89 is stuck undersized for 61m, current state active+undersized+degraded, last acting [10,14,4]
pg 9.8a is stuck undersized for 62m, current state active+undersized+degraded, last acting [21,19]
pg 9.8b is stuck undersized for 61m, current state active+undersized+degraded, last acting [8,35]
pg 9.8c is stuck undersized for 58m, current state active+undersized+remapped, last acting [10,19,5]
pg 9.8d is stuck undersized for 61m, current state active+undersized+remapped, last acting [9,6]
pg 9.8f is stuck undersized for 62m, current state active+undersized+remapped, last acting [19,26,17]
pg 9.90 is stuck undersized for 62m, current state active+undersized+remapped, last acting [35,26]
pg 9.91 is stuck undersized for 62m, current state active+undersized+degraded, last acting [17,5]
pg 9.92 is stuck undersized for 62m, current state active+undersized+degraded, last acting [21,26]
pg 9.93 is stuck undersized for 62m, current state active+undersized+remapped, last acting [19,26,5]
pg 9.94 is stuck undersized for 62m, current state active+undersized+degraded, last acting [21,11]
pg 9.95 is stuck undersized for 61m, current state active+undersized+degraded, last acting [8,19]
pg 9.96 is stuck undersized for 62m, current state active+undersized+degraded, last acting [17,6]
pg 9.97 is stuck undersized for 61m, current state active+undersized+degraded, last acting [8,9,16]
pg 9.98 is stuck undersized for 62m, current state active+undersized+degraded, last acting [6,21]
pg 9.99 is stuck undersized for 61m, current state active+undersized+degraded, last acting [10,9]
pg 9.9a is stuck undersized for 61m, current state active+undersized+remapped, last acting [4,16,10]
pg 9.9b is stuck undersized for 61m, current state active+undersized+degraded, last acting [12,4,11]
pg 9.9c is stuck undersized for 61m, current state active+undersized+degraded, last acting [9,16]
pg 9.9d is stuck undersized for 62m, current state active+undersized+degraded, last acting [26,35]
pg 9.9f is stuck undersized for 61m, current state active+undersized+degraded, last acting [9,17,26]
pg 12.70 is stuck undersized for 62m, current state active+undersized+degraded, last acting [21,35]
pg 12.71 is active+undersized+degraded, acting [6,12]
pg 12.72 is stuck undersized for 61m, current state active+undersized+degraded, last acting [10,14,4]
pg 12.73 is stuck undersized for 62m, current state active+undersized+remapped, last acting [5,17,11]
pg 12.78 is stuck undersized for 61m, current state active+undersized+degraded, last acting [5,8,35]
pg 12.79 is stuck undersized for 61m, current state active+undersized+degraded, last acting [4,17]
pg 12.7a is stuck undersized for 62m, current state active+undersized+degraded, last acting [10,21]
pg 12.7b is stuck undersized for 62m, current state active+undersized+remapped, last acting [17,21,11]
pg 12.7c is stuck undersized for 62m, current state active+undersized+degraded, last acting [32,21,16]
pg 12.7d is stuck undersized for 61m, current state active+undersized+degraded, last acting [35,6,9]
pg 12.7e is stuck undersized for 61m, current state active+undersized+degraded, last acting [26,4]
pg 12.7f is stuck undersized for 61m, current state active+undersized+degraded, last acting [9,14]
It's no longer giving me the ssh key issues but hasn't done anything to improve my situation. When the machines came up with a different IP, did this somehow throw off some kind of ssh known hosts file or pub key exchange? It's all very strange why a momentary bad IP could wreak so much havoc.
Thank you
-jeremy
On Mon, Jul 25, 2022 at 1:44 PM Frank Schilder <frans@xxxxxx<mailto:frans@xxxxxx>> wrote:
I don't use ceph-adm and I also don't know how you got the "some more info". However, I did notice that it contains instructions, starting at "Please make sure that the host is reachable ...". How about starting to follow those?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Jeremy Hansen <farnsworth.mcfadden@xxxxxxxxx<mailto:farnsworth.mcfadden@xxxxxxxxx>>
Sent: 25 July 2022 22:32:32
To: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
Subject: [Warning Possible spam] Re: Issues after a shutdown
Here's some more info:
HEALTH_WARN 2 failed cephadm daemon(s); 3 hosts fail cephadm check; 2
filesystems are degraded; 1 MDSs report slow metadata IOs; 2/5 mons down,
quorum cn02,cn03,cn01; 10 osds down; 3 hosts (17 osds) down; Reduced data
availability: 13 pgs inactive, 9 pgs down; Degraded data redundancy:
8515690/30862245 objects degraded (27.593%), 326 pgs degraded, 447 pgs
undersized
[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)
daemon osd.3 on cn01.ceph is in error state
daemon osd.2 on cn01.ceph is in error state
[WRN] CEPHADM_HOST_CHECK_FAILED: 3 hosts fail cephadm check
host cn04.ceph (192.168.30.14) failed check: Failed to connect to
cn04.ceph (192.168.30.14).
Please make sure that the host is reachable and accepts connections using
the cephadm SSH key
To add the cephadm SSH key to the host:
> ceph cephadm get-pub-key > ~/ceph.pub
> ssh-copy-id -f -i ~/ceph.pub root@192.168.30.14<mailto:root@192.168.30.14>
To check that the host is reachable open a new shell with the --no-hosts
flag:
> cephadm shell --no-hosts
Then run the following:
> ceph cephadm get-ssh-config > ssh_config
> ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key
> chmod 0600 ~/cephadm_private_key
> ssh -F ssh_config -i ~/cephadm_private_key root@192.168.30.14<mailto:root@192.168.30.14>
host cn06.ceph (192.168.30.16) failed check: Failed to connect to
cn06.ceph (192.168.30.16).
Please make sure that the host is reachable and accepts connections using
the cephadm SSH key
To add the cephadm SSH key to the host:
> ceph cephadm get-pub-key > ~/ceph.pub
> ssh-copy-id -f -i ~/ceph.pub root@192.168.30.16<mailto:root@192.168.30.16>
To check that the host is reachable open a new shell with the --no-hosts
flag:
> cephadm shell --no-hosts
Then run the following:
> ceph cephadm get-ssh-config > ssh_config
> ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key
> chmod 0600 ~/cephadm_private_key
> ssh -F ssh_config -i ~/cephadm_private_key root@192.168.30.16<mailto:root@192.168.30.16>
host cn05.ceph (192.168.30.15) failed check: Failed to connect to
cn05.ceph (192.168.30.15).
Please make sure that the host is reachable and accepts connections using
the cephadm SSH key
To add the cephadm SSH key to the host:
> ceph cephadm get-pub-key > ~/ceph.pub
> ssh-copy-id -f -i ~/ceph.pub root@192.168.30.15<mailto:root@192.168.30.15>
To check that the host is reachable open a new shell with the --no-hosts
flag:
> cephadm shell --no-hosts
Then run the following:
> ceph cephadm get-ssh-config > ssh_config
> ceph config-key get mgr/cephadm/ssh_identity_key > ~/cephadm_private_key
> chmod 0600 ~/cephadm_private_key
> ssh -F ssh_config -i ~/cephadm_private_key root@192.168.30.15<mailto:root@192.168.30.15>
[WRN] FS_DEGRADED: 2 filesystems are degraded
fs coldlogix is degraded
fs btc is degraded
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
mds.coldlogix.cn01.uriofo(mds.0): 2 slow metadata IOs are blocked > 30
secs, oldest blocked for 2096 secs
[WRN] MON_DOWN: 2/5 mons down, quorum cn02,cn03,cn01
mon.cn05 (rank 0) addr [v2:192.168.30.15:3300/0,v1:192.168.30.15:6789/0<http://192.168.30.15:3300/0,v1:192.168.30.15:6789/0>]
is down (out of quorum)
mon.cn04 (rank 3) addr [v2:192.168.30.14:3300/0,v1:192.168.30.14:6789/0<http://192.168.30.14:3300/0,v1:192.168.30.14:6789/0>]
is down (out of quorum)
[WRN] OSD_DOWN: 10 osds down
osd.0 (root=default,host=cn05) is down
osd.1 (root=default,host=cn06) is down
osd.7 (root=default,host=cn04) is down
osd.13 (root=default,host=cn06) is down
osd.15 (root=default,host=cn05) is down
osd.18 (root=default,host=cn04) is down
osd.20 (root=default,host=cn04) is down
osd.33 (root=default,host=cn06) is down
osd.34 (root=default,host=cn06) is down
osd.36 (root=default,host=cn05) is down
[WRN] OSD_HOST_DOWN: 3 hosts (17 osds) down
host cn04 (root=default) (6 osds) is down
host cn05 (root=default) (5 osds) is down
host cn06 (root=default) (6 osds) is down
[WRN] PG_AVAILABILITY: Reduced data availability: 13 pgs inactive, 9 pgs
down
pg 9.3a is down, acting [8]
pg 9.7a is down, acting [8]
pg 9.ba<http://9.ba> is down, acting [8]
pg 9.fa is down, acting [8]
pg 11.3 is stuck inactive for 39h, current state
undersized+degraded+peered, last acting [11]
pg 11.11 is down, acting [19,9]
pg 11.1f is stuck inactive for 13h, current state
undersized+degraded+peered, last acting [10]
pg 12.36 is down, acting [21,16]
pg 12.59 is down, acting [26,5]
pg 12.66 is down, acting [5]
pg 19.4 is stuck inactive for 39h, current state
undersized+degraded+peered, last acting [6]
pg 19.1c is down, acting [21,16,11]
pg 21.1 is stuck inactive for 36m, current state unknown, last acting []
[WRN] PG_DEGRADED: Degraded data redundancy: 8515690/30862245 objects
degraded (27.593%), 326 pgs degraded, 447 pgs undersized
pg 9.75 is stuck undersized for 34m, current state
active+undersized+remapped, last acting [4,8,35]
pg 9.76 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [35,10,21]
pg 9.77 is stuck undersized for 34m, current state
active+undersized+remapped, last acting [32,35,4]
pg 9.78 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [14,10]
pg 9.79 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [21,32]
pg 9.7b is stuck undersized for 34m, current state
active+undersized+degraded, last acting [8,12,5]
pg 9.7c is stuck undersized for 34m, current state
active+undersized+degraded, last acting [4,35,10]
pg 9.7d is stuck undersized for 35m, current state
active+undersized+degraded, last acting [5,19,10]
pg 9.7e is stuck undersized for 35m, current state
active+undersized+remapped, last acting [21,10,17]
pg 9.80 is stuck undersized for 34m, current state
active+undersized+degraded, last acting [8,4,17]
pg 9.81 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [14,26]
pg 9.82 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [26,16]
pg 9.83 is stuck undersized for 34m, current state
active+undersized+degraded, last acting [8,4]
pg 9.84 is stuck undersized for 34m, current state
active+undersized+degraded, last acting [4,35,6]
pg 9.85 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [32,12,9]
pg 9.86 is stuck undersized for 34m, current state
active+undersized+degraded, last acting [35,5,8]
pg 9.87 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [9,12]
pg 9.88 is stuck undersized for 35m, current state
active+undersized+remapped, last acting [19,32,35]
pg 9.89 is stuck undersized for 34m, current state
active+undersized+degraded, last acting [10,14,4]
pg 9.8a is stuck undersized for 35m, current state
active+undersized+degraded, last acting [21,19]
pg 9.8b is stuck undersized for 34m, current state
active+undersized+degraded, last acting [8,35]
pg 9.8c is stuck undersized for 31m, current state
active+undersized+remapped, last acting [10,19,5]
pg 9.8d is stuck undersized for 35m, current state
active+undersized+remapped, last acting [9,6]
pg 9.8f is stuck undersized for 35m, current state
active+undersized+remapped, last acting [19,26,17]
pg 9.90 is stuck undersized for 35m, current state
active+undersized+remapped, last acting [35,26]
pg 9.91 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [17,5]
pg 9.92 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [21,26]
pg 9.93 is stuck undersized for 35m, current state
active+undersized+remapped, last acting [19,26,5]
pg 9.94 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [21,11]
pg 9.95 is stuck undersized for 34m, current state
active+undersized+degraded, last acting [8,19]
pg 9.96 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [17,6]
pg 9.97 is stuck undersized for 34m, current state
active+undersized+degraded, last acting [8,9,16]
pg 9.98 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [6,21]
pg 9.99 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [10,9]
pg 9.9a is stuck undersized for 34m, current state
active+undersized+remapped, last acting [4,16,10]
pg 9.9b is stuck undersized for 34m, current state
active+undersized+degraded, last acting [12,4,11]
pg 9.9c is stuck undersized for 35m, current state
active+undersized+degraded, last acting [9,16]
pg 9.9d is stuck undersized for 35m, current state
active+undersized+degraded, last acting [26,35]
pg 9.9f is stuck undersized for 35m, current state
active+undersized+degraded, last acting [9,17,26]
pg 12.70 is stuck undersized for 35m, current state
active+undersized+degraded, last acting [21,35]
pg 12.71 is active+undersized+degraded, acting [6,12]
pg 12.72 is stuck undersized for 34m, current state
active+undersized+degraded, last acting [10,14,4]
pg 12.73 is stuck undersized for 35m, current state
active+undersized+remapped, last acting [5,17,11]
pg 12.78 is stuck undersized for 34m, current state
active+undersized+degraded, last acting [5,8,35]
pg 12.79 is stuck undersized for 34m, current state
active+undersized+degraded, last acting [4,17]
pg 12.7a is stuck undersized for 35m, current state
active+undersized+degraded, last acting [10,21]
pg 12.7b is stuck undersized for 35m, current state
active+undersized+remapped, last acting [17,21,11]
pg 12.7c is stuck undersized for 35m, current state
active+undersized+degraded, last acting [32,21,16]
pg 12.7d is stuck undersized for 35m, current state
active+undersized+degraded, last acting [35,6,9]
pg 12.7e is stuck undersized for 34m, current state
active+undersized+degraded, last acting [26,4]
pg 12.7f is stuck undersized for 35m, current state
active+undersized+degraded, last acting [9,14]
On Mon, Jul 25, 2022 at 12:43 PM Jeremy Hansen <
farnsworth.mcfadden@xxxxxxxxx<mailto:farnsworth.mcfadden@xxxxxxxxx>> wrote:
> Pretty desperate here. Can someone suggest what I might be able to do to
> get these OSDs back up. It looks like my recovery had stalled.
>
>
> On Mon, Jul 25, 2022 at 7:26 AM Anthony D'Atri <anthony.datri@xxxxxxxxx<mailto:anthony.datri@xxxxxxxxx>>
> wrote:
>
>> Do your values for public and cluster network include the new addresses
>> on all nodes?
>>
>
> This cluster only has one network. There is no separation between
> public and cluster. Three of the nodes momentarily came up using a
> different IP address.
>
> I've also noticed on one of the nodes that did not move or have any IP
> issue, on a single node, from the dashboard, it names the same device for
> two different osd's:
>
> 2 cn01 out destroyed hdd TOSHIBA_MG04SCA40EE_21M0A0CKFWZB Unknown sdb osd.2
>
> 3 cn01 out destroyed ssd SAMSUNG_MZILT3T8HBLS/007_S5G0NE0R200159 Unknown
> sdb osd.3
>
>
> [ceph: root@cn01 /]# ceph-volume inventory
>
> Device Path Size rotates available Model name
> /dev/sda 3.64 TB True False MG04SCA40EE
> /dev/sdb 3.49 TB False False MZILT3T8HBLS/007
> /dev/sdc 3.64 TB True False MG04SCA40EE
> /dev/sdd 3.64 TB True False MG04SCA40EE
> /dev/sde 3.49 TB False False MZILT3T8HBLS/007
> /dev/sdf 3.64 TB True False MG04SCA40EE
> /dev/sdg 698.64 GB True False SEAGATE ST375064
>
> [ceph: root@cn01 /]# ceph osd info
> osd.0 down out weight 0 up_from 30231 up_thru 30564 down_at 30688
> last_clean_interval [25500,30228) [v2:
> 192.168.30.15:6818/2512683421,v1:192.168.30.15:6819/2512683421<http://192.168.30.15:6818/2512683421,v1:192.168.30.15:6819/2512683421>] [v2:
> 192.168.30.15:6824/2512683421,v1:192.168.30.15:6826/2512683421<http://192.168.30.15:6824/2512683421,v1:192.168.30.15:6826/2512683421>]
> autoout,exists d14cf503-a303-4fa4-a713-9530b67d613a
> osd.1 down out weight 0 up_from 30393 up_thru 30688 down_at 30697
> last_clean_interval [25518,30321) [v2:
> 192.168.30.16:6834/1781855831,v1:192.168.30.16:6835/1781855831<http://192.168.30.16:6834/1781855831,v1:192.168.30.16:6835/1781855831>] [v2:
> 192.168.30.16:6836/1781855831,v1:192.168.30.16:6837/1781855831<http://192.168.30.16:6836/1781855831,v1:192.168.30.16:6837/1781855831>]
> autoout,exists 0d521411-c835-4fa3-beca-3631b4ff6bf7
> osd.2 down out weight 0 up_from 31316 up_thru 31293 down_at 31317
> last_clean_interval [31218,31296) [v2:
> 192.168.30.11:6810/894589880,v1:192.168.30.11:6811/894589880<http://192.168.30.11:6810/894589880,v1:192.168.30.11:6811/894589880>] [v2:
> 192.168.30.11:6812/894589880,v1:192.168.30.11:6813/894589880<http://192.168.30.11:6812/894589880,v1:192.168.30.11:6813/894589880>]
> destroyed,exists
> osd.3 down out weight 0 up_from 31265 up_thru 31266 down_at 31268
> last_clean_interval [31254,31256) [v2:
> 192.168.30.11:6818/1641948535,v1:192.168.30.11:6819/1641948535<http://192.168.30.11:6818/1641948535,v1:192.168.30.11:6819/1641948535>] [v2:
> 192.168.30.11:6820/1641948535,v1:192.168.30.11:6821/1641948535<http://192.168.30.11:6820/1641948535,v1:192.168.30.11:6821/1641948535>]
> destroyed,exists
> osd.4 up in weight 1 up_from 31356 up_thru 31581 down_at 31339
> last_clean_interval [31320,31338) [v2:
> 192.168.30.11:6802/2785067179,v1:192.168.30.11:6803/2785067179<http://192.168.30.11:6802/2785067179,v1:192.168.30.11:6803/2785067179>] [v2:
> 192.168.30.11:6804/2785067179,v1:192.168.30.11:6805/2785067179<http://192.168.30.11:6804/2785067179,v1:192.168.30.11:6805/2785067179>] exists,up
> 3afd06db-b91d-44fe-9305-5eb95f7a59b9
> osd.5 up in weight 1 up_from 31347 up_thru 31699 down_at 31339
> last_clean_interval [31311,31338) [v2:
> 192.168.30.11:6818/1936771540,v1:192.168.30.11:6819/1936771540<http://192.168.30.11:6818/1936771540,v1:192.168.30.11:6819/1936771540>] [v2:
> 192.168.30.11:6820/1936771540,v1:192.168.30.11:6821/1936771540<http://192.168.30.11:6820/1936771540,v1:192.168.30.11:6821/1936771540>] exists,up
> 063c2ccf-02ce-4f5e-8252-dddfbb258a95
> osd.6 up in weight 1 up_from 31218 up_thru 31711 down_at 31217
> last_clean_interval [30978,31195) [v2:
> 192.168.30.12:6816/1585973160,v1:192.168.30.12:6817/1585973160<http://192.168.30.12:6816/1585973160,v1:192.168.30.12:6817/1585973160>] [v2:
> 192.168.30.12:6818/1585973160,v1:192.168.30.12:6819/1585973160<http://192.168.30.12:6818/1585973160,v1:192.168.30.12:6819/1585973160>] exists,up
> 94250ea2-f12e-4dc6-9135-b626086ccffd
> osd.7 down out weight 0 up_from 30353 up_thru 30558 down_at 30688
> last_clean_interval [25533,30349) [v2:
> 192.168.30.14:6816/4083104061,v1:192.168.30.14:6817/4083104061<http://192.168.30.14:6816/4083104061,v1:192.168.30.14:6817/4083104061>] [v2:
> 192.168.30.14:6840/4094104061,v1:192.168.30.14:6841/4094104061<http://192.168.30.14:6840/4094104061,v1:192.168.30.14:6841/4094104061>]
> autoout,exists de351aec-b91e-4c22-a0bf-85369bc14579
> osd.8 up in weight 1 up_from 31226 up_thru 31668 down_at 31225
> last_clean_interval [30983,31195) [v2:
> 192.168.30.12:6824/1312484329,v1:192.168.30.12:6825/1312484329<http://192.168.30.12:6824/1312484329,v1:192.168.30.12:6825/1312484329>] [v2:
> 192.168.30.12:6826/1312484329,v1:192.168.30.12:6827/1312484329<http://192.168.30.12:6826/1312484329,v1:192.168.30.12:6827/1312484329>] exists,up
> 51f665b4-fa5b-4b17-8390-ed130145ef04
> osd.9 up in weight 1 up_from 31351 up_thru 31673 down_at 31340
> last_clean_interval [31315,31338) [v2:
> 192.168.30.11:6810/1446838877,v1:192.168.30.11:6811/1446838877<http://192.168.30.11:6810/1446838877,v1:192.168.30.11:6811/1446838877>] [v2:
> 192.168.30.11:6812/1446838877,v1:192.168.30.11:6813/1446838877<http://192.168.30.11:6812/1446838877,v1:192.168.30.11:6813/1446838877>] exists,up
> 985f1127-d126-4629-b8cd-03cf2d914d99
> osd.10 up in weight 1 up_from 31219 up_thru 31639 down_at 31218
> last_clean_interval [30980,31195) [v2:
> 192.168.30.12:6808/1587842953,v1:192.168.30.12:6809/1587842953<http://192.168.30.12:6808/1587842953,v1:192.168.30.12:6809/1587842953>] [v2:
> 192.168.30.12:6810/1587842953,v1:192.168.30.12:6811/1587842953<http://192.168.30.12:6810/1587842953,v1:192.168.30.12:6811/1587842953>] exists,up
> c7fca03e-4bd5-4485-a090-658ca967d5f6
> osd.11 up in weight 1 up_from 31234 up_thru 31659 down_at 31223
> last_clean_interval [30978,31195) [v2:
> 192.168.30.12:6840/3403200742,v1:192.168.30.12:6841/3403200742<http://192.168.30.12:6840/3403200742,v1:192.168.30.12:6841/3403200742>] [v2:
> 192.168.30.12:6842/3403200742,v1:192.168.30.12:6843/3403200742<http://192.168.30.12:6842/3403200742,v1:192.168.30.12:6843/3403200742>] exists,up
> 81074bd7-ad9f-4e56-8885-cca4745f6c95
> osd.12 up in weight 1 up_from 31230 up_thru 31717 down_at 31223
> last_clean_interval [30975,31195) [v2:
> 192.168.30.13:6816/4268732910,v1:192.168.30.13:6817/4268732910<http://192.168.30.13:6816/4268732910,v1:192.168.30.13:6817/4268732910>] [v2:
> 192.168.30.13:6818/4268732910,v1:192.168.30.13:6819/4268732910<http://192.168.30.13:6818/4268732910,v1:192.168.30.13:6819/4268732910>] exists,up
> af1b55dd-c110-4861-aed9-c0737cef8be1
> osd.13 down out weight 0 up_from 30389 up_thru 30688 down_at 30695
> last_clean_interval [25513,30317) [v2:
> 192.168.30.16:6804/1573803727,v1:192.168.30.16:6805/1573803727<http://192.168.30.16:6804/1573803727,v1:192.168.30.16:6805/1573803727>] [v2:
> 192.168.30.16:6806/1573803727,v1:192.168.30.16:6807/1573803727<http://192.168.30.16:6806/1573803727,v1:192.168.30.16:6807/1573803727>]
> autoout,exists 737a3234-0f1f-4286-80e9-e89b581aae41
> osd.14 up in weight 1 up_from 31218 up_thru 31709 down_at 31217
> last_clean_interval [30979,31195) [v2:
> 192.168.30.13:6834/2291187817,v1:192.168.30.13:6835/2291187817<http://192.168.30.13:6834/2291187817,v1:192.168.30.13:6835/2291187817>] [v2:
> 192.168.30.13:6836/2291187817,v1:192.168.30.13:6837/2291187817<http://192.168.30.13:6836/2291187817,v1:192.168.30.13:6837/2291187817>] exists,up
> 97cd6ac7-aca0-42fd-a049-d27289f83183
> osd.15 down out weight 0 up_from 29463 up_thru 30531 down_at 30688
> last_clean_interval [25493,29462) [v2:
> 192.168.30.15:6808/2655269991,v1:192.168.30.15:6809/2655269991<http://192.168.30.15:6808/2655269991,v1:192.168.30.15:6809/2655269991>] [v2:
> 192.168.30.15:6802/2662269991,v1:192.168.30.15:6803/2662269991<http://192.168.30.15:6802/2662269991,v1:192.168.30.15:6803/2662269991>]
> autoout,exists 61aea8f4-5905-4be3-ae32-5eacf75a514e
> osd.16 up in weight 1 up_from 31226 up_thru 31647 down_at 31223
> last_clean_interval [30970,31195) [v2:
> 192.168.30.13:6808/2624812081,v1:192.168.30.13:6809/2624812081<http://192.168.30.13:6808/2624812081,v1:192.168.30.13:6809/2624812081>] [v2:
> 192.168.30.13:6810/2624812081,v1:192.168.30.13:6811/2624812081<http://192.168.30.13:6810/2624812081,v1:192.168.30.13:6811/2624812081>] exists,up
> 791a7542-87cd-403d-a37e-8f00506b2eb6
> osd.17 up in weight 1 up_from 31219 up_thru 31703 down_at 31218
> last_clean_interval [30975,31195) [v2:
> 192.168.30.13:6800/2978036397,v1:192.168.30.13:6801/2978036397<http://192.168.30.13:6800/2978036397,v1:192.168.30.13:6801/2978036397>] [v2:
> 192.168.30.13:6802/2978036397,v1:192.168.30.13:6803/2978036397<http://192.168.30.13:6802/2978036397,v1:192.168.30.13:6803/2978036397>] exists,up
> 4a915645-412f-49e6-8477-1577469905da
> osd.18 down out weight 0 up_from 30334 up_thru 30566 down_at 30688
> last_clean_interval [25543,30327) [v2:
> 192.168.30.14:6832/985432137,v1:192.168.30.14:6833/985432137<http://192.168.30.14:6832/985432137,v1:192.168.30.14:6833/985432137>] [v2:
> 192.168.30.14:6848/998432137,v1:192.168.30.14:6849/998432137<http://192.168.30.14:6848/998432137,v1:192.168.30.14:6849/998432137>]
> autoout,exists 85f59d83-710c-4896-9200-bda4894fc3e8
> osd.19 down in weight 1 up_from 31303 up_thru 31321 down_at 31323
> last_clean_interval [31292,31296) [v2:
> 192.168.30.13:6826/375623427,v1:192.168.30.13:6827/375623427<http://192.168.30.13:6826/375623427,v1:192.168.30.13:6827/375623427>] [v2:
> 192.168.30.13:6828/375623427,v1:192.168.30.13:6829/375623427<http://192.168.30.13:6828/375623427,v1:192.168.30.13:6829/375623427>] exists
> 7d09b51a-bd6d-40f8-a009-78ab9937795d
> osd.20 down out weight 0 up_from 30445 up_thru 30531 down_at 30688
> last_clean_interval [28947,30444) [v2:
> 192.168.30.14:6810/4062649162,v1:192.168.30.14:6811/4062649162<http://192.168.30.14:6810/4062649162,v1:192.168.30.14:6811/4062649162>] [v2:
> 192.168.30.14:6800/4073649162,v1:192.168.30.14:6801/4073649162<http://192.168.30.14:6800/4073649162,v1:192.168.30.14:6801/4073649162>]
> autoout,exists 7ef6cc1a-4755-4a14-b9df-f1f538d903ce
> osd.21 up in weight 1 up_from 31345 up_thru 31567 down_at 31341
> last_clean_interval [31307,31340) [v2:
> 192.168.30.11:6826/1625231298,v1:192.168.30.11:6827/1625231298<http://192.168.30.11:6826/1625231298,v1:192.168.30.11:6827/1625231298>] [v2:
> 192.168.30.11:6828/1625231298,v1:192.168.30.11:6829/1625231298<http://192.168.30.11:6828/1625231298,v1:192.168.30.11:6829/1625231298>] exists,up
> 5ef2e39a-a353-4cb8-a49e-093fe39b94ef
> osd.22 down in weight 1 up_from 30383 up_thru 30528 down_at 30688
> last_clean_interval [25549,30317) [v2:
> 192.168.30.14:6806/1204256629,v1:192.168.30.14:6807/1204256629<http://192.168.30.14:6806/1204256629,v1:192.168.30.14:6807/1204256629>] [v2:
> 192.168.30.14:6812/1204256629,v1:192.168.30.14:6813/1204256629<http://192.168.30.14:6812/1204256629,v1:192.168.30.14:6813/1204256629>] exists
> c9befe11-a035-449c-8d17-42aaf191923d
> osd.23 down in weight 1 up_from 30334 up_thru 30576 down_at 30688
> last_clean_interval [30263,30332) [v2:
> 192.168.30.14:6802/3837786490,v1:192.168.30.14:6803/3837786490<http://192.168.30.14:6802/3837786490,v1:192.168.30.14:6803/3837786490>] [v2:
> 192.168.30.14:6830/3838786490,v1:192.168.30.14:6831/3838786490<http://192.168.30.14:6830/3838786490,v1:192.168.30.14:6831/3838786490>] exists
> 2081147b-065d-4da7-89d9-747e1ae02b8d
> osd.24 down in weight 1 up_from 29455 up_thru 30570 down_at 30688
> last_clean_interval [25487,29453) [v2:
> 192.168.30.15:6800/2008474583,v1:192.168.30.15:6801/2008474583<http://192.168.30.15:6800/2008474583,v1:192.168.30.15:6801/2008474583>] [v2:
> 192.168.30.15:6810/2016474583,v1:192.168.30.15:6811/2016474583<http://192.168.30.15:6810/2016474583,v1:192.168.30.15:6811/2016474583>] exists
> 39d78380-261c-4689-b53d-90713e6ffcca
> osd.26 up in weight 1 up_from 31208 up_thru 31643 down_at 31207
> last_clean_interval [30967,31195) [v2:
> 192.168.30.12:6800/2861018947,v1:192.168.30.12:6801/2861018947<http://192.168.30.12:6800/2861018947,v1:192.168.30.12:6801/2861018947>] [v2:
> 192.168.30.12:6802/2861018947,v1:192.168.30.12:6803/2861018947<http://192.168.30.12:6802/2861018947,v1:192.168.30.12:6803/2861018947>] exists,up
> 046622c8-c09c-4254-8c15-3dc05a2f01ed
> osd.28 down in weight 1 up_from 30389 up_thru 30574 down_at 30691
> last_clean_interval [25513,30312) [v2:
> 192.168.30.16:6820/3466284570,v1:192.168.30.16:6821/3466284570<http://192.168.30.16:6820/3466284570,v1:192.168.30.16:6821/3466284570>] [v2:
> 192.168.30.16:6822/3466284570,v1:192.168.30.16:6823/3466284570<http://192.168.30.16:6822/3466284570,v1:192.168.30.16:6823/3466284570>] exists
> 10578b97-e3c4-4553-a8d0-6dcc46af5db1
> osd.29 down in weight 1 up_from 30378 up_thru 30554 down_at 30688
> last_clean_interval [28595,30376) [v2:
> 192.168.30.14:6808/3739543672,v1:192.168.30.14:6809/3739543672<http://192.168.30.14:6808/3739543672,v1:192.168.30.14:6809/3739543672>] [v2:
> 192.168.30.14:6846/3747543672,v1:192.168.30.14:6847/3747543672<http://192.168.30.14:6846/3747543672,v1:192.168.30.14:6847/3747543672>] exists
> 9698e936-8edf-4adf-92c9-a0b5202ed01a
> osd.30 down in weight 1 up_from 30449 up_thru 30531 down_at 30688
> last_clean_interval [25502,30446) [v2:
> 192.168.30.15:6825/2375507296,v1:192.168.30.15:6827/2375507296<http://192.168.30.15:6825/2375507296,v1:192.168.30.15:6827/2375507296>] [v2:
> 192.168.30.15:6829/2375507296,v1:192.168.30.15:6831/2375507296<http://192.168.30.15:6829/2375507296,v1:192.168.30.15:6831/2375507296>] exists
> e14d2a0f-a98a-44d4-8c06-4d893f673629
> osd.31 down in weight 1 up_from 30364 up_thru 30688 down_at 30700
> last_clean_interval [25514,30361) [v2:
> 192.168.30.16:6826/2835000708,v1:192.168.30.16:6827/2835000708<http://192.168.30.16:6826/2835000708,v1:192.168.30.16:6827/2835000708>] [v2:
> 192.168.30.16:6802/2843000708,v1:192.168.30.16:6803/2843000708<http://192.168.30.16:6802/2843000708,v1:192.168.30.16:6803/2843000708>] exists
> 541bca38-e704-483a-8cb8-39e5f69007d1
> osd.32 up in weight 1 up_from 31209 up_thru 31627 down_at 31208
> last_clean_interval [30974,31195) [v2:
> 192.168.30.12:6832/3860067997,v1:192.168.30.12:6833/3860067997<http://192.168.30.12:6832/3860067997,v1:192.168.30.12:6833/3860067997>] [v2:
> 192.168.30.12:6834/3860067997,v1:192.168.30.12:6835/3860067997<http://192.168.30.12:6834/3860067997,v1:192.168.30.12:6835/3860067997>] exists,up
> 9200a57e-2845-43ff-9787-8f1f3158fe90
> osd.33 down in weight 1 up_from 30354 up_thru 30688 down_at 30693
> last_clean_interval [25521,30350) [v2:
> 192.168.30.16:6842/2342555666,v1:192.168.30.16:6843/2342555666<http://192.168.30.16:6842/2342555666,v1:192.168.30.16:6843/2342555666>] [v2:
> 192.168.30.16:6844/2364555666,v1:192.168.30.16:6845/2364555666<http://192.168.30.16:6844/2364555666,v1:192.168.30.16:6845/2364555666>] exists
> 20c55d85-cf9a-4133-a189-7fdad2318f58
> osd.34 down in weight 1 up_from 30390 up_thru 30688 down_at 30691
> last_clean_interval [25516,30314) [v2:
> 192.168.30.16:6808/2282629870,v1:192.168.30.16:6811/2282629870<http://192.168.30.16:6808/2282629870,v1:192.168.30.16:6811/2282629870>] [v2:
> 192.168.30.16:6812/2282629870,v1:192.168.30.16:6814/2282629870<http://192.168.30.16:6812/2282629870,v1:192.168.30.16:6814/2282629870>] exists
> 77e0ef8f-c047-4f84-afb2-a8ad054e562f
> osd.35 up in weight 1 up_from 31204 up_thru 31657 down_at 31203
> last_clean_interval [30958,31195) [v2:
> 192.168.30.13:6842/1919357520,v1:192.168.30.13:6843/1919357520<http://192.168.30.13:6842/1919357520,v1:192.168.30.13:6843/1919357520>] [v2:
> 192.168.30.13:6844/1919357520,v1:192.168.30.13:6845/1919357520<http://192.168.30.13:6844/1919357520,v1:192.168.30.13:6845/1919357520>] exists,up
> 2d2de0cb-6d41-4957-a473-2bbe9ce227bf
> osd.36 down in weight 1 up_from 29494 up_thru 30560 down_at 30688
> last_clean_interval [25491,29492) [v2:
> 192.168.30.15:6816/2153321591,v1:192.168.30.15:6817/2153321591<http://192.168.30.15:6816/2153321591,v1:192.168.30.15:6817/2153321591>] [v2:
> 192.168.30.15:6842/2158321591,v1:192.168.30.15:6843/2158321591<http://192.168.30.15:6842/2158321591,v1:192.168.30.15:6843/2158321591>] exists
> 26114668-68b2-458b-89c2-cbad5507ab75
>
>
>
>>
>> > On Jul 25, 2022, at 3:29 AM, Jeremy Hansen <
>> farnsworth.mcfadden@xxxxxxxxx<mailto:farnsworth.mcfadden@xxxxxxxxx>> wrote:
>> >
>> > I transitioned some servers to a new rack and now I'm having major
>> issues
>> > with Ceph upon bringing things back up.
>> >
>> > I believe the issue may be related to the ceph nodes coming back up with
>> > different IPs before VLANs were set. That's just a guess because I
>> can't
>> > think of any other reason this would happen.
>> >
>> > Current state:
>> >
>> > Every 2.0s: ceph -s
>> > cn01.ceph.la1.clx.corp: Mon Jul 25 10:13:05 2022
>> >
>> > cluster:
>> > id: bfa2ad58-c049-11eb-9098-3c8cf8ed728d
>> > health: HEALTH_WARN
>> > 1 filesystem is degraded
>> > 2 MDSs report slow metadata IOs
>> > 2/5 mons down, quorum cn02,cn03,cn01
>> > 9 osds down
>> > 3 hosts (17 osds) down
>> > Reduced data availability: 97 pgs inactive, 9 pgs down
>> > Degraded data redundancy: 13860144/30824413 objects degraded
>> > (44.965%), 411 pgs degraded, 482 pgs undersized
>> >
>> > services:
>> > mon: 5 daemons, quorum cn02,cn03,cn01 (age 62m), out of quorum: cn05,
>> > cn04
>> > mgr: cn02.arszct(active, since 5m)
>> > mds: 2/2 daemons up, 2 standby
>> > osd: 35 osds: 15 up (since 62m), 24 in (since 58m); 222 remapped pgs
>> >
>> > data:
>> > volumes: 1/2 healthy, 1 recovering
>> > pools: 8 pools, 545 pgs
>> > objects: 7.71M objects, 6.7 TiB
>> > usage: 15 TiB used, 39 TiB / 54 TiB avail
>> > pgs: 0.367% pgs unknown
>> > 17.431% pgs not active
>> > 13860144/30824413 objects degraded (44.965%)
>> > 1137693/30824413 objects misplaced (3.691%)
>> > 280 active+undersized+degraded
>> > 67 undersized+degraded+remapped+backfilling+peered
>> > 57 active+undersized+remapped
>> > 45 active+clean+remapped
>> > 44 active+undersized+degraded+remapped+backfilling
>> > 18 undersized+degraded+peered
>> > 10 active+undersized
>> > 9 down
>> > 7 active+clean
>> > 3 active+undersized+remapped+backfilling
>> > 2 active+undersized+degraded+remapped+backfill_wait
>> > 2 unknown
>> > 1 undersized+peered
>> >
>> > io:
>> > client: 170 B/s rd, 0 op/s rd, 0 op/s wr
>> > recovery: 168 MiB/s, 158 keys/s, 166 objects/s
>> >
>> > I have to disable and re-enable the dashboard just to use it. It seems
>> to
>> > get bogged down after a few moments.
>> >
>> > The three servers that were moved to the new rack Ceph has marked as
>> > "Down", but if I do a cephadm host-check, they all seem to pass:
>> >
>> > ************************ ceph ************************
>> > --------- cn01.ceph.---------
>> > podman (/usr/bin/podman) version 4.0.2 is present
>> > systemctl is present
>> > lvcreate is present
>> > Unit chronyd.service is enabled and running
>> > Host looks OK
>> > --------- cn02.ceph.---------
>> > podman (/usr/bin/podman) version 4.0.2 is present
>> > systemctl is present
>> > lvcreate is present
>> > Unit chronyd.service is enabled and running
>> > Host looks OK
>> > --------- cn03.ceph.---------
>> > podman (/usr/bin/podman) version 4.0.2 is present
>> > systemctl is present
>> > lvcreate is present
>> > Unit chronyd.service is enabled and running
>> > Host looks OK
>> > --------- cn04.ceph.---------
>> > podman (/usr/bin/podman) version 4.0.2 is present
>> > systemctl is present
>> > lvcreate is present
>> > Unit chronyd.service is enabled and running
>> > Host looks OK
>> > --------- cn05.ceph.---------
>> > podman|docker (/usr/bin/podman) is present
>> > systemctl is present
>> > lvcreate is present
>> > Unit chronyd.service is enabled and running
>> > Host looks OK
>> > --------- cn06.ceph.---------
>> > podman (/usr/bin/podman) version 4.0.2 is present
>> > systemctl is present
>> > lvcreate is present
>> > Unit chronyd.service is enabled and running
>> > Host looks OK
>> >
>> > It seems to be recovering with what it has left, but a large amount of
>> OSDs
>> > are down. When trying to restart one of the down'd OSDs, I see a huge
>> dump.
>> >
>> > Jul 25 03:19:38 cn06.ceph
>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > 2022-07-25T10:19:38.532+0000 7fce14a6c080 0 osd.34 30689 done with
>> init,
>> > starting boot process
>> > Jul 25 03:19:38 cn06.ceph
>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > 2022-07-25T10:19:38.532+0000 7fce14a6c080 1 osd.34 30689 start_boot
>> > Jul 25 03:20:10 cn06.ceph
>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > 2022-07-25T10:20:10.655+0000 7fcdfd12d700 1 osd.34 30689 start_boot
>> > Jul 25 03:20:41 cn06.ceph
>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > 2022-07-25T10:20:41.159+0000 7fcdfd12d700 1 osd.34 30689 start_boot
>> > Jul 25 03:21:11 cn06.ceph
>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>> > 2022-07-25T10:21:11.662+0000 7fcdfd12d700 1 osd.34 30689 start_boot
>> >
>> > At this point it just keeps printing start_boot, but the dashboard has
>> it
>> > marked as "in" but "down".
>> >
>> > On these three hosts that moved, there were a bunch marked as "out" and
>> > "down", and some with "in" but "down".
>> >
>> > Not sure where to go next. I'm going to let the recovery continue and
>> hope
>> > that my 4x replication on these pools saves me.
>> >
>> > Not sure where to go from here. Any help is very much appreciated.
>> This
>> > Ceph cluster holds all of our Cloudstack images... it would be
>> terrible to
>> > lose this data.
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx>
>>
>
> On Mon, Jul 25, 2022 at 10:15 AM Jeremy Hansen <
> farnsworth.mcfadden@xxxxxxxxx<mailto:farnsworth.mcfadden@xxxxxxxxx>> wrote:
>
>>
>>
>> On Mon, Jul 25, 2022 at 7:26 AM Anthony D'Atri <anthony.datri@xxxxxxxxx<mailto:anthony.datri@xxxxxxxxx>>
>> wrote:
>>
>>> Do your values for public and cluster network include the new addresses
>>> on all nodes?
>>>
>>
>> This cluster only has one network. There is no separation between
>> public and cluster. Three of the nodes momentarily came up using a
>> different IP address.
>>
>> I've also noticed on one of the nodes that did not move or have any IP
>> issue, on a single node, from the dashboard, it names the same device for
>> two different osd's:
>>
>> 2 cn01 out destroyed hdd TOSHIBA_MG04SCA40EE_21M0A0CKFWZB Unknown sdb
>> osd.2
>>
>> 3 cn01 out destroyed ssd SAMSUNG_MZILT3T8HBLS/007_S5G0NE0R200159 Unknown
>> sdb osd.3
>>
>>
>> [ceph: root@cn01 /]# ceph-volume inventory
>>
>> Device Path Size rotates available Model name
>> /dev/sda 3.64 TB True False MG04SCA40EE
>> /dev/sdb 3.49 TB False False MZILT3T8HBLS/007
>> /dev/sdc 3.64 TB True False MG04SCA40EE
>> /dev/sdd 3.64 TB True False MG04SCA40EE
>> /dev/sde 3.49 TB False False MZILT3T8HBLS/007
>> /dev/sdf 3.64 TB True False MG04SCA40EE
>> /dev/sdg 698.64 GB True False SEAGATE ST375064
>>
>> [ceph: root@cn01 /]# ceph osd info
>> osd.0 down out weight 0 up_from 30231 up_thru 30564 down_at 30688
>> last_clean_interval [25500,30228) [v2:
>> 192.168.30.15:6818/2512683421,v1:192.168.30.15:6819/2512683421<http://192.168.30.15:6818/2512683421,v1:192.168.30.15:6819/2512683421>] [v2:
>> 192.168.30.15:6824/2512683421,v1:192.168.30.15:6826/2512683421<http://192.168.30.15:6824/2512683421,v1:192.168.30.15:6826/2512683421>]
>> autoout,exists d14cf503-a303-4fa4-a713-9530b67d613a
>> osd.1 down out weight 0 up_from 30393 up_thru 30688 down_at 30697
>> last_clean_interval [25518,30321) [v2:
>> 192.168.30.16:6834/1781855831,v1:192.168.30.16:6835/1781855831<http://192.168.30.16:6834/1781855831,v1:192.168.30.16:6835/1781855831>] [v2:
>> 192.168.30.16:6836/1781855831,v1:192.168.30.16:6837/1781855831<http://192.168.30.16:6836/1781855831,v1:192.168.30.16:6837/1781855831>]
>> autoout,exists 0d521411-c835-4fa3-beca-3631b4ff6bf7
>> osd.2 down out weight 0 up_from 31316 up_thru 31293 down_at 31317
>> last_clean_interval [31218,31296) [v2:
>> 192.168.30.11:6810/894589880,v1:192.168.30.11:6811/894589880<http://192.168.30.11:6810/894589880,v1:192.168.30.11:6811/894589880>] [v2:
>> 192.168.30.11:6812/894589880,v1:192.168.30.11:6813/894589880<http://192.168.30.11:6812/894589880,v1:192.168.30.11:6813/894589880>]
>> destroyed,exists
>> osd.3 down out weight 0 up_from 31265 up_thru 31266 down_at 31268
>> last_clean_interval [31254,31256) [v2:
>> 192.168.30.11:6818/1641948535,v1:192.168.30.11:6819/1641948535<http://192.168.30.11:6818/1641948535,v1:192.168.30.11:6819/1641948535>] [v2:
>> 192.168.30.11:6820/1641948535,v1:192.168.30.11:6821/1641948535<http://192.168.30.11:6820/1641948535,v1:192.168.30.11:6821/1641948535>]
>> destroyed,exists
>> osd.4 up in weight 1 up_from 31356 up_thru 31581 down_at 31339
>> last_clean_interval [31320,31338) [v2:
>> 192.168.30.11:6802/2785067179,v1:192.168.30.11:6803/2785067179<http://192.168.30.11:6802/2785067179,v1:192.168.30.11:6803/2785067179>] [v2:
>> 192.168.30.11:6804/2785067179,v1:192.168.30.11:6805/2785067179<http://192.168.30.11:6804/2785067179,v1:192.168.30.11:6805/2785067179>]
>> exists,up 3afd06db-b91d-44fe-9305-5eb95f7a59b9
>> osd.5 up in weight 1 up_from 31347 up_thru 31699 down_at 31339
>> last_clean_interval [31311,31338) [v2:
>> 192.168.30.11:6818/1936771540,v1:192.168.30.11:6819/1936771540<http://192.168.30.11:6818/1936771540,v1:192.168.30.11:6819/1936771540>] [v2:
>> 192.168.30.11:6820/1936771540,v1:192.168.30.11:6821/1936771540<http://192.168.30.11:6820/1936771540,v1:192.168.30.11:6821/1936771540>]
>> exists,up 063c2ccf-02ce-4f5e-8252-dddfbb258a95
>> osd.6 up in weight 1 up_from 31218 up_thru 31711 down_at 31217
>> last_clean_interval [30978,31195) [v2:
>> 192.168.30.12:6816/1585973160,v1:192.168.30.12:6817/1585973160<http://192.168.30.12:6816/1585973160,v1:192.168.30.12:6817/1585973160>] [v2:
>> 192.168.30.12:6818/1585973160,v1:192.168.30.12:6819/1585973160<http://192.168.30.12:6818/1585973160,v1:192.168.30.12:6819/1585973160>]
>> exists,up 94250ea2-f12e-4dc6-9135-b626086ccffd
>> osd.7 down out weight 0 up_from 30353 up_thru 30558 down_at 30688
>> last_clean_interval [25533,30349) [v2:
>> 192.168.30.14:6816/4083104061,v1:192.168.30.14:6817/4083104061<http://192.168.30.14:6816/4083104061,v1:192.168.30.14:6817/4083104061>] [v2:
>> 192.168.30.14:6840/4094104061,v1:192.168.30.14:6841/4094104061<http://192.168.30.14:6840/4094104061,v1:192.168.30.14:6841/4094104061>]
>> autoout,exists de351aec-b91e-4c22-a0bf-85369bc14579
>> osd.8 up in weight 1 up_from 31226 up_thru 31668 down_at 31225
>> last_clean_interval [30983,31195) [v2:
>> 192.168.30.12:6824/1312484329,v1:192.168.30.12:6825/1312484329<http://192.168.30.12:6824/1312484329,v1:192.168.30.12:6825/1312484329>] [v2:
>> 192.168.30.12:6826/1312484329,v1:192.168.30.12:6827/1312484329<http://192.168.30.12:6826/1312484329,v1:192.168.30.12:6827/1312484329>]
>> exists,up 51f665b4-fa5b-4b17-8390-ed130145ef04
>> osd.9 up in weight 1 up_from 31351 up_thru 31673 down_at 31340
>> last_clean_interval [31315,31338) [v2:
>> 192.168.30.11:6810/1446838877,v1:192.168.30.11:6811/1446838877<http://192.168.30.11:6810/1446838877,v1:192.168.30.11:6811/1446838877>] [v2:
>> 192.168.30.11:6812/1446838877,v1:192.168.30.11:6813/1446838877<http://192.168.30.11:6812/1446838877,v1:192.168.30.11:6813/1446838877>]
>> exists,up 985f1127-d126-4629-b8cd-03cf2d914d99
>> osd.10 up in weight 1 up_from 31219 up_thru 31639 down_at 31218
>> last_clean_interval [30980,31195) [v2:
>> 192.168.30.12:6808/1587842953,v1:192.168.30.12:6809/1587842953<http://192.168.30.12:6808/1587842953,v1:192.168.30.12:6809/1587842953>] [v2:
>> 192.168.30.12:6810/1587842953,v1:192.168.30.12:6811/1587842953<http://192.168.30.12:6810/1587842953,v1:192.168.30.12:6811/1587842953>]
>> exists,up c7fca03e-4bd5-4485-a090-658ca967d5f6
>> osd.11 up in weight 1 up_from 31234 up_thru 31659 down_at 31223
>> last_clean_interval [30978,31195) [v2:
>> 192.168.30.12:6840/3403200742,v1:192.168.30.12:6841/3403200742<http://192.168.30.12:6840/3403200742,v1:192.168.30.12:6841/3403200742>] [v2:
>> 192.168.30.12:6842/3403200742,v1:192.168.30.12:6843/3403200742<http://192.168.30.12:6842/3403200742,v1:192.168.30.12:6843/3403200742>]
>> exists,up 81074bd7-ad9f-4e56-8885-cca4745f6c95
>> osd.12 up in weight 1 up_from 31230 up_thru 31717 down_at 31223
>> last_clean_interval [30975,31195) [v2:
>> 192.168.30.13:6816/4268732910,v1:192.168.30.13:6817/4268732910<http://192.168.30.13:6816/4268732910,v1:192.168.30.13:6817/4268732910>] [v2:
>> 192.168.30.13:6818/4268732910,v1:192.168.30.13:6819/4268732910<http://192.168.30.13:6818/4268732910,v1:192.168.30.13:6819/4268732910>]
>> exists,up af1b55dd-c110-4861-aed9-c0737cef8be1
>> osd.13 down out weight 0 up_from 30389 up_thru 30688 down_at 30695
>> last_clean_interval [25513,30317) [v2:
>> 192.168.30.16:6804/1573803727,v1:192.168.30.16:6805/1573803727<http://192.168.30.16:6804/1573803727,v1:192.168.30.16:6805/1573803727>] [v2:
>> 192.168.30.16:6806/1573803727,v1:192.168.30.16:6807/1573803727<http://192.168.30.16:6806/1573803727,v1:192.168.30.16:6807/1573803727>]
>> autoout,exists 737a3234-0f1f-4286-80e9-e89b581aae41
>> osd.14 up in weight 1 up_from 31218 up_thru 31709 down_at 31217
>> last_clean_interval [30979,31195) [v2:
>> 192.168.30.13:6834/2291187817,v1:192.168.30.13:6835/2291187817<http://192.168.30.13:6834/2291187817,v1:192.168.30.13:6835/2291187817>] [v2:
>> 192.168.30.13:6836/2291187817,v1:192.168.30.13:6837/2291187817<http://192.168.30.13:6836/2291187817,v1:192.168.30.13:6837/2291187817>]
>> exists,up 97cd6ac7-aca0-42fd-a049-d27289f83183
>> osd.15 down out weight 0 up_from 29463 up_thru 30531 down_at 30688
>> last_clean_interval [25493,29462) [v2:
>> 192.168.30.15:6808/2655269991,v1:192.168.30.15:6809/2655269991<http://192.168.30.15:6808/2655269991,v1:192.168.30.15:6809/2655269991>] [v2:
>> 192.168.30.15:6802/2662269991,v1:192.168.30.15:6803/2662269991<http://192.168.30.15:6802/2662269991,v1:192.168.30.15:6803/2662269991>]
>> autoout,exists 61aea8f4-5905-4be3-ae32-5eacf75a514e
>> osd.16 up in weight 1 up_from 31226 up_thru 31647 down_at 31223
>> last_clean_interval [30970,31195) [v2:
>> 192.168.30.13:6808/2624812081,v1:192.168.30.13:6809/2624812081<http://192.168.30.13:6808/2624812081,v1:192.168.30.13:6809/2624812081>] [v2:
>> 192.168.30.13:6810/2624812081,v1:192.168.30.13:6811/2624812081<http://192.168.30.13:6810/2624812081,v1:192.168.30.13:6811/2624812081>]
>> exists,up 791a7542-87cd-403d-a37e-8f00506b2eb6
>> osd.17 up in weight 1 up_from 31219 up_thru 31703 down_at 31218
>> last_clean_interval [30975,31195) [v2:
>> 192.168.30.13:6800/2978036397,v1:192.168.30.13:6801/2978036397<http://192.168.30.13:6800/2978036397,v1:192.168.30.13:6801/2978036397>] [v2:
>> 192.168.30.13:6802/2978036397,v1:192.168.30.13:6803/2978036397<http://192.168.30.13:6802/2978036397,v1:192.168.30.13:6803/2978036397>]
>> exists,up 4a915645-412f-49e6-8477-1577469905da
>> osd.18 down out weight 0 up_from 30334 up_thru 30566 down_at 30688
>> last_clean_interval [25543,30327) [v2:
>> 192.168.30.14:6832/985432137,v1:192.168.30.14:6833/985432137<http://192.168.30.14:6832/985432137,v1:192.168.30.14:6833/985432137>] [v2:
>> 192.168.30.14:6848/998432137,v1:192.168.30.14:6849/998432137<http://192.168.30.14:6848/998432137,v1:192.168.30.14:6849/998432137>]
>> autoout,exists 85f59d83-710c-4896-9200-bda4894fc3e8
>> osd.19 down in weight 1 up_from 31303 up_thru 31321 down_at 31323
>> last_clean_interval [31292,31296) [v2:
>> 192.168.30.13:6826/375623427,v1:192.168.30.13:6827/375623427<http://192.168.30.13:6826/375623427,v1:192.168.30.13:6827/375623427>] [v2:
>> 192.168.30.13:6828/375623427,v1:192.168.30.13:6829/375623427<http://192.168.30.13:6828/375623427,v1:192.168.30.13:6829/375623427>] exists
>> 7d09b51a-bd6d-40f8-a009-78ab9937795d
>> osd.20 down out weight 0 up_from 30445 up_thru 30531 down_at 30688
>> last_clean_interval [28947,30444) [v2:
>> 192.168.30.14:6810/4062649162,v1:192.168.30.14:6811/4062649162<http://192.168.30.14:6810/4062649162,v1:192.168.30.14:6811/4062649162>] [v2:
>> 192.168.30.14:6800/4073649162,v1:192.168.30.14:6801/4073649162<http://192.168.30.14:6800/4073649162,v1:192.168.30.14:6801/4073649162>]
>> autoout,exists 7ef6cc1a-4755-4a14-b9df-f1f538d903ce
>> osd.21 up in weight 1 up_from 31345 up_thru 31567 down_at 31341
>> last_clean_interval [31307,31340) [v2:
>> 192.168.30.11:6826/1625231298,v1:192.168.30.11:6827/1625231298<http://192.168.30.11:6826/1625231298,v1:192.168.30.11:6827/1625231298>] [v2:
>> 192.168.30.11:6828/1625231298,v1:192.168.30.11:6829/1625231298<http://192.168.30.11:6828/1625231298,v1:192.168.30.11:6829/1625231298>]
>> exists,up 5ef2e39a-a353-4cb8-a49e-093fe39b94ef
>> osd.22 down in weight 1 up_from 30383 up_thru 30528 down_at 30688
>> last_clean_interval [25549,30317) [v2:
>> 192.168.30.14:6806/1204256629,v1:192.168.30.14:6807/1204256629<http://192.168.30.14:6806/1204256629,v1:192.168.30.14:6807/1204256629>] [v2:
>> 192.168.30.14:6812/1204256629,v1:192.168.30.14:6813/1204256629<http://192.168.30.14:6812/1204256629,v1:192.168.30.14:6813/1204256629>] exists
>> c9befe11-a035-449c-8d17-42aaf191923d
>> osd.23 down in weight 1 up_from 30334 up_thru 30576 down_at 30688
>> last_clean_interval [30263,30332) [v2:
>> 192.168.30.14:6802/3837786490,v1:192.168.30.14:6803/3837786490<http://192.168.30.14:6802/3837786490,v1:192.168.30.14:6803/3837786490>] [v2:
>> 192.168.30.14:6830/3838786490,v1:192.168.30.14:6831/3838786490<http://192.168.30.14:6830/3838786490,v1:192.168.30.14:6831/3838786490>] exists
>> 2081147b-065d-4da7-89d9-747e1ae02b8d
>> osd.24 down in weight 1 up_from 29455 up_thru 30570 down_at 30688
>> last_clean_interval [25487,29453) [v2:
>> 192.168.30.15:6800/2008474583,v1:192.168.30.15:6801/2008474583<http://192.168.30.15:6800/2008474583,v1:192.168.30.15:6801/2008474583>] [v2:
>> 192.168.30.15:6810/2016474583,v1:192.168.30.15:6811/2016474583<http://192.168.30.15:6810/2016474583,v1:192.168.30.15:6811/2016474583>] exists
>> 39d78380-261c-4689-b53d-90713e6ffcca
>> osd.26 up in weight 1 up_from 31208 up_thru 31643 down_at 31207
>> last_clean_interval [30967,31195) [v2:
>> 192.168.30.12:6800/2861018947,v1:192.168.30.12:6801/2861018947<http://192.168.30.12:6800/2861018947,v1:192.168.30.12:6801/2861018947>] [v2:
>> 192.168.30.12:6802/2861018947,v1:192.168.30.12:6803/2861018947<http://192.168.30.12:6802/2861018947,v1:192.168.30.12:6803/2861018947>]
>> exists,up 046622c8-c09c-4254-8c15-3dc05a2f01ed
>> osd.28 down in weight 1 up_from 30389 up_thru 30574 down_at 30691
>> last_clean_interval [25513,30312) [v2:
>> 192.168.30.16:6820/3466284570,v1:192.168.30.16:6821/3466284570<http://192.168.30.16:6820/3466284570,v1:192.168.30.16:6821/3466284570>] [v2:
>> 192.168.30.16:6822/3466284570,v1:192.168.30.16:6823/3466284570<http://192.168.30.16:6822/3466284570,v1:192.168.30.16:6823/3466284570>] exists
>> 10578b97-e3c4-4553-a8d0-6dcc46af5db1
>> osd.29 down in weight 1 up_from 30378 up_thru 30554 down_at 30688
>> last_clean_interval [28595,30376) [v2:
>> 192.168.30.14:6808/3739543672,v1:192.168.30.14:6809/3739543672<http://192.168.30.14:6808/3739543672,v1:192.168.30.14:6809/3739543672>] [v2:
>> 192.168.30.14:6846/3747543672,v1:192.168.30.14:6847/3747543672<http://192.168.30.14:6846/3747543672,v1:192.168.30.14:6847/3747543672>] exists
>> 9698e936-8edf-4adf-92c9-a0b5202ed01a
>> osd.30 down in weight 1 up_from 30449 up_thru 30531 down_at 30688
>> last_clean_interval [25502,30446) [v2:
>> 192.168.30.15:6825/2375507296,v1:192.168.30.15:6827/2375507296<http://192.168.30.15:6825/2375507296,v1:192.168.30.15:6827/2375507296>] [v2:
>> 192.168.30.15:6829/2375507296,v1:192.168.30.15:6831/2375507296<http://192.168.30.15:6829/2375507296,v1:192.168.30.15:6831/2375507296>] exists
>> e14d2a0f-a98a-44d4-8c06-4d893f673629
>> osd.31 down in weight 1 up_from 30364 up_thru 30688 down_at 30700
>> last_clean_interval [25514,30361) [v2:
>> 192.168.30.16:6826/2835000708,v1:192.168.30.16:6827/2835000708<http://192.168.30.16:6826/2835000708,v1:192.168.30.16:6827/2835000708>] [v2:
>> 192.168.30.16:6802/2843000708,v1:192.168.30.16:6803/2843000708<http://192.168.30.16:6802/2843000708,v1:192.168.30.16:6803/2843000708>] exists
>> 541bca38-e704-483a-8cb8-39e5f69007d1
>> osd.32 up in weight 1 up_from 31209 up_thru 31627 down_at 31208
>> last_clean_interval [30974,31195) [v2:
>> 192.168.30.12:6832/3860067997,v1:192.168.30.12:6833/3860067997<http://192.168.30.12:6832/3860067997,v1:192.168.30.12:6833/3860067997>] [v2:
>> 192.168.30.12:6834/3860067997,v1:192.168.30.12:6835/3860067997<http://192.168.30.12:6834/3860067997,v1:192.168.30.12:6835/3860067997>]
>> exists,up 9200a57e-2845-43ff-9787-8f1f3158fe90
>> osd.33 down in weight 1 up_from 30354 up_thru 30688 down_at 30693
>> last_clean_interval [25521,30350) [v2:
>> 192.168.30.16:6842/2342555666,v1:192.168.30.16:6843/2342555666<http://192.168.30.16:6842/2342555666,v1:192.168.30.16:6843/2342555666>] [v2:
>> 192.168.30.16:6844/2364555666,v1:192.168.30.16:6845/2364555666<http://192.168.30.16:6844/2364555666,v1:192.168.30.16:6845/2364555666>] exists
>> 20c55d85-cf9a-4133-a189-7fdad2318f58
>> osd.34 down in weight 1 up_from 30390 up_thru 30688 down_at 30691
>> last_clean_interval [25516,30314) [v2:
>> 192.168.30.16:6808/2282629870,v1:192.168.30.16:6811/2282629870<http://192.168.30.16:6808/2282629870,v1:192.168.30.16:6811/2282629870>] [v2:
>> 192.168.30.16:6812/2282629870,v1:192.168.30.16:6814/2282629870<http://192.168.30.16:6812/2282629870,v1:192.168.30.16:6814/2282629870>] exists
>> 77e0ef8f-c047-4f84-afb2-a8ad054e562f
>> osd.35 up in weight 1 up_from 31204 up_thru 31657 down_at 31203
>> last_clean_interval [30958,31195) [v2:
>> 192.168.30.13:6842/1919357520,v1:192.168.30.13:6843/1919357520<http://192.168.30.13:6842/1919357520,v1:192.168.30.13:6843/1919357520>] [v2:
>> 192.168.30.13:6844/1919357520,v1:192.168.30.13:6845/1919357520<http://192.168.30.13:6844/1919357520,v1:192.168.30.13:6845/1919357520>]
>> exists,up 2d2de0cb-6d41-4957-a473-2bbe9ce227bf
>> osd.36 down in weight 1 up_from 29494 up_thru 30560 down_at 30688
>> last_clean_interval [25491,29492) [v2:
>> 192.168.30.15:6816/2153321591,v1:192.168.30.15:6817/2153321591<http://192.168.30.15:6816/2153321591,v1:192.168.30.15:6817/2153321591>] [v2:
>> 192.168.30.15:6842/2158321591,v1:192.168.30.15:6843/2158321591<http://192.168.30.15:6842/2158321591,v1:192.168.30.15:6843/2158321591>] exists
>> 26114668-68b2-458b-89c2-cbad5507ab75
>>
>>
>>
>>>
>>> > On Jul 25, 2022, at 3:29 AM, Jeremy Hansen <
>>> farnsworth.mcfadden@xxxxxxxxx<mailto:farnsworth.mcfadden@xxxxxxxxx>> wrote:
>>> >
>>> > I transitioned some servers to a new rack and now I'm having major
>>> issues
>>> > with Ceph upon bringing things back up.
>>> >
>>> > I believe the issue may be related to the ceph nodes coming back up
>>> with
>>> > different IPs before VLANs were set. That's just a guess because I
>>> can't
>>> > think of any other reason this would happen.
>>> >
>>> > Current state:
>>> >
>>> > Every 2.0s: ceph -s
>>> > cn01.ceph.la1.clx.corp: Mon Jul 25 10:13:05 2022
>>> >
>>> > cluster:
>>> > id: bfa2ad58-c049-11eb-9098-3c8cf8ed728d
>>> > health: HEALTH_WARN
>>> > 1 filesystem is degraded
>>> > 2 MDSs report slow metadata IOs
>>> > 2/5 mons down, quorum cn02,cn03,cn01
>>> > 9 osds down
>>> > 3 hosts (17 osds) down
>>> > Reduced data availability: 97 pgs inactive, 9 pgs down
>>> > Degraded data redundancy: 13860144/30824413 objects degraded
>>> > (44.965%), 411 pgs degraded, 482 pgs undersized
>>> >
>>> > services:
>>> > mon: 5 daemons, quorum cn02,cn03,cn01 (age 62m), out of quorum:
>>> cn05,
>>> > cn04
>>> > mgr: cn02.arszct(active, since 5m)
>>> > mds: 2/2 daemons up, 2 standby
>>> > osd: 35 osds: 15 up (since 62m), 24 in (since 58m); 222 remapped pgs
>>> >
>>> > data:
>>> > volumes: 1/2 healthy, 1 recovering
>>> > pools: 8 pools, 545 pgs
>>> > objects: 7.71M objects, 6.7 TiB
>>> > usage: 15 TiB used, 39 TiB / 54 TiB avail
>>> > pgs: 0.367% pgs unknown
>>> > 17.431% pgs not active
>>> > 13860144/30824413 objects degraded (44.965%)
>>> > 1137693/30824413 objects misplaced (3.691%)
>>> > 280 active+undersized+degraded
>>> > 67 undersized+degraded+remapped+backfilling+peered
>>> > 57 active+undersized+remapped
>>> > 45 active+clean+remapped
>>> > 44 active+undersized+degraded+remapped+backfilling
>>> > 18 undersized+degraded+peered
>>> > 10 active+undersized
>>> > 9 down
>>> > 7 active+clean
>>> > 3 active+undersized+remapped+backfilling
>>> > 2 active+undersized+degraded+remapped+backfill_wait
>>> > 2 unknown
>>> > 1 undersized+peered
>>> >
>>> > io:
>>> > client: 170 B/s rd, 0 op/s rd, 0 op/s wr
>>> > recovery: 168 MiB/s, 158 keys/s, 166 objects/s
>>> >
>>> > I have to disable and re-enable the dashboard just to use it. It
>>> seems to
>>> > get bogged down after a few moments.
>>> >
>>> > The three servers that were moved to the new rack Ceph has marked as
>>> > "Down", but if I do a cephadm host-check, they all seem to pass:
>>> >
>>> > ************************ ceph ************************
>>> > --------- cn01.ceph.---------
>>> > podman (/usr/bin/podman) version 4.0.2 is present
>>> > systemctl is present
>>> > lvcreate is present
>>> > Unit chronyd.service is enabled and running
>>> > Host looks OK
>>> > --------- cn02.ceph.---------
>>> > podman (/usr/bin/podman) version 4.0.2 is present
>>> > systemctl is present
>>> > lvcreate is present
>>> > Unit chronyd.service is enabled and running
>>> > Host looks OK
>>> > --------- cn03.ceph.---------
>>> > podman (/usr/bin/podman) version 4.0.2 is present
>>> > systemctl is present
>>> > lvcreate is present
>>> > Unit chronyd.service is enabled and running
>>> > Host looks OK
>>> > --------- cn04.ceph.---------
>>> > podman (/usr/bin/podman) version 4.0.2 is present
>>> > systemctl is present
>>> > lvcreate is present
>>> > Unit chronyd.service is enabled and running
>>> > Host looks OK
>>> > --------- cn05.ceph.---------
>>> > podman|docker (/usr/bin/podman) is present
>>> > systemctl is present
>>> > lvcreate is present
>>> > Unit chronyd.service is enabled and running
>>> > Host looks OK
>>> > --------- cn06.ceph.---------
>>> > podman (/usr/bin/podman) version 4.0.2 is present
>>> > systemctl is present
>>> > lvcreate is present
>>> > Unit chronyd.service is enabled and running
>>> > Host looks OK
>>> >
>>> > It seems to be recovering with what it has left, but a large amount of
>>> OSDs
>>> > are down. When trying to restart one of the down'd OSDs, I see a huge
>>> dump.
>>> >
>>> > Jul 25 03:19:38 cn06.ceph
>>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>>> > 2022-07-25T10:19:38.532+0000 7fce14a6c080 0 osd.34 30689 done with
>>> init,
>>> > starting boot process
>>> > Jul 25 03:19:38 cn06.ceph
>>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>>> > 2022-07-25T10:19:38.532+0000 7fce14a6c080 1 osd.34 30689 start_boot
>>> > Jul 25 03:20:10 cn06.ceph
>>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>>> > 2022-07-25T10:20:10.655+0000 7fcdfd12d700 1 osd.34 30689 start_boot
>>> > Jul 25 03:20:41 cn06.ceph
>>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>>> > 2022-07-25T10:20:41.159+0000 7fcdfd12d700 1 osd.34 30689 start_boot
>>> > Jul 25 03:21:11 cn06.ceph
>>> > ceph-bfa2ad58-c049-11eb-9098-3c8cf8ed728d-osd-34[9516]: debug
>>> > 2022-07-25T10:21:11.662+0000 7fcdfd12d700 1 osd.34 30689 start_boot
>>> >
>>> > At this point it just keeps printing start_boot, but the dashboard has
>>> it
>>> > marked as "in" but "down".
>>> >
>>> > On these three hosts that moved, there were a bunch marked as "out" and
>>> > "down", and some with "in" but "down".
>>> >
>>> > Not sure where to go next. I'm going to let the recovery continue and
>>> hope
>>> > that my 4x replication on these pools saves me.
>>> >
>>> > Not sure where to go from here. Any help is very much appreciated.
>>> This
>>> > Ceph cluster holds all of our Cloudstack images... it would be
>>> terrible to
>>> > lose this data.
>>> > _______________________________________________
>>> > ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
>>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx>
>>>
>>>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
