removed the config setting for mon. a001s016. Here it is # ceph config get mon container_image docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 # ceph config get osd container_image docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 # ceph config get mgr mgr/cephadm/container_image_base docker.io/ceph/daemon Thank you, Anantha -----Original Message----- From: Eugen Block <eblock@xxxxxx> Sent: Wednesday, April 3, 2024 12:27 AM To: Adiga, Anantha <anantha.adiga@xxxxxxxxx> Cc: ceph-users@xxxxxxx Subject: Re: Re: ceph status not showing correct monitor services I have no idea what you did there ;-) I would remove that config though and rather configure the ceph image globally, there have been several issues when cephadm tries to launch daemons with different ceph versions. Although in your case it looks like they are actually the same images according to the digest (and also in the ceph orch ps output). But it might cause some trouble anyway, so I'd recommend to remove the individual config for mon.a001s016 and only use the global config. Can you add these outputs (mask sensitive data)? ceph config get mon container_image ceph config get osd container_image ceph config get mgr mgr/cephadm/container_image_base Zitat von "Adiga, Anantha" <anantha.adiga@xxxxxxxxx>: > Hi Eugen, > > > Noticed this in the config dump: Why only "mon.a001s016 " > listed? And this is the one that is not listed in "ceph -s" > > > mon advanced > auth_allow_insecure_global_id_reclaim false > mon advanced > auth_expose_insecure_global_id_reclaim false > mon advanced > mon_compact_on_start true > mon.a001s016 basic container_image > > docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 > * > mgr advanced > mgr/cephadm/container_image_base docker.io/ceph/daemon > mgr advanced > mgr/cephadm/container_image_node_exporter > docker.io/prom/node-exporter:v0.17.0 > > > cluster: > id: 604d56db-2fab-45db-a9ea-c418f9a8cca8 > health: HEALTH_OK > > services: > mon: 2 daemons, quorum a001s018,a001s017 (age 45h) > mgr: a001s016.ctmoay(active, since 28h), standbys: a001s017.bpygfm > mds: 1/1 daemons up, 2 standby > osd: 36 osds: 36 up (since 29h), 36 in (since 2y) > rgw: 3 daemons active (3 hosts, 1 zones) > > var lib mon unit.image > ==================== > > a001s016: > # cat > /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/unit.i > mage > docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a > 2cb7ff7668f776b61b9d586 > > a001s017: > # cat > /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s017/unit.i > mage > docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a > 2cb7ff7668f776b61b9d586 > a001s018: > # cat > /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s018/unit.i > mage > docker.io/ceph/daemon:latest-pacific > > ceph image tag, digest from docker inspect of: ceph/daemon > latest-pacific 6e73176320aa 2 years ago 1.27GB > ====================================================================== > ============================ > a001s016: > "Id": > "sha256:6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", > "RepoTags": [ > "ceph/daemon:latest-pacific" > "RepoDigests": [ > > "ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586" > > a001s017: > "Id": > "sha256:6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", > "RepoTags": [ > "ceph/daemon:latest-pacific" > "RepoDigests": [ > > "ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586" > > a001s018: > "Id": > "sha256:6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", > "RepoTags": [ > "ceph/daemon:latest-pacific" > "RepoDigests": [ > > "ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586" > > -----Original Message----- > From: Adiga, Anantha > Sent: Tuesday, April 2, 2024 10:42 AM > To: Eugen Block <eblock@xxxxxx> > Cc: ceph-users@xxxxxxx > Subject: RE: Re: ceph status not showing correct monitor > services > > Hi Eugen, > > Currently there are only three nodes, but I can add a node to the > cluster and check it out. I will take a look at the mon logs > > > Thank you, > Anantha > > -----Original Message----- > From: Eugen Block <eblock@xxxxxx> > Sent: Tuesday, April 2, 2024 12:19 AM > To: Adiga, Anantha <anantha.adiga@xxxxxxxxx> > Cc: ceph-users@xxxxxxx > Subject: Re: Re: ceph status not showing correct monitor > services > > You can add a mon manually to the monmap, but that requires a downtime > of the mons. Here's an example [1] how to modify the monmap (including > network change which you don't need, of course). But that would be my > last resort, first I would try to find out why the MON fails to join > the quorum. What is that mon.a001s016 logging, and what are the other > two logging? > Do you have another host where you could place a mon daemon to see if > that works? > > > [1] > https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#examp > le-procedure > > Zitat von "Adiga, Anantha" <anantha.adiga@xxxxxxxxx>: > >> # ceph mon stat >> e6: 2 mons at >> {a001s017=[v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0],a001s018=[v >> 2 :10.45.128.28:3300/0,v1:10.45.128.28:6789/0]}, election epoch 162, >> leader 0 a001s018, quorum 0,1 >> a001s018,a001s017 >> >> # ceph orch ps | grep mon >> mon.a001s016 a001s016 running >> (3h) 6m ago 3h 527M 2048M 16.2.5 >> 6e73176320aa 39db8cfba7e1 >> mon.a001s017 a001s017 running >> (22h) 47s ago 1h 993M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 48s ago 2y 1167M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> >> # ceph mgr stat >> { >> "epoch": 1130365, >> "available": true, >> "active_name": "a001s016.ctmoay", >> "num_standby": 1 >> } >> >> # ceph orch ps | grep mgr >> mgr.a001s016.ctmoay a001s016 *:8443 running >> (18M) 109s ago 23M 518M - 16.2.5 >> 6e73176320aa 169cafcbbb99 >> mgr.a001s017.bpygfm a001s017 *:8443 running >> (19M) 5m ago 23M 501M - 16.2.5 >> 6e73176320aa 97257195158c >> mgr.a001s018.hcxnef a001s018 *:8443 running >> (20M) 5m ago 23M 113M - 16.2.5 >> 6e73176320aa 21ba5896cee2 >> >> # ceph orch ls --service_name=mgr --export >> service_type: mgr >> service_name: mgr >> placement: >> count: 3 >> hosts: >> - a001s016 >> - a001s017 >> - a001s018 >> >> # ceph orch ls --service_name=mon --export >> service_type: mon >> service_name: mon >> placement: >> count: 3 >> hosts: >> - a001s016 >> - a001s017 >> - a001s018 >> >> -----Original Message----- >> From: Adiga, Anantha >> Sent: Monday, April 1, 2024 6:06 PM >> To: Eugen Block <eblock@xxxxxx> >> Cc: ceph-users@xxxxxxx >> Subject: RE: Re: ceph status not showing correct monitor >> services >> >> # ceph tell mon.a001s016 mon_status Error ENOENT: problem getting >> command descriptions from mon.a001s016 >> >> a001s016 is outside quorum see below >> >> # ceph tell mon.a001s017 mon_status { >> "name": "a001s017", >> "rank": 1, >> "state": "peon", >> "election_epoch": 162, >> "quorum": [ >> 0, >> 1 >> ], >> "quorum_age": 79938, >> "features": { >> "required_con": "2449958747317026820", >> "required_mon": [ >> "kraken", >> "luminous", >> "mimic", >> "osdmap-prune", >> "nautilus", >> "octopus", >> "pacific", >> "elector-pinging" >> ], >> "quorum_con": "4540138297136906239", >> "quorum_mon": [ >> "kraken", >> "luminous", >> "mimic", >> "osdmap-prune", >> "nautilus", >> "octopus", >> "pacific", >> "elector-pinging" >> ] >> }, >> "outside_quorum": [], >> "extra_probe_peers": [ >> { >> "addrvec": [ >> { >> "type": "v2", >> "addr": "10.45.128.26:3300", >> "nonce": 0 >> }, >> { >> "type": "v1", >> "addr": "10.45.128.26:6789", >> "nonce": 0 >> } >> ] >> } >> ], >> "sync_provider": [], >> "monmap": { >> "epoch": 6, >> "fsid": "604d56db-2fab-45db-a9ea-c418f9a8cca8", >> "modified": "2024-03-31T23:54:18.692983Z", >> "created": "2021-09-30T16:15:12.884602Z", >> "min_mon_release": 16, >> "min_mon_release_name": "pacific", >> "election_strategy": 1, >> "disallowed_leaders: ": "", >> "stretch_mode": false, >> "features": { >> "persistent": [ >> "kraken", >> "luminous", >> "mimic", >> "osdmap-prune", >> "nautilus", >> "octopus", >> "pacific", >> "elector-pinging" >> ], >> "optional": [] >> }, >> "mons": [ >> { >> "rank": 0, >> "name": "a001s018", >> "public_addrs": { >> "addrvec": [ >> { >> "type": "v2", >> "addr": "10.45.128.28:3300", >> "nonce": 0 >> }, >> { >> "type": "v1", >> "addr": "10.45.128.28:6789", >> "nonce": 0 >> } >> ] >> }, >> "addr": "10.45.128.28:6789/0", >> "public_addr": "10.45.128.28:6789/0", >> "priority": 0, >> "weight": 0, >> "crush_location": "{}" >> }, >> { >> "rank": 1, >> "name": "a001s017", >> "public_addrs": { >> "addrvec": [ >> { >> "type": "v2", >> "addr": "10.45.128.27:3300", >> "nonce": 0 >> }, >> { >> "type": "v1", >> "addr": "10.45.128.27:6789", >> "nonce": 0 >> } >> ] >> }, >> "addr": "10.45.128.27:6789/0", >> "public_addr": "10.45.128.27:6789/0", >> "priority": 0, >> "weight": 0, >> "crush_location": "{}" >> } >> ] >> }, >> "feature_map": { >> "mon": [ >> { >> "features": "0x3f01cfb9fffdffff", >> "release": "luminous", >> "num": 1 >> } >> ], >> "mds": [ >> { >> "features": "0x3f01cfb9fffdffff", >> "release": "luminous", >> "num": 3 >> } >> ], >> "osd": [ >> { >> "features": "0x3f01cfb9fffdffff", >> "release": "luminous", >> "num": 15 >> } >> ], >> "client": [ >> { >> "features": "0x2f018fb86aa42ada", >> "release": "luminous", >> "num": 50 >> }, >> { >> "features": "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 40 >> }, >> { >> "features": "0x3f01cfb8ffedffff", >> "release": "luminous", >> "num": 1 >> }, >> { >> "features": "0x3f01cfb9fffdffff", >> "release": "luminous", >> "num": 72 >> } >> ] >> }, >> "stretch_mode": false >> } >> >> >> # ceph mon dump >> dumped monmap epoch 6 epoch 6 fsid >> 604d56db-2fab-45db-a9ea-c418f9a8cca8 >> last_changed 2024-03-31T23:54:18.692983+0000 created >> 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) >> election_strategy: 1 >> 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 >> 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 >> root@a001s016:/var/run/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8# >> >> -----Original Message----- >> From: Adiga, Anantha >> Sent: Monday, April 1, 2024 3:20 PM >> To: Adiga, Anantha <anantha.adiga@xxxxxxxxx>; Eugen Block >> <eblock@xxxxxx> >> Cc: ceph-users@xxxxxxx >> Subject: RE: Re: ceph status not showing correct monitor >> services >> >> Both methods are not update the mon map, is there a way to inject >> mon.a001s016 into the current mon map? >> >> # ceph mon dump >> dumped monmap epoch 6 >> epoch 6 >> fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 >> last_changed 2024-03-31T23:54:18.692983+0000 created >> 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) >> election_strategy: 1 >> 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 >> 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 >> >> # ceph tell mon.a001s016 mon_status >> Error ENOENT: problem getting command descriptions from mon.a001s016 >> >> # ceph tell mon.a001s016 mon_status >> Error ENOENT: problem getting command descriptions from mon.a001s016 >> >> # ceph tell mon.a001s017 mon_status >> { >> "name": "a001s017", >> "rank": 1, >> "state": "peon", >> "election_epoch": 162, >> "quorum": [ >> 0, >> 1 >> ], >> "quorum_age": 69551, >> "features": { >> .............. >> .............. >> >> >> # ceph orch ls --service_name=mon --export > mon3.yml >> service_type: mon >> service_name: mon >> placement: >> count: 3 >> hosts: >> - a001s016 >> - a001s017 >> - a001s018 >> >> # cp mon3.yml mon2.yml >> # vi mon2.yml >> #cat mon2.yml >> service_type: mon >> service_name: mon >> placement: >> count: 2 >> hosts: >> - a001s017 >> - a001s018 >> >> # ceph orch apply -i mon2.yml --dry-run WARNING! Dry-Runs are >> snapshots of a certain point in time and are bound to the current >> inventory setup. If any on these conditions changes, the preview will >> be invalid. Please make sure to have a minimal timeframe between >> planning and applying the specs. >> #################### >> SERVICESPEC PREVIEWS >> #################### >> +---------+------+--------+--------------+ >> |SERVICE |NAME |ADD_TO |REMOVE_FROM | >> +---------+------+--------+--------------+ >> |mon |mon | |mon.a001s016 | >> +---------+------+--------+--------------+ >> ################ >> OSDSPEC PREVIEWS >> ################ >> +---------+------+------+------+----+-----+ >> |SERVICE |NAME |HOST |DATA |DB |WAL | >> +---------+------+------+------+----+-----+ >> +---------+------+------+------+----+-----+ >> >> # ceph orch ls --service_name=mon --refresh NAME PORTS RUNNING >> REFRESHED AGE PLACEMENT >> mon 3/3 5m ago 18h a001s016;a001s017;a001s018;count:3 >> >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (21h) 2s ago 21h 734M 2048M 16.2.5 >> 6e73176320aa 8484a912f96a >> mon.a001s017 a001s017 running >> (18h) 2s ago 21h 976M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 2s ago 2y 1164M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (21h) 37s ago 21h 734M 2048M 16.2.5 >> 6e73176320aa 8484a912f96a >> mon.a001s017 a001s017 running >> (18h) 37s ago 21h 977M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 38s ago 2y 1166M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> >> # ceph orch apply -i mon2.yml >> Scheduled mon update... >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (21h) 21s ago 21h 734M 2048M 16.2.5 >> 6e73176320aa 8484a912f96a >> mon.a001s017 a001s017 running >> (18h) 20s ago 21h 962M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 21s ago 2y 1156M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s017 a001s017 running >> (18h) 23s ago 21h 962M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 24s ago 2y 1156M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s017 a001s017 running >> (18h) 27s ago 21h 962M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 0s ago 2y 1154M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s017 a001s017 running >> (18h) 2s ago 21h 960M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 3s ago 2y 1154M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s017 a001s017 running >> (18h) 5s ago 21h 960M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 0s ago 2y 1154M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s017 a001s017 running >> (18h) 2s ago 21h 962M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 2s ago 2y 1154M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> >> # ls /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/ >> cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931 >> keepalived.nfs.nfs.a001s016.ofqyjl osd.1 osd.22 osd.4 >> crash >> mds.cephfs.a001s016.vtyivn osd.10 osd.24 osd.7 >> crash.a001s016 >> mgr.a001s016.ctmoay osd.13 osd.27 removed >> haproxy.nfs.nfs.a001s016.vpjtxt >> nfs.nfs.0.0.a001s016.wumnjs osd.15 osd.30 >> rgw.ceph.a001s016.gjyanl >> home >> node-exporter.a001s016 osd.18 osd.33 >> >> # ceph mon dump >> dumped monmap epoch 6 >> epoch 6 >> fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 >> last_changed 2024-03-31T23:54:18.692983+0000 created >> 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) >> election_strategy: 1 >> 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 >> 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 >> >> # cat mon3.yml >> service_type: mon >> service_name: mon >> placement: >> count: 3 >> hosts: >> - a001s016 >> - a001s017 >> - a001s018 >> >> # ceph orch apply -i mon3.yml --dry-run WARNING! Dry-Runs are >> snapshots of a certain point in time and are bound to the current >> inventory setup. If any on these conditions changes, the preview will >> be invalid. Please make sure to have a minimal timeframe between >> planning and applying the specs. >> #################### >> SERVICESPEC PREVIEWS >> #################### >> +---------+------+----------+-------------+ >> |SERVICE |NAME |ADD_TO |REMOVE_FROM | >> +---------+------+----------+-------------+ >> |mon |mon |a001s016 | | >> +---------+------+----------+-------------+ >> ################ >> OSDSPEC PREVIEWS >> ################ >> +---------+------+------+------+----+-----+ >> |SERVICE |NAME |HOST |DATA |DB |WAL | >> +---------+------+------+------+----+-----+ >> +---------+------+------+------+----+-----+ >> >> # ceph orch apply -i mon3.yml >> Scheduled mon update... >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 starting >> - - - 2048M <unknown> <unknown> >> <unknown> >> mon.a001s017 a001s017 running >> (18h) 2s ago 21h 960M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 3s ago 2y 1152M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (6s) 0s ago 6s 14.4M 2048M 16.2.5 >> 6e73176320aa dcd4705c2069 >> mon.a001s017 a001s017 running >> (18h) 8s ago 21h 960M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 9s ago 2y 1152M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (10s) 4s ago 10s 14.4M 2048M 16.2.5 >> 6e73176320aa dcd4705c2069 >> mon.a001s017 a001s017 running >> (18h) 12s ago 21h 960M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 13s ago 2y 1152M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> >> # ls /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/ >> cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931 >> mgr.a001s016.ctmoay osd.15 osd.4 >> crash >> mon.a001s016 osd.18 osd.7 >> crash.a001s016 >> nfs.nfs.0.0.a001s016.wumnjs osd.22 removed >> haproxy.nfs.nfs.a001s016.vpjtxt >> node-exporter.a001s016 osd.24 rgw.ceph.a001s016.gjyanl >> home >> osd.1 osd.27 >> keepalived.nfs.nfs.a001s016.ofqyjl >> osd.10 osd.30 >> mds.cephfs.a001s016.vtyivn >> osd.13 osd.33 >> >> # ceph mon dump >> dumped monmap epoch 6 >> epoch 6 >> fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 >> last_changed 2024-03-31T23:54:18.692983+0000 created >> 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) >> election_strategy: 1 >> 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 >> 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 >> >> # ceph -s >> cluster: >> id: 604d56db-2fab-45db-a9ea-c418f9a8cca8 >> health: HEALTH_OK >> >> services: >> mon: 2 daemons, quorum a001s018,a001s017 (age 18h) >> mgr: a001s016.ctmoay(active, since 109m), standbys: a001s017.bpygfm >> mds: 1/1 daemons up, 2 standby >> osd: 36 osds: 36 up (since 2h), 36 in (since 2y) >> rgw: 3 daemons active (3 hosts, 1 zones) >> >> data: >> volumes: 1/1 healthy >> pools: 43 pools, 1633 pgs >> objects: 51.64M objects, 77 TiB >> usage: 119 TiB used, 132 TiB / 252 TiB avail >> pgs: 1633 active+clean >> >> io: >> client: 356 MiB/s rd, 633 MiB/s wr, 506 op/s rd, 755 op/s wr >> >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (91s) 7s ago 91s 14.5M 2048M 16.2.5 >> 6e73176320aa dcd4705c2069 >> mon.a001s017 a001s017 running >> (18h) 7s ago 21h 978M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 8s ago 2y 1163M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ls | grep mon >> mon 3/3 9s ago 114s >> a001s016;a001s017;a001s018;count:3 >> >> # ceph log last cephadm >> 2024-04-01T18:56:29.631337+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18014760 : cephadm [INF] Schedule restart daemon osd.6 >> 2024-04-01T18:57:05.983577+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18014780 : cephadm [INF] Schedule restart daemon osd.9 >> 2024-04-01T18:57:17.919236+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18014788 : cephadm [INF] Schedule restart daemon osd.12 >> 2024-04-01T18:57:59.823635+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18014811 : cephadm [INF] Schedule restart daemon osd.16 >> 2024-04-01T18:58:26.039310+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18014826 : cephadm [INF] Schedule restart daemon osd.19 >> 2024-04-01T18:58:57.278955+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18014844 : cephadm [INF] Schedule restart daemon osd.21 >> 2024-04-01T18:59:36.656834+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18014867 : cephadm [INF] Schedule restart daemon osd.25 >> 2024-04-01T19:00:03.272925+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18014882 : cephadm [INF] Schedule restart daemon osd.28 >> 2024-04-01T19:00:28.348138+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18014897 : cephadm [INF] Schedule restart daemon osd.31 >> 2024-04-01T19:00:56.297517+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18014913 : cephadm [INF] Schedule restart daemon osd.34 >> 2024-04-01T19:38:21.552347+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18016056 : cephadm [INF] Schedule restart daemon mgr.a001s017.bpygfm >> 2024-04-01T19:38:21.603564+0000 mgr.a001s017.bpygfm (mgr.60881489) >> 18016057 : cephadm [INF] Failing over to other MGR >> 2024-04-01T21:21:24.414308+0000 mgr.a001s016.ctmoay (mgr.61827136) >> 3096 : cephadm [INF] Saving service mon spec with placement >> a001s017;a001s018;count:2 >> 2024-04-01T21:25:02.815151+0000 mgr.a001s016.ctmoay (mgr.61827136) >> 3211 : cephadm [INF] Saving service mon spec with placement >> a001s017;a001s018;count:2 >> 2024-04-01T21:25:02.844721+0000 mgr.a001s016.ctmoay (mgr.61827136) >> 3212 : cephadm [INF] Safe to remove mon.a001s016: not in monmap >> (['a001s018', 'a001s017']) >> 2024-04-01T21:25:02.845059+0000 mgr.a001s016.ctmoay (mgr.61827136) >> 3213 : cephadm [INF] Removing monitor a001s016 from monmap... >> 2024-04-01T21:25:02.846401+0000 mgr.a001s016.ctmoay (mgr.61827136) >> 3214 : cephadm [INF] Removing daemon mon.a001s016 from a001s016 >> 2024-04-01T21:25:06.870895+0000 mgr.a001s016.ctmoay (mgr.61827136) >> 3218 : cephadm [INF] Removing key for mon. >> 2024-04-01T21:26:27.861600+0000 mgr.a001s016.ctmoay (mgr.61827136) >> 3266 : cephadm [INF] Saving service mon spec with placement >> a001s016;a001s017;a001s018;count:3 >> 2024-04-01T21:26:38.247516+0000 mgr.a001s016.ctmoay (mgr.61827136) >> 3273 : cephadm [INF] Saving service mon spec with placement >> a001s016;a001s017;a001s018;count:3 >> 2024-04-01T21:26:44.494080+0000 mgr.a001s016.ctmoay (mgr.61827136) >> 3277 : cephadm [INF] Deploying daemon mon.a001s016 on a001s016 >> >> >> >> # cat >> /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/unit. >> i >> mage >> docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5 >> a >> 2cb7ff7668f776b61b9d586 >> # ceph orch ps --daemon_type=mon >> NAME HOST PORTS STATUS REFRESHED AGE MEM >> USE MEM LIM VERSION IMAGE ID CONTAINER ID >> mon.a001s016 a001s016 running (16m) 3m ago 16m >> 16.3M 2048M 16.2.5 6e73176320aa dcd4705c2069 >> mon.a001s017 a001s017 running (18h) 4m ago 21h >> 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running (5w) 4m ago 2y >> 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 >> >> # ceph orch daemon redeploy mon.a001s016 >> docker.io/ceph/daemon:latest-pacific >> Scheduled to redeploy mon.a001s016 on host 'a001s016' >> >> # ceph orch ps --daemon_type=mon --refresh >> NAME HOST PORTS STATUS REFRESHED AGE MEM >> USE MEM LIM VERSION IMAGE ID CONTAINER ID >> mon.a001s016 a001s016 running (13s) 7s ago 18m >> 14.1M 2048M 16.2.5 6e73176320aa 1917b1edf4ab >> mon.a001s017 a001s017 running (18h) 6m ago 21h >> 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running (5w) 6m ago 2y >> 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 >> >> ===================================================================== >> = >> ======================= >> >> >> The second method also does not update ceph mon map: >> >> #ceph mon dump >> dumped monmap epoch 6 >> epoch 6 >> fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 >> last_changed 2024-03-31T23:54:18.692983+0000 created >> 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) >> election_strategy: 1 >> 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 >> 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 >> >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (81s) 63s ago 81s 14.2M 2048M 16.2.5 >> 6e73176320aa 0c0ec88139e1 >> mon.a001s017 a001s017 running >> (19h) 62s ago 22h 965M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 63s ago 2y 1157M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> >> # ceph orch daemon stop mon.a001s016 Scheduled to stop mon.a001s016 >> on host 'a001s016' >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (96s) 6s ago 96s 14.4M 2048M 16.2.5 >> 6e73176320aa 0c0ec88139e1 >> mon.a001s017 a001s017 running >> (19h) 5s ago 22h 968M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 6s ago 2y 1157M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (98s) 7s ago 98s 14.4M 2048M 16.2.5 >> 6e73176320aa 0c0ec88139e1 >> mon.a001s017 a001s017 running >> (19h) 7s ago 22h 968M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 8s ago 2y 1157M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (100s) 1s ago 100s 14.4M 2048M 16.2.5 >> 6e73176320aa 0c0ec88139e1 >> mon.a001s017 a001s017 running >> (19h) 9s ago 22h 968M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 10s ago 2y 1157M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (106s) 7s ago 106s 14.4M 2048M 16.2.5 >> 6e73176320aa 0c0ec88139e1 >> mon.a001s017 a001s017 running >> (19h) 15s ago 22h 968M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 16s ago 2y 1157M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 stopped >> 0s ago 108s - 2048M <unknown> <unknown> >> <unknown> >> mon.a001s017 a001s017 running >> (19h) 17s ago 22h 968M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 0s ago 2y 1160M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 stopped >> 2s ago 111s - 2048M <unknown> <unknown> >> <unknown> >> mon.a001s017 a001s017 running >> (19h) 2s ago 22h 972M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 3s ago 2y 1160M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> >> # ceph orch daemon rm mon.a001s016 >> Error EINVAL: must pass --force to REMOVE daemon with potentially >> PRECIOUS DATA for mon.a001s016 >> >> # ceph orch daemon rm mon.a001s016 --force Removed mon.a001s016 from >> host 'a001s016' >> >> # ceph orch ps --refresh | grep mon >> mon.a001s017 a001s017 running >> (19h) 30s ago 22h 976M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 0s ago 2y 1166M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> #ceph orch ps --refresh | grep mon >> mon.a001s017 a001s017 running >> (19h) 2s ago 22h 982M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 3s ago 2y 1166M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 starting >> - - - 2048M <unknown> <unknown> >> <unknown> >> mon.a001s017 a001s017 running >> (19h) 6s ago 22h 982M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 7s ago 2y 1166M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> # ceph orch ps --refresh | grep mon >> mon.a001s016 a001s016 running >> (8s) 2s ago 8s 14.4M 2048M 16.2.5 >> 6e73176320aa 39db8cfba7e1 >> mon.a001s017 a001s017 running >> (19h) 1s ago 22h 987M 2048M 16.2.5 >> 6e73176320aa e5e5cb6c256c >> mon.a001s018 a001s018 running >> (5w) 2s ago 2y 1171M 2048M 16.2.5 >> 6e73176320aa 7d2bb6d41f54 >> >> # ceph mon dump >> dumped monmap epoch 6 >> epoch 6 >> fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 >> last_changed 2024-03-31T23:54:18.692983+0000 created >> 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) >> election_strategy: 1 >> 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 >> 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 >> >> # >> >> -----Original Message----- >> From: Adiga, Anantha <anantha.adiga@xxxxxxxxx> >> Sent: Monday, April 1, 2024 2:01 PM >> To: Eugen Block <eblock@xxxxxx> >> Cc: ceph-users@xxxxxxx >> Subject: Re: ceph status not showing correct monitor >> services >> >> Thank you. I will try the export and import method first. >> >> Thank you, >> Anantha >> >> -----Original Message----- >> From: Eugen Block <eblock@xxxxxx> >> Sent: Monday, April 1, 2024 1:57 PM >> To: Adiga, Anantha <anantha.adiga@xxxxxxxxx> >> Cc: ceph-users@xxxxxxx >> Subject: Re: Re: ceph status not showing correct monitor >> services >> >> I have two approaches in mind, first one (and preferred) would be to >> edit the mon spec to first remove mon.a001s016 and have a clean state. >> Get the current spec with: >> >> ceph orch ls mon --export > mon-edit.yaml >> >> Edit the spec file so that mon.a001s016 is not part of it, then apply: >> >> ceph orch apply -i mon-edit.yaml >> >> This should remove the mon.a001s016 daemon. Then wait a few minutes >> or so (until the daemon is actually gone, check locally on the node >> with 'cephadm ls' and in /var/lib/ceph/<FSID>/removed) and add it >> back to the spec file, then apply again. I would expect a third MON >> to be deployed. If that doesn't work for some reason you'll need to >> inspect logs to find the root cause. >> >> The second approach would be to remove and add the daemon manually: >> >> ceph orch daemon rm mon.a001s016 >> >> Wait until it's really gone, then add it: >> >> ceph orch daemon add mon a001s016 >> >> Not entirely sure about the daemon add mon command, you might need to >> provide something else, I'm typing this by heart. >> >> Zitat von "Adiga, Anantha" <anantha.adiga@xxxxxxxxx>: >> >>> Hi Eugen, >>> >>> Yes that is it. OSDs were restarted since mon a001s017 was reporting >>> is low on available space. How to update the mon map to add >>> mon.a001s016 as it is already online? >>> And how to update mgr map to include standby mgr.a001s018 as it is >>> also running. >>> >>> >>> ceph mon dump >>> dumped monmap epoch 6 >>> epoch 6 >>> fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 >>> last_changed 2024-03-31T23:54:18.692983+0000 created >>> 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) >>> election_strategy: 1 >>> 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 >>> 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 >>> >>> >>> Thank you, >>> Anantha >>> >>> -----Original Message----- >>> From: Eugen Block <eblock@xxxxxx> >>> Sent: Monday, April 1, 2024 1:10 PM >>> To: ceph-users@xxxxxxx >>> Subject: Re: ceph status not showing correct monitor >>> services >>> >>> Maybe it’s just not in the monmap? Can you show the output of: >>> >>> ceph mon dump >>> >>> Did you do any maintenance (apparently OSDs restarted recently) and >>> maybe accidentally removed a MON from the monmap? >>> >>> >>> Zitat von "Adiga, Anantha" <anantha.adiga@xxxxxxxxx>: >>> >>>> Hi Anthony, >>>> >>>> Seeing it since last after noon. It is same with mgr services as , >>>> "ceph -s" is reporting only TWO instead of THREE >>>> >>>> Also mon and mgr shows " is_active: false" see below. >>>> >>>> # ceph orch ps --daemon_type=mgr >>>> NAME HOST PORTS STATUS REFRESHED AGE >>>> MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID >>>> mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 3m ago 23M >>>> 206M - 16.2.5 6e73176320aa 169cafcbbb99 >>>> mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 3m ago 23M >>>> 332M - 16.2.5 6e73176320aa 97257195158c >>>> mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 3m ago 23M >>>> 113M - 16.2.5 6e73176320aa 21ba5896cee2 >>>> >>>> # ceph orch ls --service_name=mgr >>>> NAME PORTS RUNNING REFRESHED AGE PLACEMENT >>>> mgr 3/3 3m ago 23M a001s016;a001s017;a001s018;count:3 >>>> >>>> >>>> # ceph orch ps --daemon_type=mon --format=json-pretty >>>> >>>> [ >>>> { >>>> "container_id": "8484a912f96a", >>>> "container_image_digests": [ >>>> >>>> docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586> >>>> ], >>>> "container_image_id": >>>> "6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", >>>> "container_image_name": >>>> docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586>, >>>> "created": "2024-03-31T23:55:16.164155Z", >>>> "daemon_id": "a001s016", >>>> "daemon_type": "mon", >>>> "hostname": "a001s016", >>>> "is_active": false, >>>> <== why is it false >>>> "last_refresh": "2024-04-01T19:38:30.929014Z", >>>> "memory_request": 2147483648, >>>> "memory_usage": 761685606, >>>> "ports": [], >>>> "service_name": "mon", >>>> "started": "2024-03-31T23:55:16.268266Z", >>>> "status": 1, >>>> "status_desc": "running", >>>> "version": "16.2.5" >>>> }, >>>> >>>> >>>> Thank you, >>>> Anantha >>>> >>>> From: Anthony D'Atri <aad@xxxxxxxxxxxxxx> >>>> Sent: Monday, April 1, 2024 12:25 PM >>>> To: Adiga, Anantha <anantha.adiga@xxxxxxxxx> >>>> Cc: ceph-users@xxxxxxx >>>> Subject: Re: ceph status not showing correct monitor >>>> services >>>> >>>> >>>> >>>> >>>> a001s017.bpygfm(active, since 13M), standbys: a001s016.ctmoay >>>> >>>> Looks like you just had an mgr failover? Could be that the >>>> secondary mgr hasn't caught up with current events. >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send >>>> an email to ceph-users-leave@xxxxxxx >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an >>> email to ceph-users-leave@xxxxxxx >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an >> email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |