Alright, quorum is good. Do you have mgr keyrings in 'ceph auth ls'
output? If they are present, is there a keyring file in
/var/lib/ceph/{fsid}/mgr.{mgr}/ and does it match the auth output?
Zitat von RIT Computer Science House <csh@xxxxxxx>:
One mon survived (it took us a while to find it since it was in a damaged
state), and we have since been able to create a new second mon where an old
mon was - quorum has been re-established. We are not able to use `ceph
orch` now to deploy new mons though, it is giving us an error from the
keyring.
The ceph commands were not working at the time of only one damaged mon, but
now most ceph commands function normally with 2 mons.
On Wed, Aug 21, 2024 at 6:27 AM Eugen Block <eblock@xxxxxx> wrote:
Hi,
Is there any command-line history available to get at least some sort
of history of events?
Are all MONs down or has one survived?
Could he have tried to change IP addresses or something? There's an
old blog post [0] explaining how to clean up. And here's some more
reading [1] how to modify a monmap in a cephadm managed cluster.
I assume none of the ceph commands work, can you confirm?
[0] https://ceph.io/en/news/blog/2015/ceph-monitor-troubleshooting/
[1]
https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#rados-mon-remove-from-unhealthy
Zitat von RIT Computer Science House <csh@xxxxxxx>:
> Hello,
>
> Our cluster has become unresponsive after a teammate's work on the
cluster.
> We are currently unable to get the full story on what he did to fully
> understand what is going on, and the only error we are able to see in any
> of the logs is the following:
> 2024-08-20T03:12:34.183+0000 7f3670246b80 -1 unable to read magic from
mon
> data
>
> Any help would be greatly appreciated!
> I can provide any information necessary to help debugging.
>
> Thanks,
> Tyler
> System Administrator @ Computer Science House
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
