One mon survived (it took us a while to find it since it was in a damaged state), and we have since been able to create a new second mon where an old mon was - quorum has been re-established. We are not able to use `ceph orch` now to deploy new mons though, it is giving us an error from the keyring. The ceph commands were not working at the time of only one damaged mon, but now most ceph commands function normally with 2 mons. On Wed, Aug 21, 2024 at 6:27 AM Eugen Block <eblock@xxxxxx> wrote: > Hi, > > Is there any command-line history available to get at least some sort > of history of events? > Are all MONs down or has one survived? > Could he have tried to change IP addresses or something? There's an > old blog post [0] explaining how to clean up. And here's some more > reading [1] how to modify a monmap in a cephadm managed cluster. > I assume none of the ceph commands work, can you confirm? > > [0] https://ceph.io/en/news/blog/2015/ceph-monitor-troubleshooting/ > [1] > > https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#rados-mon-remove-from-unhealthy > > Zitat von RIT Computer Science House <csh@xxxxxxx>: > > > Hello, > > > > Our cluster has become unresponsive after a teammate's work on the > cluster. > > We are currently unable to get the full story on what he did to fully > > understand what is going on, and the only error we are able to see in any > > of the logs is the following: > > 2024-08-20T03:12:34.183+0000 7f3670246b80 -1 unable to read magic from > mon > > data > > > > Any help would be greatly appreciated! > > I can provide any information necessary to help debugging. > > > > Thanks, > > Tyler > > System Administrator @ Computer Science House > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |