Folks, We’ve finally found the issue: MTU mismatch on the switch-side. So, my colleague noticed “tracepath” from the other monitors to the problematic one does not return and I tracked it down to an MTU mismatch (jumbo vs not) on the switch end. After fixing the mismatch all is back to normal. It turned out to be quite the head scratcher. Thanks to all who’ve offer assistance. -kc > On Oct 29, 2020, at 2:17 AM, Stefan Kooman <stefan@xxxxxx> wrote: > > On 2020-10-29 01:26, Ki Wong wrote: >> Hello, >> >> I am at my wit's end. >> >> So I made a mistake in the configuration of my router and one >> of the monitors (out of 3) dropped out of the quorum and nothing >> I’ve done allow it to rejoin. That includes reinstalling the >> monitor with ceph-ansible. > > What Ceph version? > What kernel version (on the monitors)? > > > Just to check some things: > > make sure the mon-keyring on _all_ monitors is equal and permissions are > correct (ceph can read the file) and read/write to the monstore. > > Have you enabled msgr v1 and v2? > Do you use DNS to detect the monitors [1]. > > ceph daemon mon.$mon$id daemon mon_status <- what does this give on the > out of quorum monitor? > > See the troubleshooting documentation [2] for more information. > > Gr. Stefan > > [1]: https://docs.ceph.com/en/latest/rados/configuration/mon-lookup-dns/ > [2]: > https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/ _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |