Re: Ceph orch command hangs forever

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thank you so much Boris for replying.

We have three mons.
Two mons are still in quorum. One mon is out of quorum.

But mon who down sees all of three mons out of quorum.
from 2 up mon nodes ceph -s result is
mon: 3 daemons, quorum ceph1,compute1 (age 14h), out of quorum: compute2
from down mon (ceph -s)
mon: 3 daemons, quorum  (age 32h), out of quorum: compute2, ceph1, compute1

ceph -m mon1 status   -> works
ceph -m mon2 status   -> works
ceph -m mon3 status   -> not answering

and all mons are deployed in containers

what should I do now?

Best regards,

Boloo







On Mon, Jan 10, 2022 at 8:13 PM Boris Behrens <bb@xxxxxxxxx> wrote:

> Hi Boldayer,
> I had a similar issue with radosgw-admin sync status.
>
> It was actually a problem with the mons that were not listening on the
> correct IP addresses.
> You can check with `ceph mon stat` if the mon got the correct IP addresses.
> With `ceph -m IPADDRESS status` you can check if the mons are answering.
> (IPADDRESS is the actual IP of one of you mons)
>
> After finding what the problem was, I readded the mons 1by1 with this:
>
> If your mon had the wrong IP address once, it will try to stick to it
> forever.
>
> To assign a specific IP address to a mon follow these steps:
>
>    1. Remove the mon:
>
> # assuming there is only one mon and you are connected to the host that got the mon
> systemctl stop ceph-mon.target
> ceph mon rm `hostname -s`
> rm -rf /var/lib/ceph/mon/ceph-*
>
>
>    1. fixate ip address in ceph.conf
>
> [mon.`hostname -s`]
> public_addr = IP Address
> public_bind_addr = IP Address
>
>
>    1. readd the host to the monmap
>
> ceph mon getmap -o /tmp/monmap
> sudo -u ceph ceph-mon --cluster ceph --mkfs -i `hostname -s` --monmap /tmp/monmap
> systemctl restart ceph-mon.target
> rm /tmp/monmap
>
>
>    1. cleanup ceph.conf
>
>
> Maybe it helps.
> Cheers
>  Boris
>
> Am Mo., 10. Jan. 2022 um 12:43 Uhr schrieb Boldbayar Jantsan <
> netware.bb@xxxxxxxxx>:
>
>> Hello,
>>
>> I am using Octopus version which was deployed by Cephadm.
>> After one of mon nodes rebooted, ceph orch command does not work and is
>> not
>> responsive.
>> It looks very similar to the below issue.
>>
>> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/BCINR25YUKC6CWZGRZWEG5ULB34O5FSA/#AUZSLSRJJUE5CRFPWFOBMUN4IWSGTIFE
>>
>> Is there anyone who faced this issue and solve this problem?
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux