Re: Issue with "renamed" mon, crashing

[Kamila Součková <kamila@xxxxxx>]

Hi,

I am not sure if this is the same issue as we had recently, but it looks a bit like it -- we also had a Luminous mon crashing right after syncing was done.

Turns out that the current release has a bug which causes the mon to crash if it cannot find a mgr daemon. This should be fixed in the upcoming release.

In our case we "solved" it by moving the active mgr to the mon's host. (I am not sure how to activate a specific mgr, but it appears that the mgrs get activated in FIFO order -- so just keep killing and re-starting the active one until a mgr on the mon's host is active).

Hope this helps!

Kamila

On Mon, Nov 6, 2017 at 12:44 PM Anders Olausson <anders@xxxxxxxxxxxx> wrote:

Hi,

 

I recently (yesterday) upgraded to Luminous (12.2.1) running on Ubuntu 14.04.5 LTS.

Upgrade went fine, no issues at all.

However when I was about to use ceph-deploy to configure some new disks it failed.

After some investigation I figured out that it didn’t like that my mons was named ceph03mon on the host ceph03 for example, ceph-deploy gatherkeys ceph03 failed.

So I decided to rename my mons. I started with removing one of them:

 

# stop ceph-mon id=ceph03mon

# ceph mon remove ceph03mon

# cd /var/lib/ceph/mon/

# mv ceph-ceph03mon disabled-ceph-ceph03mon

 

Created the new one:

 

# mkdir tmp

# mkdir ceph-ceph03

# ceph auth get mon. -o tmp/keyring

# ceph mon getmap -o tmp/monmap

# ceph-mon -i ceph03 --mkfs --monmap tmp/monmap --keyring tmp/keyring

# chown -R ceph:ceph ceph-ceph03

# ceph-mon -i ceph03 --public-addr 10.10.1.23:6789

# start ceph-mon id=ceph03

 

Starts OK, quorum is established, when it gets the command “ceph osd pool stat” for example, or “ceph auth list” it crashes.

 

Complete log can be found at: http://files.spacedump.se/ceph03-monerror-20171106-01.txt

Used below settings for logging in ceph.conf at the time:

 

[mon]

       debug mon = 20

       debug paxos = 20

       debug auth = 20

 

I have now rolled back to the old monitor, it works as it should, on the same box etc. But it’s the one upgraded from Hammer -> Jewel -> Luminous.

 

Any idea what the issue could be?

Thanks.

 

Best regards

  Anders Olausson

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com