Re: how to remove and re-add broken mon? SOLVED

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




On Wed, Sep 4, 2013 at 11:21 AM, Bernhard Glomm <bernhard.glomm@xxxxxxxxxxx> wrote:
Removing a monitor, also like described in
http://ceph.com/docs/master/rados/deployment/ceph-deploy-mon/
doesn't work

----------------
root@nuke36[/1]:/etc/ceph # ceph-deploy mon destroy nuke36
[ceph_deploy.mon][DEBUG ] Removing mon from nuke36
[ceph_deploy.mon][ERROR ] ceph-mon deamon did not stop
[ceph_deploy][ERROR ] GenericError: Failed to destroy 1 monitors
----------------

is this, I can't destroy the mon on which I run the destroy command?
(I could destroy
but since I just added another two monitors to my cluster,
the broken first one is alive again...

What OS is the nuke36 host?

There are a bunch of commands that `mon destroy` will call on the remote host, but unfortunately, this is one of the commands I have not fixed yet to use the new remote logging to tell you.

Can you try to manually run this on nuke36 and see what is going on?

    sudo ceph --cluster={cluster name, defaults to ceph} -n mon. -k {path to keyring}/keyring mon remove

well that should do as a work around for now
(since I'm using the dev branch and do only testing stuff)
still, would be curious if I could have repaired it any other way.

Bernhard

Am 04.09.2013 12:40:04, schrieb Bernhard Glomm:
Hi all,

after some days of successful creating and destroying rbd's, snapshots,
clones and migrating formats all of a sudden one of the monitors doesn't work anymore.
I tried to remove and re-add the monitor from the cluster, but that doesn't seem to work either.

Here's the situation:
I liked to use ceph-deploy to initiate the cluster.
Due to the broken ceph-create-keys in dumpling I turned to the gitbuilder version
now running

ceph version 0.67.2-23-g24f2669 (24f2669783e2eb9d9af5ecbe106efed93366ba63)
on uptodate raring systems
All of a sudden the host from which I ran ceph-deploy, and which should be one of the
5 monitors (from which 2 are also serving as OSDs) has fallen out of the quorum as you can see here
(yes, time is in sync on all nodes):

------------------
root@nuke36[/0]:~ # ceph -s
2013-09-04 11:09:56.039547 7f7a8820f700  1 -- :/0 messenger.start
2013-09-04 11:09:56.040646 7f7a8820f700  1 -- :/1016260 --> 192.168.242.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0 0x7f7a8000e8f0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.041304 7f7a84a08700  1 -- 192.168.242.36:0/1016260 learned my addr 192.168.242.36:0/1016260
2013-09-04 11:09:56.042843 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 1 ==== mon_map v1 ==== 776+0+0 (2241333437 0 0) 0x7f7a70000c30 con 0x7f7a8000e4e0
2013-09-04 11:09:56.043038 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (2063715990 0 0) 0x7f7a70001060 con 0x7f7a8000e4e0
2013-09-04 11:09:56.043324 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 --> 192.168.242.92:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7f7a74001af0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.044197 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (3910749728 0 0) 0x7f7a70001060 con 0x7f7a8000e4e0
2013-09-04 11:09:56.044375 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 --> 192.168.242.92:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x7f7a740020d0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.045376 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (3802320753 0 0) 0x7f7a700008f0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.045457 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 --> 192.168.242.92:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f7a8000ed80 con 0x7f7a8000e4e0
2013-09-04 11:09:56.045550 7f7a8820f700  1 -- 192.168.242.36:0/1016260 --> 192.168.242.92:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x7f7a800079f0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.045559 7f7a8820f700  1 -- 192.168.242.36:0/1016260 --> 192.168.242.92:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x7f7a8000fa10 con 0x7f7a8000e4e0
2013-09-04 11:09:56.046376 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 5 ==== mon_map v1 ==== 776+0+0 (2241333437 0 0) 0x7f7a70001290 con 0x7f7a8000e4e0
2013-09-04 11:09:56.046417 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1524320885 0 0) 0x7f7a70001480 con 0x7f7a8000e4e0
2013-09-04 11:09:56.046429 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 7 ==== osd_map(22..22 src has 1..22) v3 ==== 2355+0+0 (11792226 0 0) 0x7f7a70001f70 con 0x7f7a8000e4e0
2013-09-04 11:09:56.046828 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1524320885 0 0) 0x7f7a700008f0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.046948 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 9 ==== osd_map(22..22 src has 1..22) v3 ==== 2355+0+0 (11792226 0 0) 0x7f7a700008c0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.047071 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 10 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1524320885 0 0) 0x7f7a70000df0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.047547 7f7a8820f700  1 -- 192.168.242.36:0/1016260 --> 192.168.242.92:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- ?+0 0x7f7a8000b0f0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.050938 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 11 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) v1 ==== 72+0+24040 (1092875540 0 2922658865) 0x7f7a700008c0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.089981 7f7a8820f700  1 -- 192.168.242.36:0/1016260 --> 192.168.242.92:6789/0 -- mon_command({"prefix": "status"} v 0) v1 -- ?+0 0x7f7a8000b0d0 con 0x7f7a8000e4e0
2013-09-04 11:09:56.091348 7f7a86a0c700  1 -- 192.168.242.36:0/1016260 <== mon.3 192.168.242.92:6789/0 12 ==== mon_command_ack([{"prefix": "status"}]=0  v0) v1 ==== 54+0+558 (1155462804 0 1174924833) 0x7f7a70000db0 con 0x7f7a8000e4e0
  cluster b085fba3-8e17-443c-bb61-7758504538f8
   health HEALTH_WARN 1 mons down, quorum 0,1,3,4 atom01,atom02,ping,pong
   monmap e1: 5 mons at {atom01=192.168.242.31:6789/0,atom02=192.168.242.32:6789/0,nuke36=192.168.242.36:6789/0,ping=192.168.242.92:6789/0,pong=192.168.242.93:6789/0}, election epoch 26, quorum 0,1,3,4 atom01,atom02,ping,pong
   osdmap e22: 2 osds: 2 up, 2 in
    pgmap v46761: 1192 pgs: 1192 active+clean; 7806 MB data, 20618 MB used, 3702 GB / 3722 GB avail; 9756B/s wr, 0op/s
   mdsmap e17: 1/1/1 up {0=pong=up:active}, 1 up:standby

2013-09-04 11:09:56.096071 7f7a8820f700  1 -- 192.168.242.36:0/1016260 mark_down 0x7f7a8000e4e0 -- 0x7f7a8000e280
2013-09-04 11:09:56.096516 7f7a8820f700  1 -- 192.168.242.36:0/1016260 mark_down_all
2013-09-04 11:09:56.097065 7f7a8820f700  1 -- 192.168.242.36:0/1016260 shutdown complete.
------------------

this is the ceph.conf that was generated during ceph-deploy (well I added the debug lines obviously)

------------------
root@nuke36[/0]:~ # cat /etc/ceph/ceph.conf
[global]
fsid = b085fba3-8e17-443c-bb61-7758504538f8
mon_initial_members = ping, pong, nuke36, atom01, atom02
mon_host = 192.168.242.92,192.168.242.93,192.168.242.36,192.168.242.31,192.168.242.32
auth_supported = cephx
osd_journal_size = 1024
filestore_xattr_use_omap = true
debug ms = 1
debug mon = 20
------------------

after a reboot of the node I now find

------------------
root@nuke36[/1]:~ # ps ax | egrep ceph
  855 ?        Ssl    0:00 /usr/bin/ceph-mon --cluster=ceph -i nuke36 -f
  856 ?        Ss     0:00 /usr/bin/python /usr/sbin/ceph-create-keys --cluster=ceph -i nuke36
 1813 pts/1    R+     0:00 egrep --color=auto ceph
------------------

with ceph-create-keys runing infinetly,
while the key files but already exist

------------------
root@nuke36[/1]:~ # ls -lh /etc/ceph/
total 88K
-rw-r--r-- 1 root root  72 Aug 30 15:54 ceph.bootstrap-mds.keyring
-rw-r--r-- 1 root root  72 Aug 30 15:54 ceph.bootstrap-osd.keyring
-rw------- 1 root root  64 Aug 30 15:54 ceph.client.admin.keyring
-rw-r--r-- 1 root root 303 Sep  4 10:19 ceph.conf
-rw-r--r-- 1 root root 59K Sep  3 10:15 ceph.log
-rw-r--r-- 1 root root  73 Aug 30 15:53 ceph.mon.keyring
-rw-r--r-- 1 root root  92 Aug 30 00:03 rbdmap
------------------

and

------------------
root@nuke36[/1]:~ # tree -pfugiAD /var/lib/ceph/
/var/lib/ceph
[drwxr-xr-x root     root     Aug 30 15:54]  /var/lib/ceph/bootstrap-mds
[-rw------- root     root     Aug 30 15:54]  /var/lib/ceph/bootstrap-mds/ceph.keyring
[drwxr-xr-x root     root     Aug 30 15:54]  /var/lib/ceph/bootstrap-osd
[-rw------- root     root     Aug 30 15:54]  /var/lib/ceph/bootstrap-osd/ceph.keyring
[drwxr-xr-x root     root     Aug 30  0:00]  /var/lib/ceph/mds
[drwxr-xr-x root     root     Aug 30 15:53]  /var/lib/ceph/mon
[drwxr-xr-x root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-nuke36
[-rw-r--r-- root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-nuke36/done
[-rw-r--r-- root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-nuke36/keyring
[drwxr-xr-x root     root     Sep  4 11:16]  /var/lib/ceph/mon/ceph-nuke36/store.db
[-rw-r--r-- root     root     Sep  3  6:21]  /var/lib/ceph/mon/ceph-nuke36/store.db/007726.sst
[-rw-r--r-- root     root     Sep  3  6:21]  /var/lib/ceph/mon/ceph-nuke36/store.db/007727.sst
[-rw-r--r-- root     root     Sep  3  6:21]  /var/lib/ceph/mon/ceph-nuke36/store.db/007728.sst
[-rw-r--r-- root     root     Sep  3  6:21]  /var/lib/ceph/mon/ceph-nuke36/store.db/007729.sst
[-rw-r--r-- root     root     Sep  3  6:21]  /var/lib/ceph/mon/ceph-nuke36/store.db/007730.sst
[-rw-r--r-- root     root     Sep  4 10:14]  /var/lib/ceph/mon/ceph-nuke36/store.db/007767.sst
[-rw-r--r-- root     root     Sep  4 10:14]  /var/lib/ceph/mon/ceph-nuke36/store.db/007768.sst
[-rw-r--r-- root     root     Sep  4 10:14]  /var/lib/ceph/mon/ceph-nuke36/store.db/007769.sst
[-rw-r--r-- root     root     Sep  4 10:14]  /var/lib/ceph/mon/ceph-nuke36/store.db/007770.sst
[-rw-r--r-- root     root     Sep  4 10:14]  /var/lib/ceph/mon/ceph-nuke36/store.db/007772.sst
[-rw-r--r-- root     root     Sep  4 10:14]  /var/lib/ceph/mon/ceph-nuke36/store.db/007773.sst
[-rw-r--r-- root     root     Sep  4 10:14]  /var/lib/ceph/mon/ceph-nuke36/store.db/007774.sst
[-rw-r--r-- root     root     Sep  4 10:14]  /var/lib/ceph/mon/ceph-nuke36/store.db/007775.sst
[-rw-r--r-- root     root     Sep  4 10:18]  /var/lib/ceph/mon/ceph-nuke36/store.db/007777.sst
[-rw-r--r-- root     root     Sep  4 10:19]  /var/lib/ceph/mon/ceph-nuke36/store.db/007780.sst
[-rw-r--r-- root     root     Sep  4 11:16]  /var/lib/ceph/mon/ceph-nuke36/store.db/007783.sst
[-rw-r--r-- root     root     Sep  4 11:16]  /var/lib/ceph/mon/ceph-nuke36/store.db/007784.log
[-rw-r--r-- root     root     Sep  4 11:16]  /var/lib/ceph/mon/ceph-nuke36/store.db/CURRENT
[-rw-r--r-- root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-nuke36/store.db/LOCK
[-rw-r--r-- root     root     Sep  4 11:16]  /var/lib/ceph/mon/ceph-nuke36/store.db/LOG
[-rw-r--r-- root     root     Sep  4 10:19]  /var/lib/ceph/mon/ceph-nuke36/store.db/LOG.old
[-rw-r--r-- root     root     Sep  4 11:16]  /var/lib/ceph/mon/ceph-nuke36/store.db/MANIFEST-007782
[-rw-r--r-- root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-nuke36/upstart
[drwxr-xr-x root     root     Aug 30  0:00]  /var/lib/ceph/osd
[drwxr-xr-x root     root     Aug 30 15:53]  /var/lib/ceph/tmp
------------------

to compare with atom01 which is still running in the cluster...

------------------
root@atom01[/0]:~ # tree -pfugiAD /var/lib/ceph/
/var/lib/ceph
[drwxr-xr-x root     root     Aug 30 15:54]  /var/lib/ceph/bootstrap-mds
[-rw------- root     root     Aug 30 15:54]  /var/lib/ceph/bootstrap-mds/ceph.keyring
[drwxr-xr-x root     root     Aug 30 15:54]  /var/lib/ceph/bootstrap-osd
[-rw------- root     root     Aug 30 15:54]  /var/lib/ceph/bootstrap-osd/ceph.keyring
[drwxr-xr-x root     root     Aug 30  0:00]  /var/lib/ceph/mds
[drwxr-xr-x root     root     Aug 30 15:53]  /var/lib/ceph/mon
[drwxr-xr-x root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-atom01
[-rw-r--r-- root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-atom01/done
[-rw-r--r-- root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-atom01/keyring
[drwxr-xr-x root     root     Sep  4 11:18]  /var/lib/ceph/mon/ceph-atom01/store.db
[-rw-r--r-- root     root     Sep  4 11:25]  /var/lib/ceph/mon/ceph-atom01/store.db/006339.log
[-rw-r--r-- root     root     Sep  4 11:18]  /var/lib/ceph/mon/ceph-atom01/store.db/006342.sst
[-rw-r--r-- root     root     Sep  4 11:18]  /var/lib/ceph/mon/ceph-atom01/store.db/006343.sst
[-rw-r--r-- root     root     Sep  4 11:18]  /var/lib/ceph/mon/ceph-atom01/store.db/006344.sst
[-rw-r--r-- root     root     Sep  4 11:18]  /var/lib/ceph/mon/ceph-atom01/store.db/006345.sst
[-rw-r--r-- root     root     Sep  4 11:18]  /var/lib/ceph/mon/ceph-atom01/store.db/006346.sst
[-rw-r--r-- root     root     Sep  4 11:18]  /var/lib/ceph/mon/ceph-atom01/store.db/006347.sst
[-rw-r--r-- root     root     Sep  4 11:18]  /var/lib/ceph/mon/ceph-atom01/store.db/006348.sst
[-rw-r--r-- root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-atom01/store.db/CURRENT
[-rw-r--r-- root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-atom01/store.db/LOCK
[-rw-r--r-- root     root     Sep  4 11:18]  /var/lib/ceph/mon/ceph-atom01/store.db/LOG
[-rw-r--r-- root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-atom01/store.db/LOG.old
[-rw-r--r-- root     root     Sep  4 11:18]  /var/lib/ceph/mon/ceph-atom01/store.db/MANIFEST-000004
[-rw-r--r-- root     root     Aug 30 15:53]  /var/lib/ceph/mon/ceph-atom01/upstart
[drwxr-xr-x root     root     Aug 30  0:00]  /var/lib/ceph/osd
[drwxr-xr-x root     root     Aug 30 15:53]  /var/lib/ceph/tmp
------------------

the problem with ceph-create-keys seemed to have been fixed with wip-4924
since in that version "rbd create" wouldn't work I switched to
deb http://gitbuilder.ceph.com/ceph-deb-raring-x86_64-basic/ref/dumpling/       raring main
(runing on an uptodate raring)
I thought of just removing and than re-adding the failing mon but who do I do that?
The documentation on
http://ceph.com/docs/master/rados/operations/add-or-rm-mons/
says:
"service ceph -a stop mon.{mon-id}"

------------------
root@nuke36[/1]:~ # service ceph -a stop mon.nuke36
/etc/init.d/ceph: mon.nuke36 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph defines )
root@nuke36[/1]:~ # service ceph -a stop mon.c
/etc/init.d/ceph: mon.c not found (/etc/ceph/ceph.conf defines , /var/lib/ceph defines )
root@nuke36[/1]:~ # service ceph -a stop mon.2
/etc/init.d/ceph: mon.2 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph defines )
root@nuke36[/1]:~ # service ceph -a stop mon.ceph-nuke36
/etc/init.d/ceph: mon.ceph-nuke36 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph defines )
------------------

that doesn't help, manually stopping the daemon doesn't work either (respawning, okay)
but this combination leaves me quite curious

------------------
root@atom01[/0]:~ # ceph health detail
HEALTH_WARN 1 mons down, quorum 0,1,3,4 atom01,atom02,ping,pong
mon.nuke36 (rank 2) addr 192.168.242.36:6789/0 is down (out of quorum)
root@atom01[/0]:~ # service ceph -a stop mon.nuke36
/etc/init.d/ceph: mon.nuke36 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph defines )
/etc/init.d/ceph: mon.2 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph defines )
root@atom01[/0]:~ # service ceph -a stop mon.rank2
------------------

something seems to be out of sync, at least with the documentation?
any hint how to proceed from here?

TIA

Bernhard


--

Bernhard Glomm
IT Administration
Phone: +49 (30) 86880 134 Fax: +49 (30) 86880 100 Skype: bernhard.glomm.ecologic Ecologic Institut gemeinnützige GmbH | Pfalzburger Str. 43/44 | 10717 Berlin | Germany
GF: R. Andreas Kraemer | AG: Charlottenburg HRB 57947 | USt/VAT-IdNr.: DE811963464
Ecologic™ is a Trade Mark (TM) of Ecologic Institut gemeinnützige GmbH
root@nuke
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
*Ecologic Institute* Bernhard Glomm
IT Administration
Phone: +49 (30) 86880 134
Fax: +49 (30) 86880 100
Skype: bernhard.glomm.ecologic
Website: | Video: | Newsletter: | Facebook: | Linkedin: | Twitter: | YouTube: | Google+:
Ecologic Institut gemeinnützige GmbH | Pfalzburger Str. 43/44 | 10717 Berlin | Germany
GF: R. Andreas Kraemer | AG: Charlottenburg HRB 57947 | USt/VAT-IdNr.: DE811963464
Ecologic™ is a Trade Mark (TM) of Ecologic Institut gemeinnützige GmbH

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux