(adding ceph-users instead)
On 19/06/07 12:53pm, Lluis Arasanz i Nonell - Adam wrote:
> Hi all,
>
> I know I have a very old ceph version, but I need some help.
> Also, understand that English is not my native language, so please, take it in mind if something is not really well explained.
>
> About infra:
>
> - Ceph version 0.87.2 (Giant)
>
> - 5 OSD Servers (127 TB) and 3 Monitors
>
> - Centos 7 based
>
> We had a power supply problem that make one monitor down. (Power off). After power restart, this monitor start ceph-mon, but does not contact the other 2 monitors. Ceph-mon process logs version and nothing else, but start a memory compsumption until fill all and kernel stops process.
>
> After some tries, we "thought" to deploy a new monitor (mon04) and.... This was the start of our problems :(
>
> 4 monitors, 2 up, 1 down and one new but not in ring... so this is NOT quorum then I lost monitor access (ceph related command does not work)
>
> Then, suddenly, mon02 began to start ceph-mon, but "ceph-create-keys" module does not contact with socket. Of course, socket files is created, but ceph daemon mon.mon02 commands, not shows "mon_status", "quorum_status", and some more (only shows 13 from 19 that shows mon01).
>
> I have:
>
> Monmap (with 4 monitors, mon01 to mon04)
> Monitors keyring
>
> So I deploy a MON05 from scratch, make new filesystem and inject monmap and keyring...
>
> Mon_status on 3 monitors show:
>
> Mon01:
> [root@mon01 ~]# ceph daemon mon.mon01 mon_status
> { "name": "mon01",
> "rank": 0,
> "state": "electing",
> "election_epoch": 447,
> "quorum": [],
> "outside_quorum": [],
> "extra_probe_peers": [
> "10.10.200.21:6789\/0",
> "10.10.200.24:6789\/0"],
> "sync_provider": [],
> "monmap": { "epoch": 20,
> "fsid": "1aa318df-c6eb-47c5-a80e-2e9e43160e4e",
> "modified": "2019-06-06 16:45:47.489558",
> "created": "0.000000",
> "mons": [
> { "rank": 0,
> "name": "mon01",
> "addr": "10.10.200.20:6789\/0"},
> { "rank": 1,
> "name": "mon02",
> "addr": "10.10.200.21:6789\/0"},
> { "rank": 2,
> "name": "mon03",
> "addr": "10.10.200.22:6789\/0"},
> { "rank": 3,
> "name": "mon04",
> "addr": "10.10.200.23:6789\/0"}]}}
>
> Mon04:
> [root@mon04 ~]# ceph daemon mon.mon04 mon_status
> { "name": "mon04",
> "rank": 3,
> "state": "probing",
> "election_epoch": 0,
> "quorum": [],
> "outside_quorum": [
> "mon01",
> "mon04"],
> "extra_probe_peers": [
> "10.10.200.21:6789\/0",
> "10.10.200.22:6789\/0",
> "10.10.200.23:6789\/0",
> "10.10.200.24:6789\/0"],
> "sync_provider": [],
> "monmap": { "epoch": 20,
> "fsid": "1aa318df-c6eb-47c5-a80e-2e9e43160e4e",
> "modified": "2019-06-06 16:45:47.489558",
> "created": "0.000000",
> "mons": [
> { "rank": 0,
> "name": "mon01",
> "addr": "10.10.200.20:6789\/0"},
> { "rank": 1,
> "name": "mon02",
> "addr": "10.10.200.21:6789\/0"},
> { "rank": 2,
> "name": "mon03",
> "addr": "10.10.200.22:6789\/0"},
> { "rank": 3,
> "name": "mon04",
> "addr": "10.10.200.23:6789\/0"}]}}
>
> Mon05:
> [root@mon05 ~]# ceph daemon mon.mon05 mon_status
> { "name": "mon05",
> "rank": -1,
> "state": "probing",
> "election_epoch": 0,
> "quorum": [],
> "outside_quorum": [
> "mon01",
> "mon04"],
> "extra_probe_peers": [
> "10.10.200.21:6789\/0",
> "10.10.200.22:6789\/0",
> "10.10.200.23:6789\/0"],
> "sync_provider": [],
> "monmap": { "epoch": 20,
> "fsid": "1aa318df-c6eb-47c5-a80e-2e9e43160e4e",
> "modified": "2019-06-06 16:45:47.489558",
> "created": "0.000000",
> "mons": [
> { "rank": 0,
> "name": "mon01",
> "addr": "10.10.200.20:6789\/0"},
> { "rank": 1,
> "name": "mon02",
> "addr": "10.10.200.21:6789\/0"},
> { "rank": 2,
> "name": "mon03",
> "addr": "10.10.200.22:6789\/0"},
> { "rank": 3,
> "name": "mon04",
> "addr": "10.10.200.23:6789\/0"}]}}
>
>
> Mon02 and mon03, does not start ceph-mon.....
Why are mon02 and mon03 not starting? That would be something important to
know. You should be able to find that in the logs.
> How I tell mon01 and mon04 that mon05 is a new partner in cluster?
If, instead of injecting a monmap, you just specify the fsid and mon_host, I
think mon05 will find its way into the quorum. I think that will address the
issue.
If not, then you can always extract the monmap from the existing monitors, add
mon05 to it, and inject it into all the monitors. That ought to force the
other monitors to be aware of mon05.
-Joao
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
