Re: Monitor rename / recreate issue -- probing state

deeepdish <deeepdish@xxxxxxxxx> · Sun, 13 Dec 2015 19:41:50 -0500

Perhaps I’m not understanding something..
The “extra_probe_peers” ARE the other working monitors in quorum out of the mon_host line in ceph.conf.

In the example below 10.20.1.8 = b20s08; 10.20.10.251 = smon01s; 10.20.10.252 = smon02s

The monitor is not reaching out to the other IPs and syncing.   I’m able to ping all IPs in the extra_probe_peers list.

# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.smg01.asok mon_status
{
    "name": "smg01",
    "rank": 0,
    "state": "probing",
    "election_epoch": 0,
    "quorum": [],
    "outside_quorum": [
        "smg01"
    ],
    "extra_probe_peers": [
        "10.20.1.8:6789\/0",
        "10.20.10.251:6789\/0",
        "10.20.10.252:6789\/0"
    ],
    "sync_provider": [],
    "monmap": {
        "epoch": 0,
        "fsid": "693834c1-1f95-4237-ab97-a767b0c0e6e7",
        "modified": "0.000000",
        "created": "0.000000",
        "mons": [
            {
                "rank": 0,
                "name": "smg01",
                "addr": "10.20.10.250:6789\/0"
            },
            {
                "rank": 1,
                "name": "smon01s",
                "addr": "0.0.0.0:0\/1"
            },
            {
                "rank": 2,
                "name": "smon02s",
                "addr": "0.0.0.0:0\/2"
            },
            {
                "rank": 3,
                "name": "b02s08",
                "addr": "0.0.0.0:0\/3"
            }
        ]
    }
}

On Dec 13, 2015, at 19:18 , Joao Eduardo Luis <joao@xxxxxxx> wrote:

On 12/13/2015 12:26 PM, deeepdish wrote:

This appears to be consistent with a wrongly populated 'mon_host' and
'mon_initial_members' in your ceph.conf.

 -Joao

Thanks Joao.   I had a look but my other 3 monitors are working just
fine.   To be clear, I’ve confirmed the same behaviour on other monitor
nodes that have been removed from the cluster and rebuild with a new IP
(however same name).

I'm not entirely sure what you mean, but let me clarify what I meant a bit.

Existing monitors take their monmap from their own stores. All monitors
in a quorum will see the same monmap. Existing monitors do not care
about the configuration file for their monmap.

'mon_host' and 'mon_initial_members' are only used by clients trying to
reach the monitors AND when creating a new monitor.

Therefore, when creating a new monitor, 'mon_host' must contain the ips
of the existing monitors PLUS the monitor you are creating, and
'mon_initial_members' must contain the hosts of the existing monitors
PLUS the host of the monitor you are creating.

Your initial email reflected a lot of other ips on the
'extra_probe_peers' (which is basically the contents of mon_host during
the probing phase, while the monitor tries to find the other monitors),
which is consistent with mon_host being wrongly populated.

  -Joao

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com