Re: ceph-create-keys loops

"ST Wong (ITSC)" <ST@xxxxxxxxxxxxxxxx> · Tue, 7 May 2019 10:22:17 +0000

Some test results:
1.If using custom admin_secret for MONs, can see /etc/ceph/ceph.client.admin.keyring which was missing in previous attempts.    
2.stop firewalld on all MONs.

Both attempts gave same result as before: got stuck at collect admin and bootstrap keys as before.

Appreciate any clue.  

Instead of deploying mimic, should we deploy
luminous with latest ansible and ceph-ansible 4.0 or master, then upgrade to mimic ?  

Thanks a lot.

From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx>
On Behalf Of ST Wong (ITSC)

Sent: Tuesday, May 7, 2019 11:48 AM

To: solarflow99 <solarflow99@xxxxxxxxx>

Cc: ceph-users@xxxxxxxxxxxxxx

Subject: Re: [ceph-users] ceph-create-keys loops

Some more information:

Rpm installed on MONS:
python-cephfs-13.2.5-0.el7.x86_64
ceph-base-13.2.5-0.el7.x86_64
libcephfs2-13.2.5-0.el7.x86_64
ceph-common-13.2.5-0.el7.x86_64
ceph-selinux-13.2.5-0.el7.x86_64
ceph-mon-13.2.5-0.el7.x86_64

Doing a mon_status gives following.  Only the local host running the command can see addr/pub_addr correctly while all other MONs have addr/pub_addr “0.0.0.0”. 
 Besides, the mon feature shows it’s luminous while we’re installing mimic.  

======================

[root@cphmon2a ~]# ceph --admin-daemon /var/run/ceph/ceph-mon.cphmon2a.asok  mon_status
{
    "name": "cphmon2a",
    "rank": 0,
    "state": "probing",
    "election_epoch": 0,
    "quorum": [],
    "features": {
        "required_con": "0",
        "required_mon": [],
        "quorum_con": "0",
        "quorum_mon": []
    },
    "outside_quorum": [
        "cphmon2a"
    ],
    "extra_probe_peers": [],
    "sync_provider": [],
    "monmap": {
        "epoch": 0,
        "fsid": "cf746667-3132-476d-a287-19143404cf39",
        "modified": "0.000000",
        "created": "0.000000",
        "features": {
            "persistent": [],
            "optional": []
        },
        "mons": [
            {
                "rank": 0,
                "name": "cphmon2a",
                "addr": "123.123.7.93:6789/0",
                "public_addr": "123.123.7.93:6789/0"
            },
            {
                "rank": 1,
                "name": "cphmon1a",
                "addr": "0.0.0.0:0/1",
                "public_addr": "0.0.0.0:0/1"
            },
            {
                "rank": 2,
                "name": "cphmon3a",
                "addr": "0.0.0.0:0/2",
                "public_addr": "0.0.0.0:0/2"
            },
            {
                "rank": 3,
                "name": "cphmon4b",
                "addr": "0.0.0.0:0/3",
                "public_addr": "0.0.0.0:0/3"
            },
            {
                "rank": 4,
                "name": "cphmon5b",
                "addr": "0.0.0.0:0/4",
                "public_addr": "0.0.0.0:0/4"
            },
            {
                "rank": 5,
                "name": "cphmon6b",
                "addr": "0.0.0.0:0/5",
                "public_addr": "0.0.0.0:0/5"
            }
        ]
    },
    "feature_map": {
        "mon": [
            {
                "features": "0x3ffddff8ffacfffb",
                "release": "luminous",
                "num": 1
            }
        ]
    }
}

======================

Thanks a lot.

From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx>
On Behalf Of ST Wong (ITSC)

Sent: Tuesday, May 7, 2019 7:59 AM

To: solarflow99 <solarflow99@xxxxxxxxx>

Cc: ceph-users@xxxxxxxxxxxxxx

Subject: Re: [ceph-users] ceph-create-keys loops

yes, we’re using 3.2 stable, on RHEL 7.   Thanks.

From: solarflow99 <solarflow99@xxxxxxxxx>

Sent: Tuesday, May 7, 2019 1:40 AM

To: ST Wong (ITSC) <ST@xxxxxxxxxxxxxxxx>

Cc: ceph-users@xxxxxxxxxxxxxx

Subject: Re: [ceph-users] ceph-create-keys loops

you mention the version of ansible, that is right.  How about the branch of ceph-ansible?  should be 3.2-stable, what OS?   I haven't come across this problem myself, a lot of other ones.

On Mon, May 6, 2019 at 3:47 AM ST Wong (ITSC) <ST@xxxxxxxxxxxxxxxx> wrote:

Hi all,

I’ve problem in deploying mimic using ceph-ansible at following step:

---------- cut here -----------
TASK [ceph-mon : collect admin and bootstrap keys] *****************************
Monday 06 May 2019  17:01:23 +0800 (0:00:00.854)       0:05:38.899 ************
fatal: [cphmon3a]: FAILED! => {"changed": false, "cmd": ["ceph-create-keys", "--cluster", "ceph", "-i", "cphmon3a", "-t", "600"], "delta": "0:11:24.675833",
 "end": "2019-05-06 17:12:48.500996", "msg": "non-zero return code", "rc": 1, "start": "2019-05-06 17:01:23.825163", "stderr": "INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\n
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum:
 u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\n
---------- cut here -----------

There are 2 NIC on all MONs.   The site.yml contains some basic configuration:

----------- site.yml --------
dummy:

ceph_origin: repository
ceph_repository: community
ceph_stable_release: mimic
public_network: "123.123.7.0/24"                            
ß fake ip range of our public network
cluster_network: "192.168.77.0/24"
monitor_interface: p2p1
----------- site.yml --------

And the ceph.conf created on MONs :

-------- ceph.conf -------
# Please do not change this file directly since it is managed by Ansible and will be overwritten

[global]
fsid = 17db45c6-b5ac-47e8-b5cb-b3e5215f4af4

mon initial members = cphmon1a,cphmon2a,cphmon3a,cphmon4b,cphmon5b,cphmon6b
osd pool default crush rule = -1

mon host = 123.123.7.92,123.123.7.93,123.123.7.94,123.123.7.95,123.123.7.96,123.123.7.97

public network =
123.123.7.0/24
cluster network =
192.168.77.0/24
-------- ceph.conf -------

Also get same error if running following on one of the MONs:

#ceph-create-keys --cluster ceph --id cphmon1a
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'

We’re using Ansible 2.6 (for deploying Mimic using ceph-ansible).

Would anyone please help?

Thanks a lot.
/st wong

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com