Re: Luminous: ceph mgr crate error - mon disconnected

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Thanks a lot for your comments. 

In the end, I have reinstalled the system. 

Now, it is working perfectly but mgr dashboard looks having lot of bugs... 

El 23 jul. 2017 5:10 p. m., "Roger Brown" <rogerpbrown@xxxxxxxxx> escribió:
Mark,

Thanks for that information. I can't seem to deploy ceph-mgr either. I also have the busted mgr bootstrap key. I attempted the suggested fix, but my issue may be different somehow. Complete output follows.
-Roger

roger@desktop:~$ ceph-deploy --version
1.5.38
roger@desktop:~$ ceph mon versions
{
    "ceph version 12.1.1 (f3e663a190bf2ed12c7e3cda288b9a159572c800) luminous (rc)": 3
}
roger@desktop:~/ceph-cluster$ sudo ceph auth get client.bootstrap-mgr
exported keyring for client.bootstrap-mgr
[client.bootstrap-mgr]
key = AAAAAAAAAAAAAAAA
caps mon = "allow profile bootstrap-mgr"
roger@desktop:~/ceph-cluster$ sudo ceph auth del client.bootstrap-mgr
updated
roger@desktop:~/ceph-cluster$ sudo ceph auth get client.bootstrap-mgr
Error ENOENT: failed to find client.bootstrap-mgr in keyring
roger@desktop:~/ceph-cluster$ sudo ceph auth get-or-create client.bootstrap-mgr mon 'allow profile bootstrap-mgr'
[client.bootstrap-mgr]
key = AQBDt3RZPXdpNBAAekynuNJpVPaN1B4YTeFu4w==
roger@desktop:~/ceph-cluster$ ceph-deploy -v gatherkeys nuc1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/roger/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.38): /usr/bin/ceph-deploy -v gatherkeys nuc1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : True
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4ec6dd2bd8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  mon                           : ['nuc1']
[ceph_deploy.cli][INFO  ]  func                          : <function gatherkeys at 0x7f4ec6da1050>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.gatherkeys][INFO  ] Storing keys in temp directory /tmp/tmpdhkYYM
[nuc1][DEBUG ] connection detected need for sudo
[nuc1][DEBUG ] connected to host: nuc1 
[nuc1][DEBUG ] detect platform information from remote host
[nuc1][DEBUG ] detect machine type
[nuc1][DEBUG ] get remote short hostname
[nuc1][DEBUG ] fetch remote file
[nuc1][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.nuc1.asok mon_status
[nuc1][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-nuc1/keyring auth get client.admin
[nuc1][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-nuc1/keyring auth get client.bootstrap-mds
[nuc1][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-nuc1/keyring auth get client.bootstrap-mgr
[nuc1][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-nuc1/keyring auth get client.bootstrap-osd
[nuc1][INFO  ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-nuc1/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.client.admin.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-mds.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] Replacing 'ceph.bootstrap-mgr.keyring' and backing up old key as 'ceph.bootstrap-mgr.keyring-20170723085013'
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-osd.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.bootstrap-rgw.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmpdhkYYM
roger@desktop:~/ceph-cluster$ cat ceph.bootstrap-mgr.keyring
[client.bootstrap-mgr]
key = AQBDt3RZPXdpNBAAekynuNJpVPaN1B4YTeFu4w==
caps mon = "allow profile bootstrap-mgr"
roger@desktop:~/ceph-cluster$ cat ceph.bootstrap-mgr.keyring-20170723085013
[client.bootstrap-mgr]
key = AAAAAAAAAAAAAAAA
caps mon = "allow profile bootstrap-mgr"
roger@desktop:~/ceph-cluster$ ceph-deploy -v mgr create nuc1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/roger/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.38): /usr/bin/ceph-deploy -v mgr create nuc1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : True
[ceph_deploy.cli][INFO  ]  mgr                           : [('nuc1', 'nuc1')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f410776cc20>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mgr at 0x7f4107de3668>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts nuc1:nuc1
[nuc1][DEBUG ] connection detected need for sudo
[nuc1][DEBUG ] connected to host: nuc1 
[nuc1][DEBUG ] detect platform information from remote host
[nuc1][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: Ubuntu 16.04 xenial
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to nuc1
[nuc1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[nuc1][DEBUG ] create path if it doesn't exist
[nuc1][INFO  ] Running command: sudo ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.nuc1 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-nuc1/keyring
[nuc1][ERROR ] 2017-07-23 14:51:13.413218 7f62943cc700  0 librados: client.bootstrap-mgr authentication error (22) Invalid argument
[nuc1][ERROR ] InvalidArgumentError does not take keyword arguments
[nuc1][ERROR ] exit code from command was: 1
[ceph_deploy.mgr][ERROR ] could not create mgr
[ceph_deploy][ERROR ] GenericError: Failed to create 1 MGRs

roger@desktop:~/ceph-cluster$ 



On Sun, Jul 23, 2017 at 1:17 AM Mark Kirkwood <mark.kirkwood@xxxxxxxxxxxxxxx> wrote:
On 22/07/17 23:50, Oscar Segarra wrote:

> Hi,
>
> I have upgraded from kraken version with a simple "yum upgrade
> command". Later the upgrade, I'd like to deploy the mgr daemon on one
> node of my ceph infrastrucute.
>
> But, for any reason, It gets stuck!
>
> Let's see the complete set of commands:
>
>
> [root@vdicnode01 ~]# ceph -s
>   cluster:
>     id:     656e84b2-9192-40fe-9b81-39bd0c7a3196
>     health: HEALTH_WARN
> *            no active mgr*
>
>   services:
>     mon: 1 daemons, quorum vdicnode01
>     mgr: no daemons active
>     osd: 2 osds: 2 up, 2 in
>
>   data:
>     pools:   0 pools, 0 pgs
>     objects: 0 objects, 0 bytes
>     usage:   0 kB used, 0 kB / 0 kB avail
>     pgs:
>
> [root@vdicnode01 ~]# su - vdicceph
> Last login: Sat Jul 22 12:50:38 CEST 2017 on pts/0
> [vdicceph@vdicnode01 ~]$ cd ceph
>
> *[vdicceph@vdicnode01 ceph]$ ceph-deploy --username vdicceph -v mgr
> create vdicnode02.local*
> [ceph_deploy.conf][DEBUG ] found configuration file at:
> /home/vdicceph/.cephdeploy.conf
> [ceph_deploy.cli][INFO  ] Invoked (1.5.38): /bin/ceph-deploy
> --username vdicceph -v mgr create vdicnode02.local
> [ceph_deploy.cli][INFO  ] ceph-deploy options:
> [ceph_deploy.cli][INFO  ]  username                      : vdicceph
> [ceph_deploy.cli][INFO  ]  verbose                       : True
> [ceph_deploy.cli][INFO  ]  mgr                           :
> [('vdicnode02.local', 'vdicnode02.local')]
> [ceph_deploy.cli][INFO  ]  overwrite_conf                : False
> [ceph_deploy.cli][INFO  ]  subcommand                    : create
> [ceph_deploy.cli][INFO  ]  quiet                         : False
> [ceph_deploy.cli][INFO  ]  cd_conf                       :
> <ceph_deploy.conf.cephdeploy.Conf instance at 0x164f290>
> [ceph_deploy.cli][INFO  ]  cluster                       : ceph
> [ceph_deploy.cli][INFO  ]  func                          : <function
> mgr at 0x15db848>
> [ceph_deploy.cli][INFO  ]  ceph_conf                     : None
> [ceph_deploy.cli][INFO  ]  default_release               : False
> [ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts
> vdicnode02.local:vdicnode02.local
> [vdicnode02.local][DEBUG ] connection detected need for sudo
> [vdicnode02.local][DEBUG ] connected to host: vdicceph@vdicnode02.local
> [vdicnode02.local][DEBUG ] detect platform information from remote host
> [vdicnode02.local][DEBUG ] detect machine type
> [ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.3.1611 Core
> [ceph_deploy.mgr][DEBUG ] remote host will use systemd
> [ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to vdicnode02.local
> [vdicnode02.local][DEBUG ] write cluster configuration to
> /etc/ceph/{cluster}.conf
> [vdicnode02.local][DEBUG ] create path if it doesn't exist
> [vdicnode02.local][INFO  ] Running command: sudo ceph --cluster ceph
> --name client.bootstrap-mgr --keyring
> /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create
> mgr.vdicnode02.local mon allow profile mgr osd allow * mds allow * -o
> /var/lib/ceph/mgr/ceph-vdicnode02.local/keyring
> [vdicnode02.local][WARNIN] No data was received after 300 seconds,
> disconnecting...
> [vdicnode02.local][INFO  ] Running command: sudo systemctl enable
> ceph-mgr@vdicnode02.local
> [vdicnode02.local][WARNIN] Created symlink from
> /etc/systemd/system/ceph-mgr.target.wants/ceph-mgr@vdicnode02.local.service
> to /usr/lib/systemd/system/ceph-mgr@.service.
> [vdicnode02.local][INFO  ] Running command: sudo systemctl start
> ceph-mgr@vdicnode02.local
> [vdicnode02.local][INFO  ] Running command: sudo systemctl enable
> ceph.target
>
> *[vdicceph@vdicnode01 ceph]$ sudo ceph -s --verbose --watch-warn
> --watch-error*
> parsed_args: Namespace(admin_socket=None, admin_socket_nope=None,
> cephconf=None, client_id=None, client_name=None, cluster=None,
> cluster_timeout=None, completion=False, help=False, input_file=None,
> output_file=None, output_format=None, status=True, verbose=True,
> version=False, watch=False, watch_channel='cluster',
> watch_debug=False, watch_error=True, watch_info=False,
> watch_sec=False, watch_warn=True), childargs: []
>
> < no response for ever >
>
> Anybody has experienced the same issue? how can I make my ceph work again?
>
> Thanks a lot.
>
>
>

I've encountered this (upgrading from Jewel).

The cause seems to be a busted mgr bootstrap key (see below). Simply
restarting your Ceph mons *should* get you back to functioning (mon has
hung as the key is too short), then you can fix the key and deploy a mgr
(here's my example for deploying a mgr on my host ceph1):

$ sudo ceph auth get client.bootstrap-mgr
exported keyring for client.bootstrap-mgr
[client.bootstrap-mgr]
         key = AAAAAAAAAAAAAAAA
         caps mon = "allow profile bootstrap-mgr"


So destroy and recreate it:


$ sudo ceph auth del client.bootstrap-mgr
updated

$ sudo ceph auth get-or-create client.bootstrap-mgr mon 'allow profile
bootstrap-mgr'
[client.bootstrap-mgr]
         key = AQBDenFZW7yKJxAAYlSBQLtDADIzsnfBcdxHpg==

$ ceph-deploy -v gatherkeys ceph1
$ ceph-deploy -v mgr create ceph1


regards

Mark


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux