Sorry to reply to my own question, but I noticed that the cephx
key for client.bootstrap-mgr was inconsistent with the key in
/var/lib/ceph/bootstrap-mgr/ceph.keyring.
I deleted the entry in ceph:
ceph auth del client.bootstrap-mgr
reran the ceph-deploy gather keys:
ceph-deploy gatherkeys cephtest-admin
and used the key entry for client.bootstrap-mgr as the key value
in /var/lib/ceph/bootstrap-mgr/ceph.keyring.
I was then able to successfully run the ceph-deploy to create the
mgr.
I'm not sure where this inconsistent key came from as the
/var/lib/ceph/bootstrap-mgr directory doesn't appear on a fresh
install of jewel.
On 18/10/17 11:04 AM, Gary molenkamp
wrote:
I'm running into a permission error when attempting to use
ceph-deploy to create an mgr on a recently upgraded
jewel->luminous ceph cluster. I've attempted to track down
the permission, but so far no success. I'm doing this on a dev
environment so I can replicate:
Start with a sample jewel release (one admin node with mon, two
storage nodes with one osd each). ceph preflight complete
(cephinst user with sudo, ntp, etc):
On each storage node, the osd disk is formatted as xfs and
mounted on /var/lib/ceph/osd/ceph-X, and "chown ceph:ceph
/var/lib/ceph/osd/ceph-X"
On admin server:
su cephinst
mkdir sci-cluster
cd sci-cluster
ceph-deploy
new cephtest-admin
ceph-deploy install cephtest-admin
cephtest-stor1 cephtest-stor2
ceph-deploy mon create-initial
ceph-deploy osd prepare cephtest-stor1:/var/lib/ceph/osd/ceph-0
ceph-deploy osd prepare cephtest-stor1
ceph-deploy
osd activate cephtest-stor10
ceph-deploy osd activate cephtest-stor1
cephtest-stor1 cephtest-stor2
ceph
osd pool set rbd size 2
sudo chmod a+r /etc/ceph/ceph.client.admin.keyring
At this point I have a working, healthy jewel cluster as
reported by "ceph -s". I then upgrade the mon and try to
deploy an mgr service:
sudo sed -i 's/jewel/luminous/g'
/etc/yum.repos.d/ceph-deploy.repo
sudo
sed -i 's/jewel/luminous/g' /etc/yum.repos.d/ceph.repo
sudo systemctl stop ceph-mon@cephtest-admin.service
sudo yum clean all
sudo yum update
sudo systemctl start ceph-mon@cephtest-admin.service
"ceph -s" reports HEALTH_OK, but "mgr: no daemons active".
Attempt to deploy a mgr service with ceph-deploy and it fails:
ceph-deploy mgr create cephtest-admin
[ceph_deploy.conf][DEBUG ] found configuration file at:
/home/cephinst/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39):
/usr/bin/ceph-deploy mgr create cephtest-admin
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username :
None
[ceph_deploy.cli][INFO ] verbose :
False
[ceph_deploy.cli][INFO ] mgr :
[('cephtest-admin', 'cephtest-admin')]
[ceph_deploy.cli][INFO ] overwrite_conf :
False
[ceph_deploy.cli][INFO ] subcommand :
create
[ceph_deploy.cli][INFO ] quiet :
False
[ceph_deploy.cli][INFO ] cd_conf :
<ceph_deploy.conf.cephdeploy.Conf instance at 0x237b320>
[ceph_deploy.cli][INFO ] cluster :
ceph
[ceph_deploy.cli][INFO ] func :
<function mgr at 0x230ac08>
[ceph_deploy.cli][INFO ] ceph_conf :
None
[ceph_deploy.cli][INFO ] default_release :
False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts
cephtest-admin:cephtest-admin
[cephtest-admin][DEBUG ] connection detected need for sudo
[cephtest-admin][DEBUG ] connected to host: cephtest-admin
[cephtest-admin][DEBUG ] detect platform information from
remote host
[cephtest-admin][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO ] Distro info: CentOS Linux 7.4.1708
Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to
cephtest-admin
[cephtest-admin][DEBUG ] write cluster configuration to
/etc/ceph/{cluster}.conf
[cephtest-admin][DEBUG ] create path if it doesn't exist
[cephtest-admin][INFO ] Running command: sudo ceph --cluster
ceph --name client.bootstrap-mgr --keyring
/var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create
mgr.cephtest-admin mon allow profile mgr osd allow * mds allow
* -o /var/lib/ceph/mgr/ceph-cephtest-admin/keyring
[cephtest-admin][ERROR ] 2017-10-18 10:42:00.460124
7f9325bcd700 0 librados: client.bootstrap-mgr authentication
error (1) Operation not permitted
[cephtest-admin][ERROR ] [errno 1] error connecting to the
cluster
[cephtest-admin][ERROR ] exit code from command was: 1
[ceph_deploy.mgr][ERROR ] could not create mgr
It looks like cephx was set up properly:
ceph auth list
installed auth entries:
osd.0
key: AQChOOZZe1PVGBAAGFRiziREnCm8TZL3QwuFnw==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
osd.1
key: AQDEOOZZQVqUORAALnPX6+tuhKM33+bveQxiDw==
caps: [mgr] allow profile osd
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQD0N+ZZUSluEBAATIGq+KK3LdUYZHw4RWToXg==
caps: [mds] allow *
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *
client.bootstrap-mds
key: AQD1N+ZZZSnPABAANDqpSE9g6MqfmXglGzn6Nw==
caps: [mgr] allow r
caps: [mon] allow profile bootstrap-mds
client.bootstrap-mgr
key: AQBnZ+dZzitjIxAArptc11qf8UuPq5QXFy+I9Q==
caps: [mon] allow profile bootstrap-mgr
client.bootstrap-osd
key: AQD0N+ZZrLv5HRAAbeLByJJyplQEvjz1o1N8kg==
caps: [mgr] allow r
caps: [mon] allow profile bootstrap-osd
client.bootstrap-rgw
key: AQD0N+ZZFL/hLxAAR9WbRd/ETL/GJ+NKXum2iA==
caps: [mgr] allow r
caps: [mon] allow profile bootstrap-rgw
Any insight or hints would be appreciated.
Thanks
Gary
PS. on a previous attempt I did upgrade the ceph versions on
the osds as well before attempting to deploy the mgr; same
result.
--
Gary Molenkamp Computer Science
Systems Administrator University of Western Ontario
molenkam@xxxxxx http://www.csd.uwo.ca
(519) 661-2111 x86882 (519) 661-3566
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Gary Molenkamp Computer Science
Systems Administrator University of Western Ontario
molenkam@xxxxxx http://www.csd.uwo.ca
(519) 661-2111 x86882 (519) 661-3566
