On Wed, Oct 18, 2017 at 6:34 PM, Gary Molenkamp <molenkam@xxxxxx> wrote: > Sorry to reply to my own question, but I noticed that the cephx key for > client.bootstrap-mgr was inconsistent with the key in > /var/lib/ceph/bootstrap-mgr/ceph.keyring. > > I deleted the entry in ceph: > > ceph auth del client.bootstrap-mgr > > reran the ceph-deploy gather keys: > > ceph-deploy gatherkeys cephtest-admin > > and used the key entry for client.bootstrap-mgr as the key value in > /var/lib/ceph/bootstrap-mgr/ceph.keyring. > > I was then able to successfully run the ceph-deploy to create the mgr. > > I'm not sure where this inconsistent key came from as the > /var/lib/ceph/bootstrap-mgr directory doesn't appear on a fresh install of > jewel. Thanks for the report - probably same issue as http://tracker.ceph.com/issues/20950 Glad you could work around it, we also have a candidate fix for master being tested now (you motivated me to take another look at it :-)) John > > > > > > On 18/10/17 11:04 AM, Gary molenkamp wrote: > > I'm running into a permission error when attempting to use ceph-deploy to > create an mgr on a recently upgraded jewel->luminous ceph cluster. I've > attempted to track down the permission, but so far no success. I'm doing > this on a dev environment so I can replicate: > > Start with a sample jewel release (one admin node with mon, two storage > nodes with one osd each). ceph preflight complete (cephinst user with sudo, > ntp, etc): > > On each storage node, the osd disk is formatted as xfs and mounted on > /var/lib/ceph/osd/ceph-X, and "chown ceph:ceph /var/lib/ceph/osd/ceph-X" > > On admin server: > > su cephinst > mkdir sci-cluster > cd sci-cluster > ceph-deploy new cephtest-admin > ceph-deploy install cephtest-admin cephtest-stor1 cephtest-stor2 > ceph-deploy mon create-initial > ceph-deploy osd prepare cephtest-stor1:/var/lib/ceph/osd/ceph-0 > ceph-deploy osd prepare cephtest-stor2:/var/lib/ceph/osd/ceph-1 > ceph-deploy osd activate cephtest-stor1:/var/lib/ceph/osd/ceph-0 > ceph-deploy osd activate cephtest-stor2:/var/lib/ceph/osd/ceph-1 > ceph-deploy admin cephtest-admin cephtest-stor1 cephtest-stor2 > ceph osd pool set rbd size 2 > sudo chmod a+r /etc/ceph/ceph.client.admin.keyring > > At this point I have a working, healthy jewel cluster as reported by "ceph > -s". I then upgrade the mon and try to deploy an mgr service: > > sudo sed -i 's/jewel/luminous/g' /etc/yum.repos.d/ceph-deploy.repo > sudo sed -i 's/jewel/luminous/g' /etc/yum.repos.d/ceph.repo > sudo systemctl stop ceph-mon@cephtest-admin.service > sudo yum clean all > sudo yum update > sudo systemctl start ceph-mon@cephtest-admin.service > > > "ceph -s" reports HEALTH_OK, but "mgr: no daemons active". Attempt to deploy > a mgr service with ceph-deploy and it fails: > > ceph-deploy mgr create cephtest-admin > [ceph_deploy.conf][DEBUG ] found configuration file at: > /home/cephinst/.cephdeploy.conf > [ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy mgr create > cephtest-admin > [ceph_deploy.cli][INFO ] ceph-deploy options: > [ceph_deploy.cli][INFO ] username : None > [ceph_deploy.cli][INFO ] verbose : False > [ceph_deploy.cli][INFO ] mgr : > [('cephtest-admin', 'cephtest-admin')] > [ceph_deploy.cli][INFO ] overwrite_conf : False > [ceph_deploy.cli][INFO ] subcommand : create > [ceph_deploy.cli][INFO ] quiet : False > [ceph_deploy.cli][INFO ] cd_conf : > <ceph_deploy.conf.cephdeploy.Conf instance at 0x237b320> > [ceph_deploy.cli][INFO ] cluster : ceph > [ceph_deploy.cli][INFO ] func : <function mgr at > 0x230ac08> > [ceph_deploy.cli][INFO ] ceph_conf : None > [ceph_deploy.cli][INFO ] default_release : False > [ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts > cephtest-admin:cephtest-admin > [cephtest-admin][DEBUG ] connection detected need for sudo > [cephtest-admin][DEBUG ] connected to host: cephtest-admin > [cephtest-admin][DEBUG ] detect platform information from remote host > [cephtest-admin][DEBUG ] detect machine type > [ceph_deploy.mgr][INFO ] Distro info: CentOS Linux 7.4.1708 Core > [ceph_deploy.mgr][DEBUG ] remote host will use systemd > [ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to cephtest-admin > [cephtest-admin][DEBUG ] write cluster configuration to > /etc/ceph/{cluster}.conf > [cephtest-admin][DEBUG ] create path if it doesn't exist > [cephtest-admin][INFO ] Running command: sudo ceph --cluster ceph --name > client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth > get-or-create mgr.cephtest-admin mon allow profile mgr osd allow * mds allow > * -o /var/lib/ceph/mgr/ceph-cephtest-admin/keyring > [cephtest-admin][ERROR ] 2017-10-18 10:42:00.460124 7f9325bcd700 0 > librados: client.bootstrap-mgr authentication error (1) Operation not > permitted > [cephtest-admin][ERROR ] [errno 1] error connecting to the cluster > [cephtest-admin][ERROR ] exit code from command was: 1 > [ceph_deploy.mgr][ERROR ] could not create mgr > > It looks like cephx was set up properly: > > ceph auth list > installed auth entries: > > osd.0 > key: AQChOOZZe1PVGBAAGFRiziREnCm8TZL3QwuFnw== > caps: [mgr] allow profile osd > caps: [mon] allow profile osd > caps: [osd] allow * > osd.1 > key: AQDEOOZZQVqUORAALnPX6+tuhKM33+bveQxiDw== > caps: [mgr] allow profile osd > caps: [mon] allow profile osd > caps: [osd] allow * > client.admin > key: AQD0N+ZZUSluEBAATIGq+KK3LdUYZHw4RWToXg== > caps: [mds] allow * > caps: [mgr] allow * > caps: [mon] allow * > caps: [osd] allow * > client.bootstrap-mds > key: AQD1N+ZZZSnPABAANDqpSE9g6MqfmXglGzn6Nw== > caps: [mgr] allow r > caps: [mon] allow profile bootstrap-mds > client.bootstrap-mgr > key: AQBnZ+dZzitjIxAArptc11qf8UuPq5QXFy+I9Q== > caps: [mon] allow profile bootstrap-mgr > client.bootstrap-osd > key: AQD0N+ZZrLv5HRAAbeLByJJyplQEvjz1o1N8kg== > caps: [mgr] allow r > caps: [mon] allow profile bootstrap-osd > client.bootstrap-rgw > key: AQD0N+ZZFL/hLxAAR9WbRd/ETL/GJ+NKXum2iA== > caps: [mgr] allow r > caps: [mon] allow profile bootstrap-rgw > > > Any insight or hints would be appreciated. > Thanks > Gary > > PS. on a previous attempt I did upgrade the ceph versions on the osds as > well before attempting to deploy the mgr; same result. > > > -- > Gary Molenkamp Computer Science > Systems Administrator University of Western Ontario > molenkam@xxxxxx http://www.csd.uwo.ca > (519) 661-2111 x86882 (519) 661-3566 > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- > Gary Molenkamp Computer Science > Systems Administrator University of Western Ontario > molenkam@xxxxxx http://www.csd.uwo.ca > (519) 661-2111 x86882 (519) 661-3566 > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |