On Mon, Jun 4, 2018 at 10:40 AM, Wido den Hollander <wido@xxxxxxxx> wrote: > > > On 06/04/2018 02:05 PM, Alfredo Deza wrote: >> On Mon, Jun 4, 2018 at 7:09 AM, Wido den Hollander <wido@xxxxxxxx> wrote: >>> Hi, >>> >>> After getting back from vacation the first thing to do is deploying a >>> Mimic cluster and I ran into some problems. >>> >>> For this very test setup I'm using VMs and ceph-deploy. >>> >>> This command however fails: >>> >>> $ ceph-deploy osd create bravo --data /dev/sdb >>> >>> [bravo][DEBUG ] Running command: /bin/ln -s >>> /dev/ceph-46188fa4-f30f-4e87-8dce-6027dc851108/osd-block-6e9ba998-e69a-42a9-911c-5ed8a31fba1f >>> /var/lib/ceph/osd/ceph-1/block >>> [bravo][DEBUG ] Running command: /usr/bin/ceph --cluster ceph --name >>> client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring >>> mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap >>> [bravo][DEBUG ] stderr: got monmap epoch 1 >>> [bravo][DEBUG ] Running command: /usr/bin/ceph-authtool >>> /var/lib/ceph/osd/ceph-1/keyring --create-keyring --name osd.1 --add-key >>> AQCSEBVb8OAJNBAAK0DOoQpLdDz2WjW4YvG/CA== >>> [bravo][DEBUG ] stdout: creating /var/lib/ceph/osd/ceph-1/keyring >>> [bravo][DEBUG ] stdout: added entity osd.1 auth auth(auid = >>> 18446744073709551615 key=AQCSEBVb8OAJNBAAK0DOoQpLdDz2WjW4YvG/CA== with 0 >>> caps) >>> [bravo][DEBUG ] Running command: /bin/chown -R ceph:ceph >>> /var/lib/ceph/osd/ceph-1/keyring >>> [bravo][DEBUG ] Running command: /bin/chown -R ceph:ceph >>> /var/lib/ceph/osd/ceph-1/ >>> [bravo][DEBUG ] Running command: /usr/bin/ceph-osd --cluster ceph >>> --osd-objectstore bluestore --mkfs -i 1 --monmap >>> /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osd-data >>> /var/lib/ceph/osd/ceph-1/ --osd-uuid >>> 6e9ba998-e69a-42a9-911c-5ed8a31fba1f --setuser ceph --setgroup ceph >>> [bravo][DEBUG ] --> ceph-volume lvm prepare successful for: /dev/sdb >>> [bravo][DEBUG ] Running command: /usr/bin/ceph-bluestore-tool >>> --cluster=ceph prime-osd-dir --dev >>> /dev/ceph-46188fa4-f30f-4e87-8dce-6027dc851108/osd-block-6e9ba998-e69a-42a9-911c-5ed8a31fba1f >>> --path /var/lib/ceph/osd/ceph-1 >>> [bravo][DEBUG ] stderr: 2018-06-04 12:12:52.523 7f0a444e1a00 -1 auth: >>> unable to find a keyring on >>> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> (2) No such file or directory >>> [bravo][DEBUG ] stderr: 2018-06-04 12:12:52.523 7f0a444e1a00 -1 >>> monclient: ERROR: missing keyring, cannot use cephx for authentication >>> [bravo][DEBUG ] stderr: failed to fetch mon config (--no-mon-config to >>> skip) >>> >>> The lines get wrapped by the e-mail, but this is the failing command: >>> >>> /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev >>> /dev/ceph-46188fa4-f30f-4e87-8dce-6027dc851108/osd-block-6e9ba998-e69a-42a9-911c-5ed8a31fba1f >>> --path /var/lib/ceph/osd/ceph-1 >>> >>> 2018-06-04 13:05:08.611 7fb66d34ba00 -1 auth: unable to find a keyring >>> on >>> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >>> (2) No such file or directory >>> 2018-06-04 13:05:08.619 7fb66d34ba00 -1 monclient: ERROR: missing >>> keyring, cannot use cephx for authentication >>> failed to fetch mon config (--no-mon-config to skip) >>> >>> The */var/lib/ceph/bootstrap-osd/ceph.keyring* file is present and has >>> the proper content. >>> >>> If I put */etc/ceph/ceph.client.admin.keyring* on that system the >>> deployment of the OSD works, but that shouldn't be right. >>> >>> I manually tried to supply some arguments to *ceph-bluestore-tool* for >>> the sake of testing, but that didn't work either. >>> >>> Now, if I add '--no-mon-config' to *ceph-bluestore-tool* it primes the >>> directory just fine. >>> >>> So is this a bug in ceph-volume? I assume so, but I wanted to >>> double-check before submitting a PR and issue. >> >> This is a problem in ceph-deploy, not ceph-volume >> >> ceph-volume is tested using ceph-ansible only, and there are no issues >> there. This is probably due to the recent changes in master/mimic >> regarding keys >> > > I've tested without ceph-deploy and I run into the same problem: > > $ ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdc > > That's what I run on the CLI. It creates a new OSD in the cephx and in > OSDMap, but afterwards I get: > > Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore > bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap > --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid > 3c1eebc4-7788-4357-aca0-cdfef168b618 --setuser ceph --setgroup ceph > --> ceph-volume lvm prepare successful for: /dev/sdb > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph > prime-osd-dir --dev > /dev/ceph-a48d9959-df0a-4cb6-a575-5051981240e0/osd-block-3c1eebc4-7788-4357-aca0-cdfef168b618 > --path /var/lib/ceph/osd/ceph-2 > stderr: 2018-06-04 16:34:42.564 7fd6fc8f1a00 -1 auth: unable to find a > keyring on > /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: > (2) No such file or directory > 2018-06-04 16:34:42.564 7fd6fc8f1a00 -1 monclient: ERROR: missing > keyring, cannot use cephx for authentication > stderr: failed to fetch mon config (--no-mon-config to skip) > --> Was unable to complete a new OSD, will rollback changes > --> OSD will be fully purged from the cluster, because the ID was generated > > I see the same message here and that ceph-deploy executes the same command. This is interesting, because we aren't hitting it with ceph-ansible, I wonder what is different there that allows us to just run fine without seeing this. I did see this happening on ceph-deploy on mimic, thanks for opening the PR, I will follow up with my findings > > So this doesn't seem to be ceph-deploy related. > > For now I've created this PR which solves it for me. The OSD comes > online just fine: https://github.com/ceph/ceph/pull/22393 > > Wido > >> >>> >>> OS: Ubuntu 16.04.4 >>> Ceph: 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) >>> >>> Wido >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
