On 06/04/2018 02:05 PM, Alfredo Deza wrote: > On Mon, Jun 4, 2018 at 7:09 AM, Wido den Hollander <wido@xxxxxxxx> wrote: >> Hi, >> >> After getting back from vacation the first thing to do is deploying a >> Mimic cluster and I ran into some problems. >> >> For this very test setup I'm using VMs and ceph-deploy. >> >> This command however fails: >> >> $ ceph-deploy osd create bravo --data /dev/sdb >> >> [bravo][DEBUG ] Running command: /bin/ln -s >> /dev/ceph-46188fa4-f30f-4e87-8dce-6027dc851108/osd-block-6e9ba998-e69a-42a9-911c-5ed8a31fba1f >> /var/lib/ceph/osd/ceph-1/block >> [bravo][DEBUG ] Running command: /usr/bin/ceph --cluster ceph --name >> client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring >> mon getmap -o /var/lib/ceph/osd/ceph-1/activate.monmap >> [bravo][DEBUG ] stderr: got monmap epoch 1 >> [bravo][DEBUG ] Running command: /usr/bin/ceph-authtool >> /var/lib/ceph/osd/ceph-1/keyring --create-keyring --name osd.1 --add-key >> AQCSEBVb8OAJNBAAK0DOoQpLdDz2WjW4YvG/CA== >> [bravo][DEBUG ] stdout: creating /var/lib/ceph/osd/ceph-1/keyring >> [bravo][DEBUG ] stdout: added entity osd.1 auth auth(auid = >> 18446744073709551615 key=AQCSEBVb8OAJNBAAK0DOoQpLdDz2WjW4YvG/CA== with 0 >> caps) >> [bravo][DEBUG ] Running command: /bin/chown -R ceph:ceph >> /var/lib/ceph/osd/ceph-1/keyring >> [bravo][DEBUG ] Running command: /bin/chown -R ceph:ceph >> /var/lib/ceph/osd/ceph-1/ >> [bravo][DEBUG ] Running command: /usr/bin/ceph-osd --cluster ceph >> --osd-objectstore bluestore --mkfs -i 1 --monmap >> /var/lib/ceph/osd/ceph-1/activate.monmap --keyfile - --osd-data >> /var/lib/ceph/osd/ceph-1/ --osd-uuid >> 6e9ba998-e69a-42a9-911c-5ed8a31fba1f --setuser ceph --setgroup ceph >> [bravo][DEBUG ] --> ceph-volume lvm prepare successful for: /dev/sdb >> [bravo][DEBUG ] Running command: /usr/bin/ceph-bluestore-tool >> --cluster=ceph prime-osd-dir --dev >> /dev/ceph-46188fa4-f30f-4e87-8dce-6027dc851108/osd-block-6e9ba998-e69a-42a9-911c-5ed8a31fba1f >> --path /var/lib/ceph/osd/ceph-1 >> [bravo][DEBUG ] stderr: 2018-06-04 12:12:52.523 7f0a444e1a00 -1 auth: >> unable to find a keyring on >> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> (2) No such file or directory >> [bravo][DEBUG ] stderr: 2018-06-04 12:12:52.523 7f0a444e1a00 -1 >> monclient: ERROR: missing keyring, cannot use cephx for authentication >> [bravo][DEBUG ] stderr: failed to fetch mon config (--no-mon-config to >> skip) >> >> The lines get wrapped by the e-mail, but this is the failing command: >> >> /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev >> /dev/ceph-46188fa4-f30f-4e87-8dce-6027dc851108/osd-block-6e9ba998-e69a-42a9-911c-5ed8a31fba1f >> --path /var/lib/ceph/osd/ceph-1 >> >> 2018-06-04 13:05:08.611 7fb66d34ba00 -1 auth: unable to find a keyring >> on >> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: >> (2) No such file or directory >> 2018-06-04 13:05:08.619 7fb66d34ba00 -1 monclient: ERROR: missing >> keyring, cannot use cephx for authentication >> failed to fetch mon config (--no-mon-config to skip) >> >> The */var/lib/ceph/bootstrap-osd/ceph.keyring* file is present and has >> the proper content. >> >> If I put */etc/ceph/ceph.client.admin.keyring* on that system the >> deployment of the OSD works, but that shouldn't be right. >> >> I manually tried to supply some arguments to *ceph-bluestore-tool* for >> the sake of testing, but that didn't work either. >> >> Now, if I add '--no-mon-config' to *ceph-bluestore-tool* it primes the >> directory just fine. >> >> So is this a bug in ceph-volume? I assume so, but I wanted to >> double-check before submitting a PR and issue. > > This is a problem in ceph-deploy, not ceph-volume > > ceph-volume is tested using ceph-ansible only, and there are no issues > there. This is probably due to the recent changes in master/mimic > regarding keys > I've tested without ceph-deploy and I run into the same problem: $ ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdc That's what I run on the CLI. It creates a new OSD in the cephx and in OSDMap, but afterwards I get: Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 3c1eebc4-7788-4357-aca0-cdfef168b618 --setuser ceph --setgroup ceph --> ceph-volume lvm prepare successful for: /dev/sdb Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-a48d9959-df0a-4cb6-a575-5051981240e0/osd-block-3c1eebc4-7788-4357-aca0-cdfef168b618 --path /var/lib/ceph/osd/ceph-2 stderr: 2018-06-04 16:34:42.564 7fd6fc8f1a00 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2018-06-04 16:34:42.564 7fd6fc8f1a00 -1 monclient: ERROR: missing keyring, cannot use cephx for authentication stderr: failed to fetch mon config (--no-mon-config to skip) --> Was unable to complete a new OSD, will rollback changes --> OSD will be fully purged from the cluster, because the ID was generated I see the same message here and that ceph-deploy executes the same command. So this doesn't seem to be ceph-deploy related. For now I've created this PR which solves it for me. The OSD comes online just fine: https://github.com/ceph/ceph/pull/22393 Wido > >> >> OS: Ubuntu 16.04.4 >> Ceph: 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) >> >> Wido >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
