You didn't remove them from the auth monitor's keyring. If you're removing OSDs you need to follow the steps in the documentation. -Greg On Fri, Nov 14, 2014 at 4:42 PM, JIten Shah <jshah2005@xxxxxx> wrote: > Hi Guys, > > I had to rekick some of the hosts where OSD’s were running and after > re-kick, when I try to run puppet and install OSD’s again, it gives me a key > mismatch error (as below). After the hosts were shutdown for rekick, I > removed the OSD’s from the osd tree and the crush map too. Why is it still > tied to the old key? > > > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > + test -b /osd-data > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > + mkdir -p /osd-data > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > + ceph-disk prepare /osd-data > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > + test -b /osd-data > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > + ceph-disk activate /osd-data > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > got monmap epoch 2 > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > 2014-11-15 00:31:25.951783 7ffff7fe67a0 -1 journal FileJournal::_open: > disabling aio for non-block journal. Use journal_force_aio to force use of > aio anyway > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > 2014-11-15 00:31:26.023037 7ffff7fe67a0 -1 journal FileJournal::_open: > disabling aio for non-block journal. Use journal_force_aio to force use of > aio anyway > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > 2014-11-15 00:31:26.023809 7ffff7fe67a0 -1 filestore(/osd-data) could not > find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > 2014-11-15 00:31:26.032044 7ffff7fe67a0 -1 created object store /osd-data > journal /osd-data/journal for osd.2 fsid > 2e738cda-1930-48cd-a4b1-74bc737c5d56 > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > 2014-11-15 00:31:26.032097 7ffff7fe67a0 -1 auth: error reading file: > /osd-data/keyring: can't open /osd-data/keyring: (2) No such file or > directory > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > 2014-11-15 00:31:26.032189 7ffff7fe67a0 -1 created new key in keyring > /osd-data/keyring > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > Error EINVAL: entity osd.2 exists but key does not match > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > Traceback (most recent call last): > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > File "/usr/sbin/ceph-disk", line 2591, in <module> > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > main() > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > File "/usr/sbin/ceph-disk", line 2569, in main > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > args.func(args) > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > File "/usr/sbin/ceph-disk", line 1929, in main_activate > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > init=args.mark_init, > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > File "/usr/sbin/ceph-disk", line 1761, in activate_dir > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > (osd_id, cluster) = activate(path, activate_key_template, init) > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > File "/usr/sbin/ceph-disk", line 1897, in activate > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > keyring=keyring, > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > File "/usr/sbin/ceph-disk", line 1520, in auth_key > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > 'mon', 'allow profile osd', > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > File "/usr/sbin/ceph-disk", line 304, in command_check_call > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > return subprocess.check_call(arguments) > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > raise CalledProcessError(retcode, cmd) > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > subprocess.CalledProcessError: Command '['/usr/bin/ceph', '--cluster', > 'ceph', '--name', 'client.bootstrap-osd', '--keyring', > '/var/lib/ceph/bootstrap-osd/ceph.keyring', 'auth', 'add', 'osd.2', '-i', > '/osd-data/keyring', 'osd', 'allow *', 'mon', 'allow profile osd']' returned > non-zero exit status 22 > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > + true > Notice: > /Stage[main]/Main/Node[infrastructure_node]/Ceph::Osd[/osd-data]/Exec[ceph-osd-mkfs-/osd-data]/returns: > executed successfully > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |