Afternoon,
Happily, I resolved this issue.
Running vgdisplay showed that ceph-volume tried to create a disk on failed disk. (We didn't know we had a bad did so this is information that was new to us) and when the command failed it left three bad volume groups. Since you cannot rename them you need to use the following command to delete them.
vgdisplay to find the bad volume groups
vgremove --select vg_uuid=your uuid -f # -f forces it to be removed
Rhian Resnick
Associate Director Middleware and HPC
Office of Information Technology
Florida Atlantic University
777 Glades Road, CM22, Rm 173B
Boca Raton, FL 33431
Phone 561.297.2647
Fax 561.297.0222
From: Rhian Resnick
Sent: Saturday, April 14, 2018 12:47 PM
To: Alfredo DezaThanks all,
Here is a link to our our command being executed: https://pastebin.
com/iy8iSaKH
Here are the results from the command
Executed with debug enabled (after a zap with destroy)
[root@ceph-storage3 ~]# ceph-volume lvm create --bluestore --data /dev/sduRunning command: ceph-authtool --gen-print-keyRunning command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 664894a8-530a-4557-b2f4- 1af5b391f2b7 --> Was unable to complete a new OSD, will rollback changes--> OSD will be fully purged from the cluster, because the ID was generatedRunning command: ceph osd purge osd.140 --yes-i-really-mean-itstderr: purged osd.140Traceback (most recent call last):File "/sbin/ceph-volume", line 6, in <module>main.Volume()File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 37, in __init__ self.main(self.argv)File "/usr/lib/python2.7/site-packages/ceph_volume/ decorators.py", line 59, in newfunc return f(*a, **kw)File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, in main terminal.dispatch(self.mapper, subcommand_args)File "/usr/lib/python2.7/site-packages/ceph_volume/terminal. py", line 182, in dispatch instance.main()File "/usr/lib/python2.7/site-packages/ceph_volume/devices/ lvm/main.py", line 38, in main terminal.dispatch(self.mapper, self.argv)File "/usr/lib/python2.7/site-packages/ceph_volume/terminal. py", line 182, in dispatch instance.main()File "/usr/lib/python2.7/site-packages/ceph_volume/devices/ lvm/create.py", line 74, in main self.create(args)File "/usr/lib/python2.7/site-packages/ceph_volume/ decorators.py", line 16, in is_root return func(*a, **kw)File "/usr/lib/python2.7/site-packages/ceph_volume/devices/ lvm/create.py", line 26, in create prepare_step.safe_prepare(args) File "/usr/lib/python2.7/site-packages/ceph_volume/devices/ lvm/prepare.py", line 217, in safe_prepare self.prepare(args)File "/usr/lib/python2.7/site-packages/ceph_volume/ decorators.py", line 16, in is_root return func(*a, **kw)File "/usr/lib/python2.7/site-packages/ceph_volume/devices/ lvm/prepare.py", line 283, in prepare block_lv = self.prepare_device(args.data, 'block', cluster_fsid, osd_fsid)File "/usr/lib/python2.7/site-packages/ceph_volume/devices/ lvm/prepare.py", line 193, in prepare_device if api.get_vg(vg_name=vg_name):File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm. py", line 334, in get_vg return vgs.get(vg_name=vg_name, vg_tags=vg_tags)File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm. py", line 429, in get raise MultipleVGsError(vg_name)ceph_volume.exceptions.MultipleVGsError: Got more than 1 result looking for volume group: ceph-6a2e8f21-bca2-492b-8869- eecc995216cc
Rhian Resnick
Associate Director Middleware and HPC
Office of Information Technology
Florida Atlantic University
777 Glades Road, CM22, Rm 173B
Boca Raton, FL 33431
Phone 561.297.2647
Fax 561.297.0222
From: Alfredo Deza <adeza@xxxxxxxxxx>
Sent: Saturday, April 14, 2018 8:45 AM
To: Rhian Resnick
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: Error Creating OSD
On Fri, Apr 13, 2018 at 8:20 PM, Rhian Resnick <rresnick@xxxxxxx> wrote:
Evening,
When attempting to create an OSD we receive the following error.
[ceph-admin@ceph-storage3 ~]$ sudo ceph-volume lvm create --bluestore --data /dev/sduRunning command: ceph-authtool --gen-print-keyRunning command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c8cb8cff-dad9-48b8-8d77-6f130a 4b629d --> Was unable to complete a new OSD, will rollback changes--> OSD will be fully purged from the cluster, because the ID was generatedRunning command: ceph osd purge osd.140 --yes-i-really-mean-itstderr: purged osd.140--> MultipleVGsError: Got more than 1 result looking for volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc Any hints on what to do? This occurs when we attempt to create osd's on this node.
Can you use a paste site and get the /var/log/ceph/ceph-volume.log contents? Also, if you could try the same command but with:
CEPH_VOLUME_DEBUG=1
I think you are hitting two issues here:
1) Somehow `osd new` is not completing and failing
2) The `purge` command to wipe out the LV is getting multiple LV's and cannot make sure to match the one it used.
#2 definitely looks like something we are doing wrong, and #1 can have a lot of different causes. The logs would be tremendously helpful!
Rhian Resnick
Associate Director Middleware and HPC
Office of Information Technology
Florida Atlantic University
777 Glades Road, CM22, Rm 173B
Boca Raton, FL 33431
Phone 561.297.2647
Fax 561.297.0222
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
