|
Afternoon,
Happily, I resolved this issue.
Running vgdisplay showed that ceph-volume tried to create a disk on failed disk. (We didn't know we had a bad did so this is information that was new to us) and when the command failed it left three bad volume groups. Since you cannot rename them you need to use the following command to delete them.
vgdisplay to find the bad volume groups vgremove --select vg_uuid=your uuid -f # -f forces it to be removed
Rhian Resnick Associate Director Middleware and HPC Office of Information Technology
Florida Atlantic University 777 Glades Road, CM22, Rm 173B Boca Raton, FL 33431 Phone 561.297.2647 Fax 561.297.0222
From: Rhian Resnick
Sent: Saturday, April 14, 2018 12:47 PM To: Alfredo Deza Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: Error Creating OSD Thanks all,
Here is a link to our our command being executed: https://pastebin.com/iy8iSaKH
Here are the results from the command
Executed with debug enabled (after a zap with destroy)
[root@ceph-storage3 ~]# ceph-volume lvm create --bluestore --data /dev/sdu
Running command: ceph-authtool --gen-print-key
Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 664894a8-530a-4557-b2f4-1af5b391f2b7
--> Was unable to complete a new OSD, will rollback changes
--> OSD will be fully purged from the cluster, because the ID was generated
Running command: ceph osd purge osd.140 --yes-i-really-mean-it
stderr: purged osd.140
Traceback (most recent call last):
File "/sbin/ceph-volume", line 6, in <module>
main.Volume()
File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 37, in __init__
self.main(self.argv)
File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch
instance.main()
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", line 38, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in dispatch
instance.main()
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", line 74, in main
self.create(args)
File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create
prepare_step.safe_prepare(args)
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", line 217, in safe_prepare
self.prepare(args)
File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", line 283, in prepare
block_lv = self.prepare_device(args.data, 'block', cluster_fsid, osd_fsid)
File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", line 193, in prepare_device
if api.get_vg(vg_name=vg_name):
File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 334, in get_vg
return vgs.get(vg_name=vg_name, vg_tags=vg_tags)
File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 429, in get
raise MultipleVGsError(vg_name)
ceph_volume.exceptions.MultipleVGsError: Got more than 1 result looking for volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc
Rhian Resnick Associate Director Middleware and HPC Office of Information Technology
Florida Atlantic University 777 Glades Road, CM22, Rm 173B Boca Raton, FL 33431 Phone 561.297.2647 Fax 561.297.0222
From: Alfredo Deza <adeza@xxxxxxxxxx>
Sent: Saturday, April 14, 2018 8:45 AM To: Rhian Resnick Cc: ceph-users@xxxxxxxxxxxxxx Subject: Re: Error Creating OSD On Fri, Apr 13, 2018 at 8:20 PM, Rhian Resnick
<rresnick@xxxxxxx> wrote:
Can you use a paste site and get the /var/log/ceph/ceph-volume.log contents? Also, if you could try the same command but with:
CEPH_VOLUME_DEBUG=1
I think you are hitting two issues here:
1) Somehow `osd new` is not completing and failing
2) The `purge` command to wipe out the LV is getting multiple LV's and cannot make sure to match the one it used.
#2 definitely looks like something we are doing wrong, and #1 can have a lot of different causes. The logs would be tremendously helpful!
|
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
