On 3/19/21 6:22 PM, Philip Brown wrote:
I made *some* progress for cleanup. I could already do "ceph osd rm 33" from my master. But doing the cleanup on the actual OSD node was problematical. ceph-volume lvm zap xxx wasnt working properly.. because the device wasnt fully released.... because at the regular OS level, it cant even SEE the VGs?? That caught me by surprise. But doing cephadm shell let me see the vgs, remove it, and thus have the zap work. so now we move on to reconstructing the hybrid OSD. First off, by default, the cephadm shell did not have permission to create OSDs. so I had to do [ceph: root@dxxxx /]# ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring Unfortunately, since I had run the lvm zap on both the data /dev/sdX, AND the db lv partition.. attempting to recreate the OSD with ceph-volume lvm prepare --data /dev/sdb --block.db /dev/ceph-xx-xx-xx/osd-db-xxxx (the original db lvm on SSD, which still technically existed) FAILED, because --> blkid could not detect a PARTUUID for device: /dev/ceph-xxxx/osd-xxx --> Was unable to complete a new OSD, will rollback changes cmon.... just MAKE one for me??? :-( Happily, i could grep for osd-db-specific-id-here in /var/log/ceph/ceph-volume.log and found the exact original lvcreate syntax to remake it. BUT.... lvm prepare once again complained about not detecting a PARTUUID. I think there may be a command to do that, that is left out of the ceph-volume.log :( So.. now what can I do?
Not to try to create OSDs by hand. That might be an old way to do it, but with containers this is just a PITY. So, my best guess is that you put the OSD in a state that cephadm expects it to be to be:
https://docs.ceph.com/en/latest/cephadm/osd/#deploy-osds Make sure all the conditions are met: The device must have no partitions. The device must not have any LVM state. The device must not be mounted. The device must not contain a file system. The device must not contain a Ceph BlueStore OSD. The device must be larger than 5 GB. So make sure everything is destroyed from this device. ceph orch daemon add osd hostname:/dev/device or ceph orch apply osd --all-available-devices I guess that should be the same device (no other unused disks). And let cephadm just make it happen ... wishfull thinking at this point ;-). Gr. Stefan _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
