Hi, just a few days ago I replied to a thread [2] with some
explanations for destroy, delete and purge.
So if you "destroy" an OSD it is meant to be replaced, reusing the ID.
A failed drive may not be responsive at all so an automated wipe might
fail as well. If the db/wal is located on a different device you'll
have to clean that up manually. I don't think cephadm is able to do
that for you (yet). I just checked with a virtual Reef cluster.
So in a real world scenario if an OSD fails, you destroy it (which
marks it as "destroyed" in the crush tree), then you wipe the LV
containing db/wal manually. Unfortunately, 'ceph orch device zap'
can't deal with VG/LV syntax:
$ ceph orch device zap ceph01
/dev/ceph-0a339b77-072a-4a03-92d2-a2eda00bd12c/osd-db-5e64f473-07a3-432b-96e4-643e3bdbf2c0
--force
Error EINVAL: Device path
'/dev/ceph-0a339b77-072a-4a03-92d2-a2eda00bd12c/osd-db-5e64f473-07a3-432b-96e4-643e3bdbf2c0' not found on host
'ceph01'
So you'll have to wipe it locally with ceph-volume:
cephadm ceph-volume lvm zap --destroy /dev/ceph-{VG}/osd-db-{LV}
When the failed drive has been replaced, cephadm will redeploy the
OSD(s). You might want to pause the orchestrator (ceph orch pause)
before wiping the drive(s) as it might deploy OSDs if multiple specs
apply to the configuration.
[2]
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/74FHIF73PTIGEC2T7JBPZ2BUIEMNNMEF/
Zitat von Nigel Williams <nigel.williams@xxxxxxxxxxx>:
Thanks Eugen for following up. Sorry my second response was incomplete. I
can confirm that it works as expected too. My confusion was that the
section from the online documentation seemed to be missing/moved, and when
it initially failed I wrongly thought that the OSD-add process had changed
in the Reef release.
There might still need to be a way that "destroy" does additional clean-up
to clear remnants of LVM fingerprints on the devices as this tripped me up
when the OSDspec apply failed due to "filesystem on device" checks.
Documentation has been improved and OSD spec is now under this heading for
Reef:
https://docs.ceph.com/en/reef/cephadm/services/osd/#advanced-osd-service-specifications
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
