Re: Reef - what happened to OSD spec?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


Just to clarify, I realized that my statement regarding the zapping of LVs for db/wal ist likely misleading. What I meant was that I don't see an option in the dashboard for that, but if you remove an osd in the command line the respective LVs are cleaned up:

$ ceph orch osd rm 0 --zap --force

will remove the osd as well as the corresponding db/wal device. Sorry for any confusion.

Zitat von Eugen Block <eblock@xxxxxx>:

Hi, just a few days ago I replied to a thread [2] with some explanations for destroy, delete and purge.

So if you "destroy" an OSD it is meant to be replaced, reusing the ID. A failed drive may not be responsive at all so an automated wipe might fail as well. If the db/wal is located on a different device you'll have to clean that up manually. I don't think cephadm is able to do that for you (yet). I just checked with a virtual Reef cluster. So in a real world scenario if an OSD fails, you destroy it (which marks it as "destroyed" in the crush tree), then you wipe the LV containing db/wal manually. Unfortunately, 'ceph orch device zap' can't deal with VG/LV syntax:

$ ceph orch device zap ceph01 /dev/ceph-0a339b77-072a-4a03-92d2-a2eda00bd12c/osd-db-5e64f473-07a3-432b-96e4-643e3bdbf2c0 --force Error EINVAL: Device path '/dev/ceph-0a339b77-072a-4a03-92d2-a2eda00bd12c/osd-db-5e64f473-07a3-432b-96e4-643e3bdbf2c0' not found on host 'ceph01'

So you'll have to wipe it locally with ceph-volume:

cephadm ceph-volume lvm zap --destroy /dev/ceph-{VG}/osd-db-{LV}

When the failed drive has been replaced, cephadm will redeploy the OSD(s). You might want to pause the orchestrator (ceph orch pause) before wiping the drive(s) as it might deploy OSDs if multiple specs apply to the configuration.


Zitat von Nigel Williams <nigel.williams@xxxxxxxxxxx>:

Thanks Eugen for following up. Sorry my second response was incomplete. I
can confirm that it works as expected too. My confusion was that the
section from the online documentation seemed to be missing/moved, and when
it initially failed I wrongly thought that the OSD-add process had changed
in the Reef release.

There might still need to be a way that "destroy" does additional clean-up
to clear remnants of LVM fingerprints on the devices as this tripped me up
when the OSDspec apply failed due to "filesystem on device" checks.

Documentation has been improved and OSD spec is now under this heading for
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux