I don't follow what ceph-deploy has to do with the man page for ceph-volume. Is ceph-volume also out-of-tree and as such the man pages aren't version specific with its capabilities? It's very disconcerting to need to ignore the man pages for CLI tools.
On Mon, Feb 26, 2018 at 12:10 PM Alfredo Deza <adeza@xxxxxxxxxx> wrote:
On Mon, Feb 26, 2018 at 11:24 AM, David Turner <drakonstein@xxxxxxxxx> wrote:
> If we're asking for documentation updates, the man page for ceph-volume is
> incredibly outdated. In 12.2.3 it still says that bluestore is not yet
> implemented and that it's planned to be supported.
> '[--bluestore] filestore objectstore (not yet implemented)'
> 'using a filestore setup (bluestore support is planned)'.
This is a bit hard to track because ceph-deploy is an out-of-tree
project that gets pulled into the Ceph repo, and the man page lives in
the Ceph source tree.
We have updated the man page and the references to ceph-deploy to
correctly show the new API and all the flags supported, but this is in
master and was not backported
to luminous.
>
> On Mon, Feb 26, 2018 at 7:05 AM Oliver Freyermuth
> <freyermuth@xxxxxxxxxxxxxxxxxx> wrote:
>>
>> Am 26.02.2018 um 13:02 schrieb Alfredo Deza:
>> > On Sat, Feb 24, 2018 at 1:26 PM, Oliver Freyermuth
>> > <freyermuth@xxxxxxxxxxxxxxxxxx> wrote:
>> >> Dear Cephalopodians,
>> >>
>> >> when purging a single OSD on a host (created via ceph-deploy 2.0, i.e.
>> >> using ceph-volume lvm), I currently proceed as follows:
>> >>
>> >> On the OSD-host:
>> >> $ systemctl stop ceph-osd@4.service
>> >> $ ls -la /var/lib/ceph/osd/ceph-4
>> >> # Check block und block.db links:
>> >> lrwxrwxrwx. 1 ceph ceph 93 23. Feb 01:28 block ->
>> >> /dev/ceph-69b1fbe5-f084-4410-a99a-ab57417e7846/osd-block-cd273506-e805-40ac-b23d-c7b9ff45d874
>> >> lrwxrwxrwx. 1 root root 43 23. Feb 01:28 block.db ->
>> >> /dev/ceph-osd-blockdb-ssd-1/db-for-disk-sda
>> >> # resolve actual underlying device:
>> >> $ pvs | grep ceph-69b1fbe5-f084-4410-a99a-ab57417e7846
>> >> /dev/sda ceph-69b1fbe5-f084-4410-a99a-ab57417e7846 lvm2 a--
>> >> <3,64t 0
>> >> # Zap the device:
>> >> $ ceph-volume lvm zap --destroy /dev/sda
>> >>
>> >> Now, on the mon:
>> >> # purge the OSD:
>> >> $ ceph osd purge osd.4 --yes-i-really-mean-it
>> >>
>> >> Then I re-deploy using:
>> >> $ ceph-deploy --overwrite-conf osd create --bluestore --block-db
>> >> ceph-osd-blockdb-ssd-1/db-for-disk-sda --data /dev/sda osd001
>> >>
>> >> from the admin-machine.
>> >>
>> >> This works just fine, however, it leaves a stray ceph-volume service
>> >> behind:
>> >> $ ls -la /etc/systemd/system/multi-user.target.wants/ -1 | grep
>> >> ceph-volume@lvm-4
>> >> lrwxrwxrwx. 1 root root 44 24. Feb 18:30
>> >> ceph-volume@lvm-4-5a984083-48e1-4c2f-a1f3-3458c941e597.service ->
>> >> /usr/lib/systemd/system/ceph-volume@.service
>> >> lrwxrwxrwx. 1 root root 44 23. Feb 01:28
>> >> ceph-volume@lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874.service ->
>> >> /usr/lib/systemd/system/ceph-volume@.service
>> >>
>> >> This stray service then, after reboot of the machine, stays in
>> >> activating state (since the disk will of course never come back):
>> >> -----------------------------------
>> >> $ systemctl status
>> >> ceph-volume@lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874.service
>> >> ● ceph-volume@lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874.service - Ceph
>> >> Volume activation: lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874
>> >> Loaded: loaded (/usr/lib/systemd/system/ceph-volume@.service;
>> >> enabled; vendor preset: disabled)
>> >> Active: activating (start) since Sa 2018-02-24 19:21:47 CET; 1min
>> >> 12s ago
>> >> Main PID: 1866 (timeout)
>> >> CGroup:
>> >> /system.slice/system-ceph\x2dvolume.slice/ceph-volume@lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874.service
>> >> ├─1866 timeout 10000 /usr/sbin/ceph-volume-systemd
>> >> lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874
>> >> └─1872 /usr/bin/python2.7 /usr/sbin/ceph-volume-systemd
>> >> lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874
>> >>
>> >> Feb 24 19:21:47 osd001.baf.physik.uni-bonn.de systemd[1]: Starting Ceph
>> >> Volume activation: lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874...
>> >> -----------------------------------
>> >> Manually, I can fix this by running:
>> >> $ systemctl disable
>> >> ceph-volume@lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874.service
>> >>
>> >> My question is: Should I really remove that manually?
>> >> Should "ceph-volume lvm zap --destroy" have taken care of it (bug)?
>> >
>> > You should remove it manually. The problem with zapping is that we
>> > might not have the information we need to remove the systemd unit.
>> > Since an OSD can be made out of different devices, ceph-volume might
>> > be asked to "zap" a device which it can't compute to what OSD it
>> > belongs. The systemd units are tied to the ID and UUID of the OSD.
>>
>> Understood, thanks for the reply!
>>
>> Could this be added to the documentation at some point for all the other
>> users operating the cluster manually / with ceph-deploy?
>> This would likely be best to prevent others from falling into this trap
>> ;-).
>> Should I open a ticket asking for this?
>>
>> Cheers,
>> Oliver
>>
>> >
>> >
>> >> Am I missing a step?
>> >>
>> >> Cheers,
>> >> Oliver
>> >>
>> >>
>> >> _______________________________________________
>> >> ceph-users mailing list
>> >> ceph-users@xxxxxxxxxxxxxx
>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
