On Mon, Feb 26, 2018 at 12:51 PM, David Turner <drakonstein@xxxxxxxxx> wrote: > I don't follow what ceph-deploy has to do with the man page for ceph-volume. > Is ceph-volume also out-of-tree and as such the man pages aren't version > specific with its capabilities? It's very disconcerting to need to ignore > the man pages for CLI tools. Sorry, it seems like I completely got confused there :) We forgot to backport the man page changes for the luminous branch. Thanks for pointing this out. Created http://tracker.ceph.com/issues/23142 to track this You are absolutely right, apologies for the misfire. > > On Mon, Feb 26, 2018 at 12:10 PM Alfredo Deza <adeza@xxxxxxxxxx> wrote: >> >> On Mon, Feb 26, 2018 at 11:24 AM, David Turner <drakonstein@xxxxxxxxx> >> wrote: >> > If we're asking for documentation updates, the man page for ceph-volume >> > is >> > incredibly outdated. In 12.2.3 it still says that bluestore is not yet >> > implemented and that it's planned to be supported. >> > '[--bluestore] filestore objectstore (not yet implemented)' >> > 'using a filestore setup (bluestore support is planned)'. >> >> This is a bit hard to track because ceph-deploy is an out-of-tree >> project that gets pulled into the Ceph repo, and the man page lives in >> the Ceph source tree. >> >> We have updated the man page and the references to ceph-deploy to >> correctly show the new API and all the flags supported, but this is in >> master and was not backported >> to luminous. >> >> > >> > On Mon, Feb 26, 2018 at 7:05 AM Oliver Freyermuth >> > <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: >> >> >> >> Am 26.02.2018 um 13:02 schrieb Alfredo Deza: >> >> > On Sat, Feb 24, 2018 at 1:26 PM, Oliver Freyermuth >> >> > <freyermuth@xxxxxxxxxxxxxxxxxx> wrote: >> >> >> Dear Cephalopodians, >> >> >> >> >> >> when purging a single OSD on a host (created via ceph-deploy 2.0, >> >> >> i.e. >> >> >> using ceph-volume lvm), I currently proceed as follows: >> >> >> >> >> >> On the OSD-host: >> >> >> $ systemctl stop ceph-osd@4.service >> >> >> $ ls -la /var/lib/ceph/osd/ceph-4 >> >> >> # Check block und block.db links: >> >> >> lrwxrwxrwx. 1 ceph ceph 93 23. Feb 01:28 block -> >> >> >> >> >> >> /dev/ceph-69b1fbe5-f084-4410-a99a-ab57417e7846/osd-block-cd273506-e805-40ac-b23d-c7b9ff45d874 >> >> >> lrwxrwxrwx. 1 root root 43 23. Feb 01:28 block.db -> >> >> >> /dev/ceph-osd-blockdb-ssd-1/db-for-disk-sda >> >> >> # resolve actual underlying device: >> >> >> $ pvs | grep ceph-69b1fbe5-f084-4410-a99a-ab57417e7846 >> >> >> /dev/sda ceph-69b1fbe5-f084-4410-a99a-ab57417e7846 lvm2 a-- >> >> >> <3,64t 0 >> >> >> # Zap the device: >> >> >> $ ceph-volume lvm zap --destroy /dev/sda >> >> >> >> >> >> Now, on the mon: >> >> >> # purge the OSD: >> >> >> $ ceph osd purge osd.4 --yes-i-really-mean-it >> >> >> >> >> >> Then I re-deploy using: >> >> >> $ ceph-deploy --overwrite-conf osd create --bluestore --block-db >> >> >> ceph-osd-blockdb-ssd-1/db-for-disk-sda --data /dev/sda osd001 >> >> >> >> >> >> from the admin-machine. >> >> >> >> >> >> This works just fine, however, it leaves a stray ceph-volume service >> >> >> behind: >> >> >> $ ls -la /etc/systemd/system/multi-user.target.wants/ -1 | grep >> >> >> ceph-volume@lvm-4 >> >> >> lrwxrwxrwx. 1 root root 44 24. Feb 18:30 >> >> >> ceph-volume@lvm-4-5a984083-48e1-4c2f-a1f3-3458c941e597.service -> >> >> >> /usr/lib/systemd/system/ceph-volume@.service >> >> >> lrwxrwxrwx. 1 root root 44 23. Feb 01:28 >> >> >> ceph-volume@lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874.service -> >> >> >> /usr/lib/systemd/system/ceph-volume@.service >> >> >> >> >> >> This stray service then, after reboot of the machine, stays in >> >> >> activating state (since the disk will of course never come back): >> >> >> ----------------------------------- >> >> >> $ systemctl status >> >> >> ceph-volume@lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874.service >> >> >> ● ceph-volume@lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874.service - >> >> >> Ceph >> >> >> Volume activation: lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874 >> >> >> Loaded: loaded (/usr/lib/systemd/system/ceph-volume@.service; >> >> >> enabled; vendor preset: disabled) >> >> >> Active: activating (start) since Sa 2018-02-24 19:21:47 CET; 1min >> >> >> 12s ago >> >> >> Main PID: 1866 (timeout) >> >> >> CGroup: >> >> >> >> >> >> /system.slice/system-ceph\x2dvolume.slice/ceph-volume@lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874.service >> >> >> ├─1866 timeout 10000 /usr/sbin/ceph-volume-systemd >> >> >> lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874 >> >> >> └─1872 /usr/bin/python2.7 /usr/sbin/ceph-volume-systemd >> >> >> lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874 >> >> >> >> >> >> Feb 24 19:21:47 osd001.baf.physik.uni-bonn.de systemd[1]: Starting >> >> >> Ceph >> >> >> Volume activation: lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874... >> >> >> ----------------------------------- >> >> >> Manually, I can fix this by running: >> >> >> $ systemctl disable >> >> >> ceph-volume@lvm-4-cd273506-e805-40ac-b23d-c7b9ff45d874.service >> >> >> >> >> >> My question is: Should I really remove that manually? >> >> >> Should "ceph-volume lvm zap --destroy" have taken care of it (bug)? >> >> > >> >> > You should remove it manually. The problem with zapping is that we >> >> > might not have the information we need to remove the systemd unit. >> >> > Since an OSD can be made out of different devices, ceph-volume might >> >> > be asked to "zap" a device which it can't compute to what OSD it >> >> > belongs. The systemd units are tied to the ID and UUID of the OSD. >> >> >> >> Understood, thanks for the reply! >> >> >> >> Could this be added to the documentation at some point for all the >> >> other >> >> users operating the cluster manually / with ceph-deploy? >> >> This would likely be best to prevent others from falling into this trap >> >> ;-). >> >> Should I open a ticket asking for this? >> >> >> >> Cheers, >> >> Oliver >> >> >> >> > >> >> > >> >> >> Am I missing a step? >> >> >> >> >> >> Cheers, >> >> >> Oliver >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> ceph-users mailing list >> >> >> ceph-users@xxxxxxxxxxxxxx >> >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> >> >> >> _______________________________________________ >> >> ceph-users mailing list >> >> ceph-users@xxxxxxxxxxxxxx >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |