Answering my own question: I exported the spec, removed the failed drive and re-applied the spec again; the spec appears to have been updated correctly and the warning is gone. /Z On Fri, 16 Feb 2024 at 14:33, Zakhar Kirpichenko <zakhar@xxxxxxxxx> wrote: > Many thanks for your response, Eugen! > > I tried to fail mgr twice, unfortunately that had no effect on the issue. > Neither `cephadm ceph-volume inventory` nor `ceph device ls-by-host ceph03` > have the failed drive on the list. > > Though your assumption is correct, the spec appears to explicitly include > the failed drive: > > --- > service_type: osd > service_id: ceph03_combined_osd > service_name: osd.ceph03_combined_osd > placement: > hosts: > - ceph03 > spec: > data_devices: > paths: > ... > - /dev/sde > ... > db_devices: > paths: > - /dev/nvme0n1 > - /dev/nvme1n1 > filter_logic: AND > objectstore: bluestore > --- > > Do you know the best way to remove the device from the spec? > > /Z > > On Fri, 16 Feb 2024 at 14:10, Eugen Block <eblock@xxxxxx> wrote: > >> Hi, >> >> sometimes the easiest fix is to failover the mgr, have you tried that? >> If that didn't work, can you share the drivegroup spec? >> >> ceph orch ls <your_osd_spec> --export >> >> Does it contain specific device paths or something? Does 'cephadm ls' >> on that node show any traces of the previous OSD? >> I'd probably try to check some things like >> >> cephadm ceph-volume inventory >> ceph device ls-by-host <host> >> >> Regards, >> Eugen >> >> Zitat von Zakhar Kirpichenko <zakhar@xxxxxxxxx>: >> >> > Hi, >> > >> > We had a physical drive malfunction in one of our Ceph OSD hosts >> managed by >> > cephadm (Ceph 16.2.14). I have removed the drive from the system, and >> the >> > kernel no longer sees it: >> > >> > ceph03 ~]# ls -al /dev/sde >> > ls: cannot access '/dev/sde': No such file or directory >> > >> > I have removed the corresponding OSD from cephadm, crush map, etc. For >> all >> > intents and purposes that OSD and its block device no longer exist: >> > >> > root@ceph01:/# ceph orch ps | grep osd.26 >> > root@ceph01:/# ceph osd tree| grep 26 >> > root@ceph01:/# ceph orch device ls | grep -E "ceph03.*sde" >> > >> > None of the above commands return anything. Cephadm correctly sees 8 >> > remaining OSDs on the host: >> > >> > root@ceph01:/# ceph orch ls | grep ceph03_c >> > osd.ceph03_combined_osd 8 33s ago 2y ceph03 >> > >> > Unfortunately, cephadm appears to be trying to apply a spec to host >> ceph03 >> > including the disk that is now missing: >> > >> > RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host >> > --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume >> > --privileged --group-add=disk --init -e CONTAINER_IMAGE= >> > >> quay.io/ceph/ceph@sha256:843f112990e6489362c625229c3ea3d90b8734bd5e14e0aeaf89942fbb980a8b >> > -e NODE_NAME=ceph03 -e CEPH_USE_RANDOM_NONCE=1 -e >> > CEPH_VOLUME_OSDSPEC_AFFINITY=ceph03_combined_osd -e >> > CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v >> > /var/run/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/run/ceph:z -v >> > /var/log/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/log/ceph:z -v >> > >> /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z >> > -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v >> > /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v >> > /tmp/ceph-tmpc7b33pf0:/etc/ceph/ceph.conf:z -v >> > /tmp/ceph-tmpq45nkmd6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z >> > >> quay.io/ceph/ceph@sha256:843f112990e6489362c625229c3ea3d90b8734bd5e14e0aeaf89942fbb980a8b >> > lvm batch --no-auto /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde >> /dev/sdf >> > /dev/sdg /dev/sdh /dev/sdi --db-devices /dev/nvme0n1 /dev/nvme1n1 --yes >> > --no-systemd >> > >> > Note that `lvm batch` includes the missing drive, /dev/sde. This fails >> > because the drive no longer exists. Other than this cephadm ceph-volume >> > thingy, the cluster is healthy.How can I tell cephadm that it should >> stop >> > trying to use /dev/sde, which no longer exists, without affecting other >> > OSDs on the host? >> > >> > I would very much appreciate any advice or pointers. >> > >> > Best regards, >> > Zakhar >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |