It depends on your available resources, but I really do recommend
destroying and re-creating that OSD. If you have to spin up a VM and
set up a temporary OSD just to keep the overall system happy, even that
is a small price to pay.
As I said, you can't unlink/disable the container systemd, because it's
created and re-recreated dynamically. The non-container systemd unit is
easier to deal with, since it's static but if you're sharing resources
like I was, I still think it safer if you migrate everything out of
that OSD, totally destroy all traces of both realizations of it, then
re-create and migrate back. Ceph can do that, and although it can take
a while to do the migration, it's well worth the effort, Ceph is very
good at keeping data safe, but I'd not abuse the privilege.
Oh yeah, you have multiple OSDs with that problem. Do one at a time and
you'll go faster, Doing a mass shovel of data in and out of one OSD
takes long enough, but trying to run the process in parallel may well
bog down the whole Ceph network.
As I recall, as part of my own fumbling approach, I first stopped and
masked the osd-style systemd unit. But as I said, different subsystems
look in different places to monitor OSDs and I'd get conflicting status
reports. So while in theory, you could brute-force remove the systemd
unit and the /car/lib/ceph/osd.x directory subtree (preferably while
the container is also offline, I think it cleaner and safer overall to
simply go back to a blank slate.
On Fri, 2024-08-16 at 21:05 +0000, Dan O'Brien wrote:
> OK... I've been in the Circle of Hell where systemd lives and I
> *THINK* I have convinced myself I'm OK. I *REALLY* don't want to
> trash and rebuild the OSDs.
>
> In the manpage for systemd.unit, I found
>
> UNIT GARBAGE COLLECTION
> The system and service manager loads a unit's configuration
> automatically when a unit is referenced for the first time. It will
> automatically unload the unit configuration and state again when the
> unit is not needed anymore ("garbage collection").
>
> I've disabled the systemd units (which removes the symlink from the
> target) for the non-cephadm OSDs I created by mistake and I'm PRETTY
> SURE if I wait long enough (or reboot) that I won't see them any
> more, since there won't be a unit for systemd to care about.
>
> I *WILL* have to clean up /var/lib/ceph/osd eventually. I tried just
> now, but it says "device busy." I think that's because there's some
> OTHER systemd cruft that shows a mount:
> [root@ceph02 ~]# systemctl --all | grep ceph | grep mount
> var-lib-ceph-osd-ceph\x2d11.mount loaded active
> mounted /var/lib/ceph/osd/ceph-11
> var-lib-ceph-osd-ceph\x2d25.mount loaded active
> mounted /var/lib/ceph/osd/ceph-25
> var-lib-ceph-osd-ceph\x2d9.mount loaded active
> mounted /var/lib/ceph/osd/ceph-9
>
> When things settle down, I *MIGHT* put in a RFE to change the default
> for ceph-volume to --no-systemd to save someone else from this
> anguish.
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
