I had a situation like this, and the only operation that solved was a full reboot of the cluster (it was due the a watchdog alarm), but when the cluster return, the stray osds were gone. On Fri, 28 Jan 2022, 19:32 Adam King, <adking@xxxxxxxxxx> wrote: > Hello Vlad, > > Just some insight into how CEPHADM_STRAY_DAEMON works: This health warning > is specifically designed to point out daemons in the cluster that cephadm > is not aware of/in control of. It does this by comparing the daemons it has > cached info on (this cached info is what you see in "ceph orch ps") with > the return value of a core mgr function designed to list the servers in the > cluster and what daemons are on them. This function, from cephadm's point > of view, is a bit of a black box (by design, as it is meant to find > daemons cephadm is not aware of/in control of). If you'd like to see a > rough estimate of what that looks like I'd check the output of "ceph node > ls" (you may see your non-existent osds listed there). This means, a daemon > that does not exist that cephadm is falsely reporting as a stray daemon > cannot typically be resolved through "ceph orch . . ." commands. In the > past I've found sometimes just doing a mgr failover ("ceph mgr fail") will > clear this in the case of false reports so that's what I'd try first. If > that doesn't, I'd maybe try checking if the osd is till listed in the crush > map and if so, remove it (first step in > > https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/#removing-the-osd > I think). It's possible that the reason the daemon rm commands hung is one > of the cleanup operations cephadm was trying to run under the hood when > removing the osd hung and so the osd is still believed to be present by the > cluster. > > - Adam > > On Fri, Jan 28, 2022 at 11:28 AM Vladimir Brik < > vladimir.brik@xxxxxxxxxxxxxxxx> wrote: > > > Hello > > > > I needed to permanently remove two drives from my pool so I > > ran "ceph orch daemon rm XXX". The command hung for both > > OSDs, but the daemons were removed. I then purged the two OSDs. > > > > Now ceph status is complaining about them with > > CEPHADM_STRAY_DAEMON, but the daemons aren't running and are > > not showing up in ceph orch ps. If I try to "daemon rm" > > again I get Error EINVAL: Unable to find daemon(s). > > > > Anybody have an idea about what could have happened or how > > to stop ceph status from listing the non-existing daemons as > > stray? > > > > > > Thanks, > > > > Vlad > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx