Hi,
thank you Eugen and Tim.
> did you fail the mgr?
I think I didn't.
> Or how exactly did you drain that host?
```
cephadm shell -- ceph orch host drain cephfs-cluster-node-2
cephadm shell -- ceph orch host rm cephfs-cluster-node-2
```
> `ceph config-key get mgr/cephadm/host.cephfs-cluster-node-2 | jq`
Error ENOENT:
> Is the VM still reachable?
No. It wasn't reachable for majority of the time the warning was displayed.
> Remove the directory so cephadm forgets about it.
What directory do you mean?
I have since made the VM reachable again and removed it again a few times.
I even tried removing the mon.X that was running on X while (and ceph knew
about it) while ceph was complaining that mon.X is running on
cephfs-cluster-node-2.
I disabled and re-enabled the stray warnings.
I found some custom config pertaining to cephfs-cluster-node-0 (which
`health detail` doesn't report as stray; namely: `osd
host:cephfs-cluster-node-0 basic osd_memory_target 5829100748`).
I tried enabling mgr module cli, to query list_servers, which I believe is
behind what `health detail` queries, but ceph refused to enable that module.
So I queried restful `/server`.
I ran `grep -R cephfs-cluster-node-2` in /var/lib/ceph*/ and found a few
logs and binary files that matched, among them some of the osd dirs.
I tried to drop one of the osds and re-add it later.
None of these things seemed to have any immediate effect.
But suddenly after days of warning about stray host with a daemon (that
does not even belong to that host), the warning disappeared.
I am afraid I may have done some other things out of desperation that I do
not recall at the moment.
I hope the stray host + stray daemon warning does not return.
Thanks again!
Jakub
On Mon, 18 Nov 2024 at 16:11, Eugen Block <eblock@xxxxxx> wrote:
> Hi,
>
> just to be safe, did you fail the mgr? If not, try 'ceph mgr fail' and
> see if it still reports that information. It sounds like you didn't
> clean up your virtual MON after you drained the OSDs. Or how exactly
> did you drain that host? If you run 'ceph orch host drain {host}' the
> orchestrator will remove all daemons, not only OSDs. I assume that
> there are still some entries in the ceph config-key database where the
> 'device ls-by-host' output comes from, I haven't verified that though.
> Is this key present?
>
> ceph config-key get mgr/cephadm/host.cephfs-cluster-node-2 | jq
>
> Is the VM still reachable? If it is, check the MON directory under
> /var/lib/ceph/{FSID}/mon.X and remove it. You can also first check
> with 'cephadm ls --no-detail' if cephadm believes that there's a MON
> daemon located. Remove the directory so cephadm forgets about it.
>
> Zitat von Jakub Daniel <jakub.daniel@xxxxxxxxx>:
>
> > I have found these two threads
> >
> >
> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/VHZ7IJ7PAL7L2INLSHNVYY7V7ZCXD46G/#TSWERUMAEEGZPSYXG6PSS4YMRXPP3L63
> >
> >
> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/NG5QVRTVCLLYNLK56CSYLIPE4WBFXS5U/#HJDBAJFX27KATC4WV2MKGLVGLN2HTWWD
> >
> > but I didn't exactly figure out what to do. I've since found remnants of
> > removed cephfs-cluster-node-{0,1,2} in crush map buckets. Which I removed
> > with no effect on health detail. I found out that ceph dashboard lists
> the
> > non-existent cephfs-cluster-node-2 among ceph hosts (while orch host ls
> > doesn't). On the other hand, device ls-by-host cephfs-cluster-node-2
> lists
> > an entry with device name coinciding with a drive on the live X host,
> with
> > daemons listing the mon.X. Meanwhile device ls-by-host X lists among
> other
> > things the exact same entry with the difference that in the dev column
> > there is the actual device nvme0n1 whereas with cephfs-cluster-node-2 the
> > column was empty
> >
> > root@X:~# cephadm shell -- ceph device ls-by-host cephfs-cluster-node-2
> > DEVICE DEV DAEMONS EXPECTED FAILURE
> > Samsung_SSD_970_PRO_1TB_S462NF0M310269L mon.X
> >
> > root@X:~# cephadm shell -- ceph device ls-by-host X
> > DEVICE DEV DAEMONS
> > EXPECTED FAILURE
> > Samsung_SSD_850_EVO_250GB_S2R6NX0J423123P sdb osd.3
> > Samsung_SSD_860_EVO_1TB_S3Z9NY0M431048H mon.Y
> > Samsung_SSD_970_PRO_1TB_S462NF0M310269L nvme0n1 mon.X
> >
> > This output is confusing since it lists mon.Y also when asked for host X.
> >
> > I will continue investigating. If anyone has any hints what to try or
> where
> > to look, I would be very grateful.
> >
> > Jakub
> >
> > On Sun, 17 Nov 2024, 15:34 Tim Holloway, <timh@xxxxxxxxxxxxx> wrote:
> >
> >> I think I can count 5 sources that Ceph can query to
> >> report/display/control its resources.
> >>
> >> 1. The /dec/ceph/ceph.conf file. Mostly supplanted bt the Ceph
> >> configuration database.
> >>
> >> 2. The ceph configuration database. A namelesskey/value store internal
> >> to a ceph filesystem. It's distributed (no fixed location), accessed by
> >> Ceph commands and APIs/
> >>
> >> 3. Legacy Ceph resources/ Stuff found under a host's /var/lib/ceph
> >> directory.
> >>
> >> 4. Managed Ceph resources. Stuff found under a host's
> >> /var/lib/ceph/{fsid} diirectory.
> >>
> >> 5. The live machine state of Ceph. Since this not only can vary from
> >> host to host, but also service to service, I don't think that this is
> >> considered to be an authoritative source of information.
> >>
> >> Compounding this is that current releases of Ceph can all too easily
> >> end up in a "forbidden" state where you may have, for example a legacy
> >> OSD.6 and a managed OSD.6 on the same host. In such a case, system is
> >> generally operable, but functionally corrupt and ideally should be
> >> corrected to remove the redundant resource.
> >>
> >> The real issue is that depending on what Ceph interface you're querying
> >> (or "ceph health" is querying!), you don't always get your answer from
> >> a single authoritative source, so you'll get conflicting results and
> >> annoying error reports. The "stray daemon" condition is an especially
> >> egregious example of this, and it's not only possible because of a
> >> false detection from one of the above sources, but also, I think can
> >> come from "dead" daemons being referenced in CRUSH.
> >>
> >> You might want to run through this lists's history for "phantom host"
> >> postings made by me back around this past June because I was absolutely
> >> plagued with them. Eugen Block helped me eventually purge them all.
> >>
> >> Regards,
> >> Tim
> >>
> >> On Sat, 2024-11-16 at 21:42 +0100, Jakub Daniel wrote:
> >>> Hello,
> >>>
> >>> I'm pretty new to ceph deployment. I have setup my first cephfs
> >>> cluster
> >>> using cephadm. Initially, I deployed ceph in 3 virtualbox instances
> >>> that I
> >>> called cephfs-cluster-node-{0, 1, 2} just to test things. Later, I
> >>> added 5
> >>> more real hardware nodes. Later I decided I'd remove the
> >>> virtualboxes, so I
> >>> drained the osds and removed the hosts. Suddenly, ceph status detail
> >>> started reporting
> >>>
> >>> HEALTH_WARN 1 stray host(s) with 1 daemon(s) not managed by cephadm
> >>> [WRN]
> >>> CEPHADM_STRAY_HOST: 1 stray host(s) with 1 daemon(s) not managed by
> >>> cephadm
> >>> stray host cephfs-cluster-node-2 has 1 stray daemons: ['mon.X']
> >>>
> >>> The cephfs-cluster-node-2 is no longer listed among hosts, it is (and
> >>> has
> >>> been for tens of hours) offline (powered down). The mon.X doesn't
> >>> even
> >>> belong to that node, it is one of the real hardware nodes. I am
> >>> unaware of
> >>> mon.X ever running on cephfs-cluster-node-2 (never noticed it among
> >>> systemd
> >>> units).
> >>>
> >>> Where does cephadm shell -- ceph status detail come to the conclusion
> >>> there
> >>> is something stray? How can I address this?
> >>>
> >>> Thank you for any insights
> >>> Jakub
> >>> _______________________________________________
> >>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>
> >>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
