Hello,
Wall of text warning! :-)
I am playing with a new 5 cluster install (actually now 4 since the
fifth server isn't started yet), reading and following the doc and
watching what happens. Pretty weirdly multiple things went wrong which
almost never happened with ceph in the past many years, and I feel
cephadm is still not quite ready for prime time. I considered opening
issues but maybe it's better to filter user errors from problems first,
so I depend on your expertise and help.
I decided to use pacific and podman 3.x, and first obstacle was the
mismatching documetation but that's already
https://tracker.ceph.com/issues/55053 and I have resolved it by manual
fiddling, so I ought to be on real pacific now, fingers crossed.
Small problem with first stages of installing ceph on debian stable
(bullseye): ceph repo keys are probably in old format so they aren't
automagically recognised, require apt-key add manually.
Requirements neglect to mention a few small things, like examples need
curl and I am not sure whether missing firewalld caused some of the
pains later (at least I have seen complaining and error mentioning it,
but maybe that's just to b ignored).
I think the Doc doesn't specifically mention that hostnames (especially
when using bare hostnames) shall be in the mon network prefix, which is
important if the host has a separate management subnet from mon and osd
subnets.
Nevertheless default install of prometheus and the dashboard seems to
be broken because it tries to use IP instead of hostnames and fail on
the SSL cert not containing IP (as they shouldn't, really); this needs
a bit of googling and manual setting of URLs and things, otherwise the
dashboard is full of 500 errors without any explanation (since it only
gets logged into syslog in a very ugly and pretty verbose way).
My main problems are, however, started when adding new hosts. Everything
was connectable [ssh keys], requirements fulfilled (apart from the
mentioned firewalld), except some modifications _suggested_ a reboot,
which I didn't follow on two hosts (out of 4 online).
The result was this: two working servers (master and the one I rebooted
for a different reason), and two non-working ones. The nonworking mean
state where master thinks they have mon/crash/vol/mgr running, but
nothing was running because podman died with a mysterious error; for
the record:
ceph: stderr Error: cannot open sd-bus: No such file or directory:
OCI not found
it was caused by not rebooting, as it turned out. However after reboot
the daemons were started, except they didn't join the cluster. Also
ceph orch had little effect on their behaviour, while cephadm seemed to
be able to do actions there, starting or stopping daemons, without much
use (so it isn't a connectivity issue).
I have tried to `ceph orch host (drain|rm)` them, which more or less
succeeded. It have resulted "stopping" state of the daemons' view on
the master... which still blocks rm. Tried to remove them manually, with
success. Then host rm. Worked.
Adding back the hosts (`ceph orch add host <name> <ip> _admin`) gave
the result of all the daemons in "starting" on master view, while
visibly running on the host, and not joining the cluster. It stays
"starting" forever.
root@alai:~# ceph orch ps olhado --refresh
NAME HOST PORTS STATUS REFRESHED AGE MEM
USE MEM LIM VERSION IMAGE ID
crash.olhado olhado starting - -
- - <unknown> <unknown>
mon.olhado olhado starting - -
- 2048M <unknown> <unknown>
node-exporter.olhado olhado *:9100 starting - -
- - <unknown> <unknown>
The next problem is to find the logs. It used to be so easy, go to
/var/log/ceph/ and find ceph-mon.0.log and likes. Not anymore. I guess
mon logs into syslog, using the generic name "conmon", probably mixed
up with the other daemons? (I have seen in the docs that I could force
it to create logfiles again, didn't feel like yet another
reconfiguration.)
Anyway, I see no error in the host syslog from mon, it said:
Mar 25 11:55:40 olhado conmon[33551]: debug
2022-03-25T10:55:40.083+0000 7fa24f6b0700 1
mon.olhado@-1(synchronizing) e2 sync_obtain_latest_monmap
Mar 25 11:55:40 olhado conmon[33551]: debug
2022-03-25T10:55:40.083+0000 7fa24f6b0700 1
mon.olhado@-1(synchronizing) e2 sync_obtain_latest_monmap obtained
monmap e2
and the master says, every second:
Mar 25 14:19:24 alai conmon[822444]: debug
2022-03-25T13:19:24.729+0000 7f09c18df700 1 mon.alai@0(leader) e2
adding peer [v2:***:3300/0,v1:***:6789/0] to list of hints
but it's not in `ceph status` or `ceph mon dump` on the master (and
local `ceph` gets infinite wait, it's really hard to tell why since it
seems to be communicating with the master mon).
I have tried in due course putting them in maintenance mode since I
tried to get rid of "starting" state, first using ceph orch then by
local cephadm. These two does not seem to communicate well, which is
aptly demonstrated here:
root@alai:~# ceph health detail
HEALTH_WARN 1 host is in maintenance mode
[WRN] HOST_IN_MAINTENANCE: 1 host is in maintenance mode
olhado is in maintenance
root@alai:~# ceph orch host maintenance exit olhado
Error EINVAL: Host olhado is not in maintenance mode
I guess I need to find a combo when cephadm and ceph orch see the same
state, probably enabling maint by cephadm then trying to remove using
ceph orch.
I also had a state when `ceph orch *` simply got into forever waiting,
and it was only resolvable by stopping ceph-volume on the host using
podman stop. I was not able to see any log or debug which would have
explained to me what is happening and why.
Generally my problem is that I don't (yet?) see a simple way to see
what is happening:
- when ceph should deploy automagically but doesn't
- when cephadm/orch say "starting", "stopping" something and it doesn't
change
- where are the daemon logs and how to follow them easily (right now
I'm using `podman logs ...` but I am not sure it's the proper way)
- whether `ceph orch host rm` _really_ removes the host, so it can be
added later, or does it need manual deletion of something? It seems
it does since the aforementioned maintenance mode seems to have styed
throughout removal.
I will scratch the thing soon, there is no harm done, apart from the
time spent on watching (or rather guessing) what cephadm does.
I wonder whether some of these are bugs to be fixed (either cephadm or
documentation) or are they all preventable user errors.
Sorry for the wall of text.
Thanks,
Peter
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
