Recent upgrade from 18.2 to 19.2, upgrade went fine. Since the upgrade and a manager fail over, I can no longer run orchestrator commands. The only error I can find on an active manager daemon is the following, or it is the only one that stands out.
2025-01-06T18:48:41.698+0000 7fcf42b99640 -1 mgr load Failed to construct class in 'cephadm'
2025-01-06T18:48:41.698+0000 7fcf42b99640 -1 mgr load Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/module.py", line 667, in __init__
self.cert_key_store.load()
File "/usr/share/ceph/mgr/cephadm/inventory.py", line 2073, in load
self.known_certs[entity] = json.loads(v)
File "/lib64/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/lib64/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/lib64/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2025-01-06T18:48:41.702+0000 7fcf42b99640 -1 mgr operator() Failed to run module in active mode ('cephadm')
For any thing to really work in the dashboard I must have the balancer off. While the balance is off I can make changes in dashboard to the orchestrator, and it doesn't give me any trouble. When trying different commands from a ceph node say "ceph cephadm config-check status" it returns "Error ENOTSUP: Module 'cephadm' is not enabled/loaded (required by command 'cephadm config-check status'): use `ceph mgr module enable cephadm` to enable it". Running "ceph mgr module enable cephadm" returns "module 'cephadm' is already enabled". I really don't know where to look or what to try to resolve this. Any "ceph orch" command results in "Error ENOENT: Module not found"
I don't know that my issue is related to https://tracker.ceph.com/issues/68657, but maybe it is.
I have tried the following.
Manually adding a new mgr daemon on different node, it starts runs the dashboard fine, but things are still not functional.
Failed the mgr several times.
Disabled/Enabled balancer.
Disabled/Enabled mgr modules.
Disabled/Enabled dashboard.
All physical nodes are running Debian 12.
Frank Frampton
Senior Network Services Administrator
Salt Lake City School District
Desk: (801) 578-8223
Follow the district: Facebook<https://www.facebook.com/slcschools> | Instagram<https://instagram.com/slcschools> | Twitter<https://twitter.com/slcschools> | www.slcschools.org<http://www.slcschools.org/>
Excellence and Equity: every student, every classroom, every day
Scanned By Microsoft EOP
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
