Am 09.03.2025 um 15:22 schrieb Moritz Baumann:
I dont want to sound stupid, but how do I get the log?
Am 09.03.2025 um 11:11 schrieb Eugen Block:
ceph versions
ceph01[22]:~# cephadm shell -- ceph log last cephadm
Inferring fsid 4cfb1138-d2a5-11ef-87eb-1c34da408e52
Inferring config
/var/lib/ceph/4cfb1138-d2a5-11ef-87eb-1c34da408e52/mon.ceph01/config
Not using image
'f2efb0401a30ec7eda97b6da76b314bd081fcb910cc5dcd826bc7c72c9dfdd7d' as
it's not in list of non-dangling images with ceph=True label
2025-02-25T08:42:54.590667+0000 mgr.ceph01.ombogc (mgr.180813) 1706 :
cephadm [INF] maintenance mode request for ceph05.inf.ethz.ch has SET
the noout group
2025-02-25T08:43:38.380838+0000 mgr.ceph01.ombogc (mgr.180813) 1729 :
cephadm [ERR] Detected invalid grafana certificate on host
ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',
'no start line')]
2025-02-25T08:44:40.343420+0000 mgr.ceph01.ombogc (mgr.180813) 1762 :
cephadm [INF] Adjusting osd_memory_target on ceph04.inf.ethz.ch to
33423M
2025-02-25T08:44:40.860933+0000 mgr.ceph01.ombogc (mgr.180813) 1764 :
cephadm [ERR] Detected invalid grafana certificate on host
ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',
'no start line')]
2025-02-25T08:45:41.993564+0000 mgr.ceph01.ombogc (mgr.180813) 1797 :
cephadm [ERR] Detected invalid grafana certificate on host
ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',
'no start line')]
2025-02-25T08:46:42.758717+0000 mgr.ceph01.ombogc (mgr.180813) 1830 :
cephadm [ERR] Detected invalid grafana certificate on host
ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',
'no start line')]
2025-02-25T08:47:44.190946+0000 mgr.ceph01.ombogc (mgr.180813) 1864 :
cephadm [ERR] Unable to reach remote host ceph05.inf.ethz.ch. SSH
connection closed
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 245, in
_execute_command
r = await conn.run(str(rcmd), input=stdin)
File "/lib/python3.9/site-packages/asyncssh/connection.py", line
4250, in run
process = await self.create_process(*args, **kwargs) # type: ignore
File "/lib/python3.9/site-packages/asyncssh/connection.py", line
4128, in create_process
chan, process = await self.create_session(
File "/lib/python3.9/site-packages/asyncssh/connection.py", line
4018, in create_session
chan = SSHClientChannel(self, self._loop, encoding, errors,
File "/lib/python3.9/site-packages/asyncssh/channel.py", line 1085,
in __init__
super().__init__(conn, loop, encoding, errors, window, max_pktsize)
File "/lib/python3.9/site-packages/asyncssh/channel.py", line 140,
in __init__
self._recv_chan: Optional[int] = conn.add_channel(self)
File "/lib/python3.9/site-packages/asyncssh/connection.py", line
1307, in add_channel
raise ChannelOpenError(OPEN_CONNECT_FAILED,
asyncssh.misc.ChannelOpenError: SSH connection closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in
wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/cephadm/module.py", line 162, in wrapper
return func(*args, **kwargs)
File "/usr/share/ceph/mgr/cephadm/module.py", line 2224, in
exit_host_maintenance
outs, errs, _code = self.wait_async(
File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async
return self.event_loop.get_result(coro, timeout)
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result
return future.result(timeout)
File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in
result
return self.__get_result()
File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in
__get_result
raise self._exception
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1669, in
_run_cephadm
python = await self.mgr.ssh._check_execute_command(host, cmd,
addr=addr)
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 300, in
_check_execute_command
out, err, code = await self._execute_command(host, cmd, stdin,
addr, log_command)
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 253, in
_execute_command
raise HostConnectionError(f'Unable to reach remote host {host}.
{str(e)}', host, address)
cephadm.ssh.HostConnectionError: Unable to reach remote host
ceph05.inf.ethz.ch. SSH connection closed
2025-02-25T08:47:44.762770+0000 mgr.ceph01.ombogc (mgr.180813) 1865 :
cephadm [ERR] Detected invalid grafana certificate on host
ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',
'no start line')]
2025-02-25T08:47:55.631018+0000 mgr.ceph01.ombogc (mgr.180813) 1874 :
cephadm [INF] exit maintenance request has UNSET for the noout group
on host ceph05.inf.ethz.ch
2025-02-25T08:48:47.623644+0000 mgr.ceph01.ombogc (mgr.180813) 1901 :
cephadm [INF] Adjusting osd_memory_target on ceph05.inf.ethz.ch to
32911M
2025-02-25T08:48:47.827026+0000 mgr.ceph01.ombogc (mgr.180813) 1903 :
cephadm [ERR] Detected invalid grafana certificate on host
ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',
'no start line')]
2025-02-25T08:49:21.108932+0000 mgr.ceph01.ombogc (mgr.180813) 1925 :
cephadm [INF] Schedule redeploy daemon grafana.ceph05
2025-02-25T08:49:21.292879+0000 mgr.ceph01.ombogc (mgr.180813) 1926 :
cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates
2025-02-25T08:49:21.328348+0000 mgr.ceph01.ombogc (mgr.180813) 1929 :
cephadm [INF] Deploying daemon grafana.ceph05 on ceph05.inf.ethz.ch
2025-02-25T09:07:00.824841+0000 mgr.ceph01.ombogc (mgr.180813) 2461 :
cephadm [INF] Detected new or changed devices on ceph05.inf.ethz.ch
2025-02-25T09:07:01.376245+0000 mgr.ceph01.ombogc (mgr.180813) 2462 :
cephadm [INF] Detected new or changed devices on ceph04.inf.ethz.ch
2025-02-28T06:18:29.439683+0000 mgr.ceph01.ombogc (mgr.180813) 137418
: cephadm [ERR] [28/Feb/2025:06:18:29] ENGINE
ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not
appear to be an IPv4 or IPv6 address')
Traceback (most recent call last):
File "/lib/python3.9/site-packages/cheroot/server.py", line 1281,
in communicate
req.parse_request()
File "/lib/python3.9/site-packages/cheroot/server.py", line 714, in
parse_request
success = self.read_request_line()
File "/lib/python3.9/site-packages/cheroot/server.py", line 815, in
read_request_line
scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri)
File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit
_check_bracketed_host(bracketed_host)
File "/lib64/python3.9/urllib/parse.py", line 453, in
_check_bracketed_host
ip = ipaddress.ip_address(hostname) # Throws Value Error if not
IPv6 or IPv4
File "/lib64/python3.9/ipaddress.py", line 53, in ip_address
raise ValueError(f'{address!r} does not appear to be an IPv4 or
IPv6 address')
ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not
appear to be an IPv4 or IPv6 address
2025-02-28T06:18:29.441452+0000 mgr.ceph01.ombogc (mgr.180813) 137419
: cephadm [ERR] [28/Feb/2025:06:18:29] ENGINE
ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not
appear to be an IPv4 or IPv6 address')
Traceback (most recent call last):
File "/lib/python3.9/site-packages/cheroot/server.py", line 1281,
in communicate
req.parse_request()
File "/lib/python3.9/site-packages/cheroot/server.py", line 714, in
parse_request
success = self.read_request_line()
File "/lib/python3.9/site-packages/cheroot/server.py", line 815, in
read_request_line
scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri)
File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit
_check_bracketed_host(bracketed_host)
File "/lib64/python3.9/urllib/parse.py", line 453, in
_check_bracketed_host
ip = ipaddress.ip_address(hostname) # Throws Value Error if not
IPv6 or IPv4
File "/lib64/python3.9/ipaddress.py", line 53, in ip_address
raise ValueError(f'{address!r} does not appear to be an IPv4 or
IPv6 address')
ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not
appear to be an IPv4 or IPv6 address
2025-03-07T06:17:28.282574+0000 mgr.ceph01.ombogc (mgr.180813) 447472
: cephadm [ERR] [07/Mar/2025:06:17:28] ENGINE
ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not
appear to be an IPv4 or IPv6 address')
Traceback (most recent call last):
File "/lib/python3.9/site-packages/cheroot/server.py", line 1281,
in communicate
req.parse_request()
File "/lib/python3.9/site-packages/cheroot/server.py", line 714, in
parse_request
success = self.read_request_line()
File "/lib/python3.9/site-packages/cheroot/server.py", line 815, in
read_request_line
scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri)
File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit
_check_bracketed_host(bracketed_host)
File "/lib64/python3.9/urllib/parse.py", line 453, in
_check_bracketed_host
ip = ipaddress.ip_address(hostname) # Throws Value Error if not
IPv6 or IPv4
File "/lib64/python3.9/ipaddress.py", line 53, in ip_address
raise ValueError(f'{address!r} does not appear to be an IPv4 or
IPv6 address')
ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not
appear to be an IPv4 or IPv6 address
2025-03-09T07:36:00.522250+0000 mgr.ceph01.ombogc (mgr.180813) 536135
: cephadm [ERR] ALERT: Cannot stop active Mgr daemon, Please switch
active Mgrs with 'ceph mgr fail ceph01.ombogc'
Note: Warnings can be bypassed with the --force flag
Traceback (most recent call last):
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in
wrapper
return OrchResult(f(*args, **kwargs))
File "/usr/share/ceph/mgr/cephadm/module.py", line 162, in wrapper
return func(*args, **kwargs)
File "/usr/share/ceph/mgr/cephadm/module.py", line 2165, in
enter_host_maintenance
raise OrchestratorError(
orchestrator._interface.OrchestratorError: ALERT: Cannot stop active
Mgr daemon, Please switch active Mgrs with 'ceph mgr fail ceph01.ombogc'
Note: Warnings can be bypassed with the --force flag
