Re: ceph orch command not working anymore on squid (19.2.1)

[Eugen Block <eblock@xxxxxx>]

(Don't drop the list in your responses)
There's a lot going on in the cephadm.log, it's not clear which of  
these are relevant to the issue. This one looks bad though:

ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not  
appear to be an IPv4 or IPv6 address

Did you build this cluster with 19.2.1 or did you upgrade? Do you have  
the output from 'ceph orch host ls' lying around (I know, currently  
it's not working)? Are you using IPv6? I haven't seen this issue yet.

Zitat von Moritz Baumann <mo@xxxxxxxxxxxxx>:

> Am 09.03.2025 um 15:22 schrieb Moritz Baumann:
> > I dont want to sound stupid, but how do I get the log?
> >
> > Am 09.03.2025 um 11:11 schrieb Eugen Block:
> > > ceph versions
>
>
> ceph01[22]:~# cephadm shell -- ceph log last cephadm
> Inferring fsid 4cfb1138-d2a5-11ef-87eb-1c34da408e52
> Inferring config  
> /var/lib/ceph/4cfb1138-d2a5-11ef-87eb-1c34da408e52/mon.ceph01/config
> Not using image  
> 'f2efb0401a30ec7eda97b6da76b314bd081fcb910cc5dcd826bc7c72c9dfdd7d'  
> as it's not in list of non-dangling images with ceph=True label
> 2025-02-25T08:42:54.590667+0000 mgr.ceph01.ombogc (mgr.180813) 1706  
> : cephadm [INF] maintenance mode request for ceph05.inf.ethz.ch has  
> SET the noout group
> 2025-02-25T08:43:38.380838+0000 mgr.ceph01.ombogc (mgr.180813) 1729  
> : cephadm [ERR] Detected invalid grafana certificate on host  
> ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',  
> 'no start line')]
> 2025-02-25T08:44:40.343420+0000 mgr.ceph01.ombogc (mgr.180813) 1762  
> : cephadm [INF] Adjusting osd_memory_target on ceph04.inf.ethz.ch to  
> 33423M
> 2025-02-25T08:44:40.860933+0000 mgr.ceph01.ombogc (mgr.180813) 1764  
> : cephadm [ERR] Detected invalid grafana certificate on host  
> ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',  
> 'no start line')]
> 2025-02-25T08:45:41.993564+0000 mgr.ceph01.ombogc (mgr.180813) 1797  
> : cephadm [ERR] Detected invalid grafana certificate on host  
> ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',  
> 'no start line')]
> 2025-02-25T08:46:42.758717+0000 mgr.ceph01.ombogc (mgr.180813) 1830  
> : cephadm [ERR] Detected invalid grafana certificate on host  
> ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',  
> 'no start line')]
> 2025-02-25T08:47:44.190946+0000 mgr.ceph01.ombogc (mgr.180813) 1864  
> : cephadm [ERR] Unable to reach remote host ceph05.inf.ethz.ch. SSH  
> connection closed
> Traceback (most recent call last):
>   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 245, in _execute_command
>     r = await conn.run(str(rcmd), input=stdin)
>   File "/lib/python3.9/site-packages/asyncssh/connection.py", line  
> 4250, in run
>     process = await self.create_process(*args, **kwargs) # type: ignore
>   File "/lib/python3.9/site-packages/asyncssh/connection.py", line  
> 4128, in create_process
>     chan, process = await self.create_session(
>   File "/lib/python3.9/site-packages/asyncssh/connection.py", line  
> 4018, in create_session
>     chan = SSHClientChannel(self, self._loop, encoding, errors,
>   File "/lib/python3.9/site-packages/asyncssh/channel.py", line  
> 1085, in __init__
>     super().__init__(conn, loop, encoding, errors, window, max_pktsize)
>   File "/lib/python3.9/site-packages/asyncssh/channel.py", line 140,  
> in __init__
>     self._recv_chan: Optional[int] = conn.add_channel(self)
>   File "/lib/python3.9/site-packages/asyncssh/connection.py", line  
> 1307, in add_channel
>     raise ChannelOpenError(OPEN_CONNECT_FAILED,
> asyncssh.misc.ChannelOpenError: SSH connection closed
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper
>     return OrchResult(f(*args, **kwargs))
>   File "/usr/share/ceph/mgr/cephadm/module.py", line 162, in wrapper
>     return func(*args, **kwargs)
>   File "/usr/share/ceph/mgr/cephadm/module.py", line 2224, in  
> exit_host_maintenance
>     outs, errs, _code = self.wait_async(
>   File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async
>     return self.event_loop.get_result(coro, timeout)
>   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result
>     return future.result(timeout)
>   File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result
>     return self.__get_result()
>   File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in  
> __get_result
>     raise self._exception
>   File "/usr/share/ceph/mgr/cephadm/serve.py", line 1669, in _run_cephadm
>     python = await self.mgr.ssh._check_execute_command(host, cmd, addr=addr)
>   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 300, in  
> _check_execute_command
>     out, err, code = await self._execute_command(host, cmd, stdin,  
> addr, log_command)
>   File "/usr/share/ceph/mgr/cephadm/ssh.py", line 253, in _execute_command
>     raise HostConnectionError(f'Unable to reach remote host {host}.  
> {str(e)}', host, address)
> cephadm.ssh.HostConnectionError: Unable to reach remote host  
> ceph05.inf.ethz.ch. SSH connection closed
> 2025-02-25T08:47:44.762770+0000 mgr.ceph01.ombogc (mgr.180813) 1865  
> : cephadm [ERR] Detected invalid grafana certificate on host  
> ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',  
> 'no start line')]
> 2025-02-25T08:47:55.631018+0000 mgr.ceph01.ombogc (mgr.180813) 1874  
> : cephadm [INF] exit maintenance request has UNSET for the noout  
> group on host ceph05.inf.ethz.ch
> 2025-02-25T08:48:47.623644+0000 mgr.ceph01.ombogc (mgr.180813) 1901  
> : cephadm [INF] Adjusting osd_memory_target on ceph05.inf.ethz.ch to  
> 32911M
> 2025-02-25T08:48:47.827026+0000 mgr.ceph01.ombogc (mgr.180813) 1903  
> : cephadm [ERR] Detected invalid grafana certificate on host  
> ceph05.inf.ethz.ch: Invalid certificate key: [('PEM routines', '',  
> 'no start line')]
> 2025-02-25T08:49:21.108932+0000 mgr.ceph01.ombogc (mgr.180813) 1925  
> : cephadm [INF] Schedule redeploy daemon grafana.ceph05
> 2025-02-25T08:49:21.292879+0000 mgr.ceph01.ombogc (mgr.180813) 1926  
> : cephadm [INF] Regenerating cephadm self-signed grafana TLS  
> certificates
> 2025-02-25T08:49:21.328348+0000 mgr.ceph01.ombogc (mgr.180813) 1929  
> : cephadm [INF] Deploying daemon grafana.ceph05 on ceph05.inf.ethz.ch
> 2025-02-25T09:07:00.824841+0000 mgr.ceph01.ombogc (mgr.180813) 2461  
> : cephadm [INF] Detected new or changed devices on ceph05.inf.ethz.ch
> 2025-02-25T09:07:01.376245+0000 mgr.ceph01.ombogc (mgr.180813) 2462  
> : cephadm [INF] Detected new or changed devices on ceph04.inf.ethz.ch
> 2025-02-28T06:18:29.439683+0000 mgr.ceph01.ombogc (mgr.180813)  
> 137418 : cephadm [ERR] [28/Feb/2025:06:18:29] ENGINE  
> ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not  
> appear to be an IPv4 or IPv6 address')
> Traceback (most recent call last):
>   File "/lib/python3.9/site-packages/cheroot/server.py", line 1281,  
> in communicate
>     req.parse_request()
>   File "/lib/python3.9/site-packages/cheroot/server.py", line 714,  
> in parse_request
>     success = self.read_request_line()
>   File "/lib/python3.9/site-packages/cheroot/server.py", line 815,  
> in read_request_line
>     scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri)
>   File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit
>     _check_bracketed_host(bracketed_host)
>   File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host
>     ip = ipaddress.ip_address(hostname) # Throws Value Error if not  
> IPv6 or IPv4
>   File "/lib64/python3.9/ipaddress.py", line 53, in ip_address
>     raise ValueError(f'{address!r} does not appear to be an IPv4 or  
> IPv6 address')
> ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not  
> appear to be an IPv4 or IPv6 address
>
> 2025-02-28T06:18:29.441452+0000 mgr.ceph01.ombogc (mgr.180813)  
> 137419 : cephadm [ERR] [28/Feb/2025:06:18:29] ENGINE  
> ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not  
> appear to be an IPv4 or IPv6 address')
> Traceback (most recent call last):
>   File "/lib/python3.9/site-packages/cheroot/server.py", line 1281,  
> in communicate
>     req.parse_request()
>   File "/lib/python3.9/site-packages/cheroot/server.py", line 714,  
> in parse_request
>     success = self.read_request_line()
>   File "/lib/python3.9/site-packages/cheroot/server.py", line 815,  
> in read_request_line
>     scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri)
>   File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit
>     _check_bracketed_host(bracketed_host)
>   File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host
>     ip = ipaddress.ip_address(hostname) # Throws Value Error if not  
> IPv6 or IPv4
>   File "/lib64/python3.9/ipaddress.py", line 53, in ip_address
>     raise ValueError(f'{address!r} does not appear to be an IPv4 or  
> IPv6 address')
> ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not  
> appear to be an IPv4 or IPv6 address
>
> 2025-03-07T06:17:28.282574+0000 mgr.ceph01.ombogc (mgr.180813)  
> 447472 : cephadm [ERR] [07/Mar/2025:06:17:28] ENGINE  
> ValueError('"\'xwork.MethodAccessor.denyMethodExecution\'" does not  
> appear to be an IPv4 or IPv6 address')
> Traceback (most recent call last):
>   File "/lib/python3.9/site-packages/cheroot/server.py", line 1281,  
> in communicate
>     req.parse_request()
>   File "/lib/python3.9/site-packages/cheroot/server.py", line 714,  
> in parse_request
>     success = self.read_request_line()
>   File "/lib/python3.9/site-packages/cheroot/server.py", line 815,  
> in read_request_line
>     scheme, authority, path, qs, fragment = urllib.parse.urlsplit(uri)
>   File "/lib64/python3.9/urllib/parse.py", line 510, in urlsplit
>     _check_bracketed_host(bracketed_host)
>   File "/lib64/python3.9/urllib/parse.py", line 453, in _check_bracketed_host
>     ip = ipaddress.ip_address(hostname) # Throws Value Error if not  
> IPv6 or IPv4
>   File "/lib64/python3.9/ipaddress.py", line 53, in ip_address
>     raise ValueError(f'{address!r} does not appear to be an IPv4 or  
> IPv6 address')
> ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not  
> appear to be an IPv4 or IPv6 address
>
> 2025-03-09T07:36:00.522250+0000 mgr.ceph01.ombogc (mgr.180813)  
> 536135 : cephadm [ERR] ALERT: Cannot stop active Mgr daemon, Please  
> switch active Mgrs with 'ceph mgr fail ceph01.ombogc'
> Note: Warnings can be bypassed with the --force flag
> Traceback (most recent call last):
>   File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 138, in wrapper
>     return OrchResult(f(*args, **kwargs))
>   File "/usr/share/ceph/mgr/cephadm/module.py", line 162, in wrapper
>     return func(*args, **kwargs)
>   File "/usr/share/ceph/mgr/cephadm/module.py", line 2165, in  
> enter_host_maintenance
>     raise OrchestratorError(
> orchestrator._interface.OrchestratorError: ALERT: Cannot stop active  
> Mgr daemon, Please switch active Mgrs with 'ceph mgr fail  
> ceph01.ombogc'
> Note: Warnings can be bypassed with the --force flag


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx