Hi! If is because old ssh client was replaced with asyncssh ( https://github.com/ceph/ceph/pull/51899) and only ported to reef, when will be added to quincy? For us is a blocker as we cannot move to cephadm anymore, as we planned for Q4. Is there a workarround? Thank you for your efforts! Paul On Sat, Oct 7, 2023 at 12:03 PM Paul JURCO <paul.jurco@xxxxxxxxx> wrote: > Resent due to moderation when using web interface. > > Hi ceph users, > We have a few clusters with quincy 17.2.6 and we are preparing to migrate > from ceph-deploy to cephadm for better management. > We are using Ubuntu20 with latest updates (latest openssh). > While testing the migration to cephadm on a test cluster with octopus (v16 > latest) we had no issues replacing ceph generated cert/key with our own CA > signed certs (ECDSA). > After upgrading to quincy the test cluster and test again the migration we > cannot add hosts due to the errors below, ssh access errors specified a > while ago in a tracker. > We use the following type of certs: > Type: ecdsa-sha2-nistp384-cert-v01@xxxxxxxxxxx user certificate > The certificate works everytime when using ssh client from shell to > connect to all hosts in the cluster. > We do a ceph mgr fail every time we replace cert/key so they are restarted. > > ----- cephadm logs from mgr ------ > Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to > 10.10.10.232, port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at > 10.10.10.232, port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: > 10.10.12.160, port 51870 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, > port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: > Permission denied > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection > Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/ssh.py", line 111, in redirect_log > Oct 06 09:23:27 ceph-m2 bash[1363]: yield > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/ssh.py", line 90, in _remote_connection > Oct 06 09:23:27 ceph-m2 bash[1363]: preferred_auth=['publickey'], > options=ssh_options) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect > Oct 06 09:23:27 ceph-m2 bash[1363]: 'Opening SSH connection to') > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect > Oct 06 09:23:27 ceph-m2 bash[1363]: await conn.wait_established() > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in > wait_established > Oct 06 09:23:27 ceph-m2 bash[1363]: await self._waiter > Oct 06 09:23:27 ceph-m2 bash[1363]: asyncssh.misc.PermissionDenied: > Permission denied > Oct 06 09:23:27 ceph-m2 bash[1363]: During handling of the above > exception, another exception occurred: > Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper > Oct 06 09:23:27 ceph-m2 bash[1363]: return OrchResult(f(*args, > **kwargs)) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/module.py", line 2810, in apply > Oct 06 09:23:27 ceph-m2 bash[1363]: results.append(self._apply(spec)) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/module.py", line 2558, in _apply > Oct 06 09:23:27 ceph-m2 bash[1363]: return > self._add_host(cast(HostSpec, spec)) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/module.py", line 1434, in _add_host > Oct 06 09:23:27 ceph-m2 bash[1363]: ip_addr = > self._check_valid_addr(spec.hostname, spec.addr) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/module.py", line 1415, in _check_valid_addr > Oct 06 09:23:27 ceph-m2 bash[1363]: error_ok=True, no_fsid=True)) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async > Oct 06 09:23:27 ceph-m2 bash[1363]: return > self.event_loop.get_result(coro) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result > Oct 06 09:23:27 ceph-m2 bash[1363]: return > asyncio.run_coroutine_threadsafe(coro, self._loop).result() > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result > Oct 06 09:23:27 ceph-m2 bash[1363]: return self.__get_result() > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result > Oct 06 09:23:27 ceph-m2 bash[1363]: raise self._exception > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/serve.py", line 1361, in _run_cephadm > Oct 06 09:23:27 ceph-m2 bash[1363]: await > self.mgr.ssh._remote_connection(host, addr) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in _remote_connection > Oct 06 09:23:27 ceph-m2 bash[1363]: raise > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib64/python3.6/contextlib.py", line 99, in __exit__ > Oct 06 09:23:27 ceph-m2 bash[1363]: self.gen.throw(type, value, > traceback) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/ssh.py", line 123, in redirect_log > Oct 06 09:23:27 ceph-m2 bash[1363]: raise HostConnectionError(msg, > host, addr) > Oct 06 09:23:27 ceph-m2 bash[1363]: cephadm.ssh.HostConnectionError: > Failed to connect to ceph-m1 (10.10.10.232). Permission denied > Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to > 10.10.10.232, port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at > 10.10.10.232, port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: > 10.10.12.160, port 51870 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, > port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: > Permission denied > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection > Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.081+0000 > 7f78d86d8700 -1 log_channel(cephadm) log [ERR] : Failed to connect to > ceph-m1 (10.10.10.232). Permission denied > Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to > 10.10.10.232, port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at > 10.10.10.232, port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: > 10.10.12.160, port 51870 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, > port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: > Permission denied > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection > Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/ssh.py", line 111, in redirect_log > Oct 06 09:23:27 ceph-m2 bash[1363]: yield > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/ssh.py", line 90, in _remote_connection > Oct 06 09:23:27 ceph-m2 bash[1363]: preferred_auth=['publickey'], > options=ssh_options) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib/python3.6/site-packages/asyncssh/connection.py", line 6804, in connect > Oct 06 09:23:27 ceph-m2 bash[1363]: 'Opening SSH connection to') > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib/python3.6/site-packages/asyncssh/connection.py", line 303, in _connect > Oct 06 09:23:27 ceph-m2 bash[1363]: await conn.wait_established() > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib/python3.6/site-packages/asyncssh/connection.py", line 2243, in > wait_established > Oct 06 09:23:27 ceph-m2 bash[1363]: await self._waiter > Oct 06 09:23:27 ceph-m2 bash[1363]: asyncssh.misc.PermissionDenied: > Permission denied > Oct 06 09:23:27 ceph-m2 bash[1363]: During handling of the above > exception, another exception occurred: > Oct 06 09:23:27 ceph-m2 bash[1363]: Traceback (most recent call last): > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper > Oct 06 09:23:27 ceph-m2 bash[1363]: return OrchResult(f(*args, > **kwargs)) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/module.py", line 2810, in apply > Oct 06 09:23:27 ceph-m2 bash[1363]: results.append(self._apply(spec)) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/module.py", line 2558, in _apply > Oct 06 09:23:27 ceph-m2 bash[1363]: return > self._add_host(cast(HostSpec, spec)) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/module.py", line 1434, in _add_host > Oct 06 09:23:27 ceph-m2 bash[1363]: ip_addr = > self._check_valid_addr(spec.hostname, spec.addr) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/module.py", line 1415, in _check_valid_addr > Oct 06 09:23:27 ceph-m2 bash[1363]: error_ok=True, no_fsid=True)) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async > Oct 06 09:23:27 ceph-m2 bash[1363]: return > self.event_loop.get_result(coro) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result > Oct 06 09:23:27 ceph-m2 bash[1363]: return > asyncio.run_coroutine_threadsafe(coro, self._loop).result() > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result > Oct 06 09:23:27 ceph-m2 bash[1363]: return self.__get_result() > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result > Oct 06 09:23:27 ceph-m2 bash[1363]: raise self._exception > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/serve.py", line 1361, in _run_cephadm > Oct 06 09:23:27 ceph-m2 bash[1363]: await > self.mgr.ssh._remote_connection(host, addr) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/ssh.py", line 96, in _remote_connection > Oct 06 09:23:27 ceph-m2 bash[1363]: raise > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/lib64/python3.6/contextlib.py", line 99, in __exit__ > Oct 06 09:23:27 ceph-m2 bash[1363]: self.gen.throw(type, value, > traceback) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/cephadm/ssh.py", line 123, in redirect_log > Oct 06 09:23:27 ceph-m2 bash[1363]: raise HostConnectionError(msg, > host, addr) > Oct 06 09:23:27 ceph-m2 bash[1363]: cephadm.ssh.HostConnectionError: > Failed to connect to ceph-m1 (10.10.10.232). Permission denied > Oct 06 09:23:27 ceph-m2 bash[1363]: Log: Opening SSH connection to > 10.10.10.232, port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connected to SSH server at > 10.10.10.232, port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Local address: > 10.10.12.160, port 51870 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Peer address: 10.10.10.232, > port 22 > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Beginning auth for user root > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Auth failed for user root > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Connection failure: > Permission denied > Oct 06 09:23:27 ceph-m2 bash[1363]: [conn=3] Aborting connection > Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.081+0000 > 7f78d86d8700 -1 mgr handle_command module 'orchestrator' command handler > threw exception: __init__() missing 2 required positional arguments: > > Oct 06 09:23:27 ceph-m2 bash[1363]: debug 2023-10-06T09:23:27.093+0000 > 7f78d86d8700 -1 mgr.server reply reply (22) Invalid argument Traceback > (most recent call last): > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command > Oct 06 09:23:27 ceph-m2 bash[1363]: return self.handle_command(inbuf, > cmd) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in > handle_command > Oct 06 09:23:27 ceph-m2 bash[1363]: return > dispatch[cmd['prefix']].call(self, cmd, inbuf) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/mgr_module.py", line 462, in call > Oct 06 09:23:27 ceph-m2 bash[1363]: return self.func(mgr, **kwargs) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in <lambda> > Oct 06 09:23:27 ceph-m2 bash[1363]: wrapper_copy = lambda *l_args, > **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper > Oct 06 09:23:27 ceph-m2 bash[1363]: return func(*args, **kwargs) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/orchestrator/module.py", line 356, in _add_host > Oct 06 09:23:27 ceph-m2 bash[1363]: return self._apply_misc([s], > False, Format.plain) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/orchestrator/module.py", line 1092, in _apply_misc > Oct 06 09:23:27 ceph-m2 bash[1363]: raise_if_exception(completion) > Oct 06 09:23:27 ceph-m2 bash[1363]: File > "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in > raise_if_exception > Oct 06 09:23:27 ceph-m2 bash[1363]: e = > pickle.loads(c.serialized_exception) > Oct 06 09:23:27 ceph-m2 bash[1363]: TypeError: __init__() missing 2 > required positional arguments: 'hostname' and 'addr' > ----- cephadm logs from mgr ------ > > > ----- sshd logs DEBUG3 level ------ > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: input_userauth_request: try > method publickey [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: userauth_pubkey: valid user > root querying public key ecdsa-sha2-nistp384 > AAAAE2VjZHNhLXNoYTItbmlzdHAzO------------ [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: userauth_pubkey: test pkalg > ecdsa-sha2-nistp384 pkblob ECDSA SHA256:m6Q0ZQVjjDLWxbmCn0hcGQ2---------- > [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_key_allowed entering > [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_send entering: > type 22 [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_key_allowed: waiting for > MONITOR_ANS_KEYALLOWED [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive_expect > entering: type 23 [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering > [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: monitor_read: checking > request 22 > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed entering > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed: > key_from_blob: 0x5568f0aa7880 > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: temporarily_use_uid: 0/0 > (e=0/0) > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: trying public key file > /etc/ssh/fake_authorized_keys > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: fd 5 clearing O_NONBLOCK > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: restore_uid: 0/0 > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_answer_keyallowed: > publickey authentication test: ECDSA key is not allowed > Oct 6 09:33:09 ceph-m1 sshd[57168]: Failed publickey for root from > 10.10.12.160 port 40854 ssh2: ECDSA > SHA256:m6Q0ZQVjjDLWxbmCn0hcGQ24gbpk------------- > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_send entering: > type 23 > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug2: userauth_pubkey: > authenticated 0 pkalg ecdsa-sha2-nistp384 [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: user_specific_delay: user > specific delay 0.000ms [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: ensure_minimum_time_since: > elapsed 8.263ms, delaying 8.080ms (requested 8.171ms) [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: userauth_finish: failure > partial=0 next methods="publickey" [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: send packet: type 51 [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: Connection closed by authenticating > user root 10.10.12.160 port 40854 [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: do_cleanup [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: PAM: sshpam_thread_cleanup > entering [preauth] > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: monitor_read_log: child log > fd closed > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: mm_request_receive entering > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: do_cleanup > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: PAM: cleanup > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug3: PAM: sshpam_thread_cleanup > entering > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: Killing privsep child 57169 > Oct 6 09:33:09 ceph-m1 sshd[57168]: debug1: audit_event: unhandled event > 12 > Oct 6 09:33:09 ceph-m1 sshd[757]: debug1: main_sigchld_handler: Child > exited > --------------- > > > I get "ECDSA key is not allowed" above. > From sshd logs, it looks like the client is not sending what is required > or in the expected format. > > Now, what was changed in quincy/mgr on ssh client? > Is anyone else using ECDSA keys and it works with quincy? > I could not find in PRs something specific to this that could block the > access, but it might be. > Any suggestion? > > Thank you! > Paul > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |