Indeed it looks to be this issue. Luis Domingues Proton AG ------- Original Message ------- On Friday, April 22nd, 2022 at 13:47, Adam King <adking@xxxxxxxxxx> wrote: > Did the 16.2.7 cluster have a non-root ssh user set and a host with an > _admin label? If so, could you try removing the _admin label from the host > and retrying the upgrade? It sounds like > https://tracker.ceph.com/issues/54620. > > Thanks, > - Adam King > > On Fri, Apr 22, 2022 at 7:25 AM Luis Domingues luis.domingues@xxxxxxxxx > > wrote: > > > Hello, > > > > We are testing the upgrade path from ceph 16.2.7 to ceph 17.2.0 on a small > > testing cluster. > > > > Basically, we just bootstrap a ceph cluster with cephadm, make sure we > > have 3 mgrs, 3 mons and 6 osds. > > > > Everytime we try the upgrade using `ceph orch upgrade start ceph-version 17.2.0`, we get 2 mgrs to 17.2.0, but the upgrade stops there. We end up > > having: > > > > `ceph versions { "mon": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 1, "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 }, "osd": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 6 }, "mds": {}, "overall": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 10, "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 } }` > > > > The error we get from ceph log last cephadm is: > > > > ``` > > 2022-04-22T11:14:36.512558+0000 mgr.ip-10-12-0-68.dyvuxt (mgr.34315) 32 : > > cephadm [ERR] executing > > refresh((['ip-10-12-0-15.eu-central-1.compute.internal', > > 'ip-10-12-0-222.eu-central-1.compute.internal', > > 'ip-10-12-0-250.eu-central-1.compute.internal', > > 'ip-10-12-0-68.eu-central-1.compute.internal', > > 'ip-10-12-0-78.eu-central-1.compute.internal', > > 'ip-10-12-0-85.eu-central-1.compute.internal'],)) failed. > > Traceback (most recent call last): > > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 221, in _write_remote_file > > await asyncssh.scp(f.name, (conn, tmp_path)) > > File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp > > await source.run(srcpath) > > File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run > > self.handle_error(exc) > > File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in > > handle_error > > raise exc from None > > File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run > > await self._send_files(path, b'') > > File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in > > _send_files > > self.handle_error(exc) > > File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in > > handle_error > > raise exc from None > > File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in > > _send_files > > await self._send_file(srcpath, dstpath, attrs) > > File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in > > _send_file > > await self._make_cd_request(b'C', attrs, size, srcpath) > > File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in > > _make_cd_request > > self._fs.basename(path)) > > File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in > > make_request > > raise exc > > asyncssh.sftp.SFTPFailure: scp: /tmp/etc/ceph/ceph.conf.new: Permission > > denied > > > > During handling of the above exception, another exception occurred: > > > > Traceback (most recent call last): > > File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work > > return f(*arg) > > File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh > > self._write_client_files(client_files, host) > > File "/usr/share/ceph/mgr/cephadm/serve.py", line 1052, in > > _write_client_files > > self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) > > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 238, in write_remote_file > > host, path, content, mode, uid, gid, addr)) > > File "/usr/share/ceph/mgr/cephadm/module.py", line 569, in wait_async > > return self.event_loop.get_result(coro) > > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result > > return asyncio.run_coroutine_threadsafe(coro, self._loop).result() > > File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result > > return self.__get_result() > > File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in > > __get_result > > raise self._exception > > File "/usr/share/ceph/mgr/cephadm/ssh.py", line 226, in _write_remote_file > > raise OrchestratorError(msg)orchestrator._interface.OrchestratorError: > > Unable to write > > ip-10-12-0-15.eu-central-1.compute.internal:/etc/ceph/ceph.conf: scp: > > /tmp/etc/ceph/ceph.conf.new: Permission denied > > ``` > > > > But if we bootstrap the cluster using 15.2.16 instead of 16.2.7, the > > upgrade just goes perfectly, on the exact same setup. > > > > I do not know what could cause that. Does someone has an idea to help me > > try to find what is going wrong when upgrading a fresh 16.2.7 install? > > > > Thanks > > > > Luis Domingues > > Proton AG > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |