for setting the user, `ceph cephadm set-user` command should do it. Bit surprised by the second part of that though. With passwordless sudo access I would have expected that to start working. On Thu, May 4, 2023 at 11:27 AM Reza Bakhshayeshi <reza.b2008@xxxxxxxxx> wrote: > Thank you. > I don't see any more errors rather than: > > 2023-05-04T15:07:38.003+0000 7ff96cbe0700 0 log_channel(cephadm) log > [DBG] : Running command: sudo which python3 > 2023-05-04T15:07:38.025+0000 7ff96cbe0700 0 log_channel(cephadm) log > [DBG] : Connection to host1 failed. Process exited with non-zero exit > status 3 > 2023-05-04T15:07:38.025+0000 7ff96cbe0700 0 log_channel(cephadm) log > [DBG] : _reset_con close host1 > > What is the best way to safely change the cephadm user to root for the > existing cluster? It seems "ceph cephadm set-ssh-config" is not effective > (BTW, my cephadmin user can run "sudo which python3" without prompting > password on other hosts now, but nothing has been solved) > > Best regards, > Reza > > On Tue, 2 May 2023 at 19:00, Adam King <adking@xxxxxxxxxx> wrote: > >> The number of mgr daemons thing is expected. The way it works is it first >> upgrades all the standby mgrs (which will be all but one) and then fails >> over so the previously active mgr can be upgraded as well. After that >> failover is when it's first actually running the newer cephadm code, which >> is when you're hitting this issue. Are the logs still saying something >> similar about how "sudo which python3" is failing? I'm thinking this >> might just be a general issue with the user being used not having >> passwordless sudo access, that sort of accidentally working in pacific, but >> now not working any more in quincy. If the log lines confirm the same, we >> might have to work on something in order to handle this case (making the >> sudo optional somehow). As mentioned in the previous email, that setup >> wasn't intended to be supported even in pacific, although if it did work, >> we could bring something in to make it usable in quincy onward as well. >> >> On Tue, May 2, 2023 at 10:58 AM Reza Bakhshayeshi <reza.b2008@xxxxxxxxx> >> wrote: >> >>> Hi Adam, >>> >>> I'm still struggling with this issue. I also checked it one more time >>> with newer versions, upgrading the cluster from 16.2.11 to 16.2.12 was >>> successful but from 16.2.12 to 17.2.6 failed again with the same ssh errors >>> (I checked >>> https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors a >>> couple of times and all keys/access are fine). >>> >>> [root@host1 ~]# ceph health detail >>> HEALTH_ERR Upgrade: Failed to connect to host host2 at addr (x.x.x.x) >>> [ERR] UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host host2 at >>> addr (x.x.x.x) >>> SSH connection failed to host2 at addr (x.x.x.x): Host(s) were >>> marked offline: {'host2', 'host6', 'host9', 'host4', 'host3', 'host5', >>> 'host1', 'host7', 'host8'} >>> >>> The interesting thing is that always (total number of mgrs) - 1 is >>> upgraded, If I provision 5 MGRs then 4 of them, and for 3, 2 of them! >>> >>> As long as I'm in an internal environment, I also checked the process >>> with Quincy cephadm binary file. FYI I'm using stretch mode on this cluster. >>> >>> I don't understand why Quincy MGRs cannot ssh into Pacific nodes, if you >>> have any more hints I would be really glad to hear. >>> >>> Best regards, >>> Reza >>> >>> >>> >>> On Wed, 12 Apr 2023 at 17:18, Adam King <adking@xxxxxxxxxx> wrote: >>> >>>> Ah, okay. Someone else had opened an issue about the same thing after >>>> the 17.2.5 release I believe. It's changed in 17.2.6 at least to only use >>>> sudo for non-root users >>>> https://github.com/ceph/ceph/blob/v17.2.6/src/pybind/mgr/cephadm/ssh.py#L148-L153. >>>> But it looks like you're also using a non-root user anyway. We've required >>>> passwordless sudo access for custom ssh users for a long time I think (e.g. >>>> it's in pacific docs >>>> https://docs.ceph.com/en/pacific/cephadm/install/#further-information-about-cephadm-bootstrap, >>>> see the point on "--ssh-user"). Did this actually work for you before in >>>> pacific with a non-root user that doesn't have sudo privileges? I had >>>> assumed that had never worked. >>>> >>>> On Wed, Apr 12, 2023 at 10:38 AM Reza Bakhshayeshi < >>>> reza.b2008@xxxxxxxxx> wrote: >>>> >>>>> Thank you Adam for your response, >>>>> >>>>> I tried all your comments and the troubleshooting link you sent. From >>>>> the Quincy mgrs containers, they can ssh into all other Pacific nodes >>>>> successfully by running the exact command in the log output and vice versa. >>>>> >>>>> Here are some debug logs from the cephadm while updating: >>>>> >>>>> 2023-04-12T11:35:56.260958+0000 mgr.host8.jukgqm (mgr.4468627) 103 : >>>>> cephadm [DBG] Opening connection to cephadmin@x.x.x.x with ssh >>>>> options '-F /tmp/cephadm-conf-2bbfubub -i /tmp/cephadm-identity-7x2m8gvr' >>>>> 2023-04-12T11:35:56.525091+0000 mgr.host8.jukgqm (mgr.4468627) 144 : >>>>> cephadm [DBG] _run_cephadm : command = ls >>>>> 2023-04-12T11:35:56.525406+0000 mgr.host8.jukgqm (mgr.4468627) 145 : >>>>> cephadm [DBG] _run_cephadm : args = [] >>>>> 2023-04-12T11:35:56.525571+0000 mgr.host8.jukgqm (mgr.4468627) 146 : >>>>> cephadm [DBG] mon container image my-private-repo/quay-io/ceph/ceph@sha256 >>>>> :1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add >>>>> 2023-04-12T11:35:56.525619+0000 mgr.host8.jukgqm (mgr.4468627) 147 : >>>>> cephadm [DBG] args: --image my-private-repo/quay-io/ceph/ceph@sha256:1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add >>>>> ls >>>>> 2023-04-12T11:35:56.525738+0000 mgr.host8.jukgqm (mgr.4468627) 148 : >>>>> cephadm [DBG] Running command: sudo which python3 >>>>> 2023-04-12T11:35:56.534227+0000 mgr.host8.jukgqm (mgr.4468627) 149 : >>>>> cephadm [DBG] Connection to host1 failed. Process exited with non-zero exit >>>>> status 3 >>>>> 2023-04-12T11:35:56.534275+0000 mgr.host8.jukgqm (mgr.4468627) 150 : >>>>> cephadm [DBG] _reset_con close host1 >>>>> 2023-04-12T11:35:56.540135+0000 mgr.host8.jukgqm (mgr.4468627) 158 : >>>>> cephadm [DBG] Host "host1" marked as offline. Skipping gather facts refresh >>>>> 2023-04-12T11:35:56.540178+0000 mgr.host8.jukgqm (mgr.4468627) 159 : >>>>> cephadm [DBG] Host "host1" marked as offline. Skipping network refresh >>>>> 2023-04-12T11:35:56.540408+0000 mgr.host8.jukgqm (mgr.4468627) 160 : >>>>> cephadm [DBG] Host "host1" marked as offline. Skipping device refresh >>>>> 2023-04-12T11:35:56.540490+0000 mgr.host8.jukgqm (mgr.4468627) 161 : >>>>> cephadm [DBG] Host "host1" marked as offline. Skipping osdspec preview >>>>> refresh >>>>> 2023-04-12T11:35:56.540527+0000 mgr.host8.jukgqm (mgr.4468627) 162 : >>>>> cephadm [DBG] Host "host1" marked as offline. Skipping autotune >>>>> 2023-04-12T11:35:56.540978+0000 mgr.host8.jukgqm (mgr.4468627) 163 : >>>>> cephadm [DBG] Connection to host1 failed. Process exited with non-zero exit >>>>> status 3 >>>>> 2023-04-12T11:35:56.796966+0000 mgr.host8.jukgqm (mgr.4468627) 728 : >>>>> cephadm [ERR] Upgrade: Paused due to UPGRADE_OFFLINE_HOST: Upgrade: Failed >>>>> to connect to host host1 at addr (x.x.x.x) >>>>> >>>>> As I can see here, it turns out sudo is added to the code to be able >>>>> to continue: >>>>> >>>>> >>>>> https://github.com/ceph/ceph/blob/v17.2.5/src/pybind/mgr/cephadm/ssh.py#L143 >>>>> >>>>> I cannot privilege the cephadmin user to run sudo commands for some >>>>> policy reasons, could this be the root cause of the issue? >>>>> >>>>> Best regards, >>>>> Reza >>>>> >>>>> On Thu, 6 Apr 2023 at 14:59, Adam King <adking@xxxxxxxxxx> wrote: >>>>> >>>>>> Does "ceph health detail" give any insight into what the unexpected >>>>>> exception was? If not, I'm pretty confident some traceback would end up >>>>>> being logged. Could maybe still grab it with "ceph log last 200 info >>>>>> cephadm" if not a lot else has happened. Also, probably need to find out if >>>>>> the check-host is failing due to the check on the host actually failing or >>>>>> failing to connect to the host. Could try putting a copy of the cephadm >>>>>> binary on one and running "cephadm check-host --expect-hostname <hostname>" >>>>>> where the hostname is the name cephadm knows the host by. If that's not an >>>>>> issue I'd expect it's a connection thing. Could maybe try going through >>>>>> https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors >>>>>> <https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors>. >>>>>> Cephadm changed the backend ssh library from pacific to quincy due to the >>>>>> one used in pacific no longer being supported so it's possible some general >>>>>> ssh error has popped up in your env as a result. >>>>>> >>>>>> On Thu, Apr 6, 2023 at 8:38 AM Reza Bakhshayeshi < >>>>>> reza.b2008@xxxxxxxxx> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I have a problem regarding upgrading Ceph cluster from Pacific to >>>>>>> Quincy >>>>>>> version with cephadm. I have successfully upgraded the cluster to the >>>>>>> latest Pacific (16.2.11). But when I run the following command to >>>>>>> upgrade >>>>>>> the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process >>>>>>> stops >>>>>>> with "Unexpected error". (everything is on a private network) >>>>>>> >>>>>>> ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 >>>>>>> >>>>>>> I also tried the 17.2.4 version. >>>>>>> >>>>>>> cephadm fails to check the hosts' status and marks them as offline: >>>>>>> >>>>>>> cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd >>>>>>> (mgr.4516356) 5782 >>>>>>> : cephadm [DBG] host host4 (x.x.x.x) failed check >>>>>>> cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd >>>>>>> (mgr.4516356) 5783 >>>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping daemon >>>>>>> refresh >>>>>>> cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd >>>>>>> (mgr.4516356) 5784 >>>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts >>>>>>> refresh >>>>>>> cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd >>>>>>> (mgr.4516356) 5785 >>>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping network >>>>>>> refresh >>>>>>> cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd >>>>>>> (mgr.4516356) 5786 >>>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping device >>>>>>> refresh >>>>>>> cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd >>>>>>> (mgr.4516356) 5787 >>>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec >>>>>>> preview >>>>>>> refresh >>>>>>> cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd >>>>>>> (mgr.4516356) 5788 >>>>>>> : cephadm [DBG] Host "host4" marked as offline. Skipping autotune >>>>>>> cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : >>>>>>> cluster >>>>>>> [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: >>>>>>> failed >>>>>>> due to an unexpected exception >>>>>>> cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : >>>>>>> cluster >>>>>>> [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check >>>>>>> cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : >>>>>>> cluster >>>>>>> [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote >>>>>>> host >>>>>>> host7. Process exited with non-zero exit status 3 >>>>>>> cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : >>>>>>> cluster >>>>>>> [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote >>>>>>> host >>>>>>> host2. Process exited with non-zero exit status 3 >>>>>>> cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : >>>>>>> cluster >>>>>>> [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote >>>>>>> host >>>>>>> host8. Process exited with non-zero exit status 3 >>>>>>> cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : >>>>>>> cluster >>>>>>> [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote >>>>>>> host >>>>>>> host4. Process exited with non-zero exit status 3 >>>>>>> cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : >>>>>>> cluster >>>>>>> [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote >>>>>>> host >>>>>>> host3. Process exited with non-zero exit status 3 >>>>>>> >>>>>>> and here are some outputs of the commands: >>>>>>> >>>>>>> [root@host8 ~]# ceph -s >>>>>>> cluster: >>>>>>> id: xxx >>>>>>> health: HEALTH_ERR >>>>>>> 9 hosts fail cephadm check >>>>>>> Upgrade: failed due to an unexpected exception >>>>>>> >>>>>>> services: >>>>>>> mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) >>>>>>> mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, >>>>>>> host1.warjsr, host2.qyavjj >>>>>>> mds: 1/1 daemons up, 3 standby >>>>>>> osd: 37 osds: 37 up (since 8h), 37 in (since 3w) >>>>>>> >>>>>>> data: >>>>>>> >>>>>>> >>>>>>> io: >>>>>>> client: >>>>>>> >>>>>>> progress: >>>>>>> Upgrade to 17.2.5 (0s) >>>>>>> [............................] >>>>>>> >>>>>>> [root@host8 ~]# ceph orch upgrade status >>>>>>> { >>>>>>> "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 >>>>>>> :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", >>>>>>> "in_progress": true, >>>>>>> "which": "Upgrading all daemon types on all hosts", >>>>>>> "services_complete": [], >>>>>>> "progress": "3/59 daemons upgraded", >>>>>>> "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an >>>>>>> unexpected exception", >>>>>>> "is_paused": true >>>>>>> } >>>>>>> [root@host8 ~]# ceph cephadm check-host host7 >>>>>>> check-host failed: >>>>>>> Host 'host7' not found. Use 'ceph orch host ls' to see all managed >>>>>>> hosts. >>>>>>> [root@host8 ~]# ceph versions >>>>>>> { >>>>>>> "mon": { >>>>>>> "ceph version 16.2.11 >>>>>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>>>> pacific (stable)": 5 >>>>>>> }, >>>>>>> "mgr": { >>>>>>> "ceph version 16.2.11 >>>>>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>>>> pacific (stable)": 1, >>>>>>> "ceph version 17.2.5 >>>>>>> (98318ae89f1a893a6ded3a640405cdbb33e08757) >>>>>>> quincy (stable)": 3 >>>>>>> }, >>>>>>> "osd": { >>>>>>> "ceph version 16.2.11 >>>>>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>>>> pacific (stable)": 37 >>>>>>> }, >>>>>>> "mds": { >>>>>>> "ceph version 16.2.11 >>>>>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>>>> pacific (stable)": 4 >>>>>>> }, >>>>>>> "overall": { >>>>>>> "ceph version 16.2.11 >>>>>>> (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) >>>>>>> pacific (stable)": 47, >>>>>>> "ceph version 17.2.5 >>>>>>> (98318ae89f1a893a6ded3a640405cdbb33e08757) >>>>>>> quincy (stable)": 3 >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> The strange thing is I can rollback the cluster status by failing to >>>>>>> not-upgraded mgr like this: >>>>>>> >>>>>>> ceph mgr fail >>>>>>> ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 >>>>>>> >>>>>>> Would you happen to have any idea about this? >>>>>>> >>>>>>> Best regards, >>>>>>> Reza >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>>>>>> >>>>>>> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |