Hi Adam, thanks for the suggestion. I did what you proposed and it worked. On Tue, 23 Nov 2021 at 14:53, Adam King <adking@xxxxxxxxxx> wrote: > One thing you could maybe try is, if you have a host available that > doesn't have a mgr, moving the mgr service to that host. I.e. if you have > the mgr thrashing happening and have a mgr on host1 and host2 but host3 has > no mgr you could run" ceph orch apply mgr host3". Then once it settles down > move the mgr service back to its original spot. > > On Tue, Nov 23, 2021 at 8:19 AM Roman Steinhart <roman@xxxxxxxxxxx> wrote: > >> Hi Adam, >> >> I'm on 15.2.15 >> Yep, I remember seeing such a "scheduled" message for each of my managers >> >> >> >> On Tue, 23 Nov 2021 at 14:14, Adam King <adking@xxxxxxxxxx> wrote: >> >>> Hi Roman, what ceph version are you on? Also, when you ran the >>> restart command originally, did you get a message about scheduling the >>> restarts or no output? >>> >>> >>> >>> On Tue, Nov 23, 2021 at 6:04 AM Roman Steinhart <roman@xxxxxxxxxxx> >>> wrote: >>> >>>> Hi all, >>>> >>>> while digging down another issue I had with the managers I restarted >>>> them >>>> using "ceph orch restart mgr". >>>> After running that command the main manager is now in a restart loop. >>>> The only way for me to stop this is running "ceph orch pause", as soon >>>> as I >>>> do "ceph orch resume" the loop starts again. >>>> >>>> Has anyone a suggestion on how I can remove that stuck "restart" job? >>>> "ceph orch cancel" does not work, it returns "This Orchestrator does not >>>> support `orch cancel`" >>>> >>>> While googling for this issue it seems I'm not the first one having that >>>> issue: >>>> >>>> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/IRC4ZPSSZYELWPU5D2FHKWJ2VU7IP3JG/ >>>> >>>> https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/WZUWZGVYPFII5HTUTW6OAQXIZ6VNT2E2/ >>>> >>>> See the logs below: >>>> user@ceph1:~# journalctl -fu >>>> ceph-6d588189-f434-4cb1-8c60-6e48cbf43a2a@mgr.ceph1.service --since "1 >>>> day >>>> ago" -g "Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a" >>>> Nov 22 15:31:38 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 15:31:51 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:31:51 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:34:49 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 15:35:00 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:35:00 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:38:39 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 15:38:50 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:38:50 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:43:35 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 15:43:46 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:43:46 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:50:19 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 15:50:30 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:50:30 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:58:26 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 15:58:37 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 15:58:37 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:10:16 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 16:10:28 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:10:28 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:17:17 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 16:17:29 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:17:29 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:40:52 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 16:41:07 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:41:07 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:45:34 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 16:45:47 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:45:47 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:49:34 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 16:49:39 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:49:39 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:51:47 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 16:51:51 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:51:51 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:54:02 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 16:54:08 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:54:08 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:57:37 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 16:57:41 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:57:41 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 16:59:58 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 17:00:02 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 17:00:02 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 17:02:42 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 17:02:45 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 17:02:45 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 17:04:40 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 17:04:43 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 17:04:43 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 17:07:28 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 17:07:35 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 17:07:35 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 17:10:50 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a... >>>> Nov 22 17:10:54 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> Nov 22 17:10:54 ceph1 systemd[1]: Started Ceph mgr.ceph1 for >>>> 6d588189-f434-4cb1-8c60-6e48cbf43a2a. >>>> >>>> Thanks in advance >>>> ~ Roman >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>>> >>>> >> >> -- >> >> >> -- *Roman Steinhart* 📧 roman@xxxxxxxxxxx 🏢 Rheinweg 24, 53113 Bonn <https://www.google.de/maps/place/Aternos+GmbH/@50.7164952,7.1153599,17z/data=!3m1!4b1!4m5!3m4!1s0x47bee12aa186d85b:0x9969ffbf8cedfb6b!8m2!3d50.7164952!4d7.1175486> 🌍 https://aternos.org 📱 +49 228 18038356 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |