Hi Roman, what ceph version are you on? Also, when you ran the restart command originally, did you get a message about scheduling the restarts or no output? On Tue, Nov 23, 2021 at 6:04 AM Roman Steinhart <roman@xxxxxxxxxxx> wrote: > Hi all, > > while digging down another issue I had with the managers I restarted them > using "ceph orch restart mgr". > After running that command the main manager is now in a restart loop. > The only way for me to stop this is running "ceph orch pause", as soon as I > do "ceph orch resume" the loop starts again. > > Has anyone a suggestion on how I can remove that stuck "restart" job? > "ceph orch cancel" does not work, it returns "This Orchestrator does not > support `orch cancel`" > > While googling for this issue it seems I'm not the first one having that > issue: > > https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/IRC4ZPSSZYELWPU5D2FHKWJ2VU7IP3JG/ > > https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/WZUWZGVYPFII5HTUTW6OAQXIZ6VNT2E2/ > > See the logs below: > user@ceph1:~# journalctl -fu > ceph-6d588189-f434-4cb1-8c60-6e48cbf43a2a@mgr.ceph1.service --since "1 day > ago" -g "Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a" > Nov 22 15:31:38 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:31:51 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:31:51 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:34:49 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:35:00 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:35:00 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:38:39 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:38:50 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:38:50 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:43:35 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:43:46 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:43:46 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:50:19 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:50:30 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:50:30 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:58:26 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 15:58:37 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 15:58:37 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:10:16 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:10:28 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:10:28 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:17:17 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:17:29 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:17:29 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:40:52 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:41:07 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:41:07 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:45:34 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:45:47 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:45:47 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:49:34 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:49:39 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:49:39 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:51:47 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:51:51 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:51:51 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:54:02 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:54:08 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:54:08 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:57:37 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 16:57:41 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:57:41 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 16:59:58 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 17:00:02 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:00:02 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:02:42 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 17:02:45 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:02:45 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:04:40 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 17:04:43 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:04:43 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:07:28 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 17:07:35 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:07:35 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:10:50 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a... > Nov 22 17:10:54 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > Nov 22 17:10:54 ceph1 systemd[1]: Started Ceph mgr.ceph1 for > 6d588189-f434-4cb1-8c60-6e48cbf43a2a. > > Thanks in advance > ~ Roman > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |