Hi all, while digging down another issue I had with the managers I restarted them using "ceph orch restart mgr". After running that command the main manager is now in a restart loop. The only way for me to stop this is running "ceph orch pause", as soon as I do "ceph orch resume" the loop starts again. Has anyone a suggestion on how I can remove that stuck "restart" job? "ceph orch cancel" does not work, it returns "This Orchestrator does not support `orch cancel`" While googling for this issue it seems I'm not the first one having that issue: https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/IRC4ZPSSZYELWPU5D2FHKWJ2VU7IP3JG/ https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/message/WZUWZGVYPFII5HTUTW6OAQXIZ6VNT2E2/ See the logs below: user@ceph1:~# journalctl -fu ceph-6d588189-f434-4cb1-8c60-6e48cbf43a2a@mgr.ceph1.service --since "1 day ago" -g "Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a" Nov 22 15:31:38 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 15:31:51 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:31:51 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:34:49 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 15:35:00 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:35:00 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:38:39 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 15:38:50 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:38:50 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:43:35 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 15:43:46 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:43:46 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:50:19 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 15:50:30 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:50:30 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:58:26 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 15:58:37 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 15:58:37 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:10:16 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 16:10:28 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:10:28 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:17:17 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 16:17:29 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:17:29 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:40:52 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 16:41:07 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:41:07 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:45:34 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 16:45:47 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:45:47 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:49:34 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 16:49:39 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:49:39 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:51:47 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 16:51:51 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:51:51 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:54:02 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 16:54:08 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:54:08 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:57:37 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 16:57:41 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:57:41 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 16:59:58 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 17:00:02 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 17:00:02 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 17:02:42 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 17:02:45 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 17:02:45 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 17:04:40 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 17:04:43 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 17:04:43 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 17:07:28 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 17:07:35 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 17:07:35 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 17:10:50 ceph1 systemd[1]: Stopping Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a... Nov 22 17:10:54 ceph1 systemd[1]: Stopped Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Nov 22 17:10:54 ceph1 systemd[1]: Started Ceph mgr.ceph1 for 6d588189-f434-4cb1-8c60-6e48cbf43a2a. Thanks in advance ~ Roman _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |