Dear Ceph community, After upgrading our cluster to Quincy with cephadm (ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.1), I struggle to re-activate the snapshot schedule module: 0|0[root@osd-1 ~]# ceph mgr module enable snap_schedule 0|1[root@osd-1 ~]# ceph mgr module ls | grep snap snap_schedule on 0|0[root@osd-1 ~]# ceph fs snap-schedule list / --recursive Error ENOENT: Module 'snap_schedule' is not available I tried restarting the MGR daemons and failed over a restarted one. But with no change. 0|0[root@osd-1 ~]# ceph orch restart mgr Scheduled to restart mgr.osd-1 on host 'osd-1' Scheduled to restart mgr.osd-2 on host 'osd-2' Scheduled to restart mgr.osd-3 on host 'osd-3' Scheduled to restart mgr.osd-4.oylrhe on host 'osd-4' Scheduled to restart mgr.osd-5.jcfyqe on host 'osd-5' 0|0[root@osd-1 ~]# ceph orch ps --daemon_type mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.osd-1 osd-1 *:8443,9283 running (61s) 35s ago 9M 402M - 17.2.1 e5af760fa1c1 64f7ec70a6aa mgr.osd-2 osd-2 *:8443,9283 running (47s) 36s ago 9M 103M - 17.2.1 e5af760fa1c1 d25fdc793ff8 mgr.osd-3 osd-3 *:8443,9283 running (7h) 36s ago 9M 457M - 17.2.1 e5af760fa1c1 46d5091e50d6 mgr.osd-4.oylrhe osd-4 *:8443,9283 running (7h) 79s ago 9M 795M - 17.2.1 e5af760fa1c1 efb2a7cc06c5 mgr.osd-5.jcfyqe osd-5 *:8443,9283 running (8h) 37s ago 9M 448M - 17.2.1 e5af760fa1c1 96dd03817f32 0|0[root@osd-1 ~]# ceph mgr fail The MGR confirms, that the snap_schedule module is not available: 0|0[root@osd-1 ~]# journalctl -eu ceph-55633ec3-6c0c-4a02-990c-0f87e0f7a01f@xxxxxxx-1.service<mailto:ceph-55633ec3-6c0c-4a02-990c-0f87e0f7a01f@xxxxxxx-1.service> Jul 01 16:25:49 osd-1 bash[662895]: debug 2022-07-01T14:25:49.825+0000 7f0486408700 0 log_channel(audit) log [DBG] : from='client.90801080 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch Jul 01 16:25:49 osd-1 bash[662895]: debug 2022-07-01T14:25:49.825+0000 7f0486c09700 -1 mgr.server reply reply (2) No such file or directory Module 'snap_schedule' is not available But I'm not sure where the MGR is actually looking. The module path is: 0|22[root@osd-1 ~]# ceph config get mgr mgr_module_path /usr/share/ceph/mgr And while it is not available on the host (I assume these are just remnants from before our change to cephadm/docker, anyways): 0|0[root@osd-1 ~]# ll /usr/share/ceph/mgr ... drwxr-xr-x. 4 root root 144 22. Sep 2021 restful drwxr-xr-x. 3 root root 61 22. Sep 2021 selftest drwxr-xr-x. 3 root root 61 22. Sep 2021 status drwxr-xr-x. 3 root root 117 22. Sep 2021 telegraf ... The module is available in the MGR container (which I assume is where the MGR would look): 0|0[root@osd-1 ~]# docker exec -it ceph-55633ec3-6c0c-4a02-990c-0f87e0f7a01f-mgr-osd-1 /bin/bash [root@osd-1 /]# ls -l /usr/share/ceph/mgr ... drwxr-xr-x. 4 root root 65 Jun 23 19:48 snap_schedule ... The module was available before on Pacific which was also cephadm deployed. Has anybody an idea how I can further investigate this? Thanks again for all you help! Best Wishes, Mathias _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |