Hi all,
I'm preparing and testing an ansible playbook for octopus deployment and upgrade from mimic to octopus. During testing of octopus deployment I came across an annoying race condition in the first MGR that I couldn't figure out how to avoid. This race condition does not occur with mimic. Any hints are appreciated. Here the symptom:
State of daemon deployment:
- first MON started and active
- first MGR started and active
After the first MGR is active, we enable a few modules. This works fine for mimic, but fails every time on octopus with this error message:
TASK [ceph-mgrs : Start container ceph-mgr.] **********************************************************************************************************
changed: [tceph-01]
TASK [ceph-mgrs : Wait for MGR to become active.] *****************************************************************************************************
ok: [tceph-01]
TASK [ceph-mgrs : Enable modules.] ********************************************************************************************************************
failed: [tceph-01] (item=dashboard) => changed=true
ansible_loop_var: item
cmd: docker exec ceph-adm ceph mgr module enable "dashboard"
delta: '0:00:00.440099'
end: '2022-07-26 13:08:03.967133'
item: dashboard
msg: non-zero return code
rc: 2
start: '2022-07-26 13:08:03.527034'
stderr: 'Error ENOENT: all mgr daemons do not support module ''dashboard'', pass --force to force enablement'
stderr_lines: <omitted>
stdout: ''
stdout_lines: <omitted>
changed: [tceph-01] => (item=status)
changed: [tceph-01] => (item=iostat)
Waiting a few seconds and executing the same module enable commands works. There is clearly some initialisation unfinished despite the MGR being reported as active. We check that the MGR is active by inspecting the output of
ceph-adm ceph status -f json | jq -r ".mgrmap.active_name"
Why is the MGR state active when it cannot handle all commands yet? Is there any way to avoid this race condition, that is, is there some other command we can look at that guarantees that module commands will execute properly?
Thanks and best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
