Hi Sebastian, Juan,
To follow up from the orchestrator call today, I've got DeepSea
configuring the deepsea orchestrator module during cluster deployment,
immediately after the mgr daemons themselves are started. The problem I
had was that mgr doesn't know about the available modules yet until a
few seconds after it starts up for the first time.
DeepSea is (effectively) doing approximately this during deployment:
# systemctl start ceph-mgr@$(hostname)
# ceph mgr module enable orchestrator_cli
# ceph mgr module enable deepsea
# ceph orchestrator set backend deepsea
# ceph deepsea config-set salt_api_url $URL
# ceph deepsea config-set salt_api_username $USERNAME
# ceph deepsea config-set salt_api_password $PASSWORD
The `ceph mgr module enable` invocations would immediately fail with
"all mgr daemons do not support module [...], pass --force to force
enablement", and the subsequent module commands would fail because the
modules weren't enabled.
We can try using --force, e.g.:
# ceph mgr module enable orchestrator_cli --force
But then the module still isn't loaded quickly enough at this point
(remember, it's only a fraction of a second after ceph-mgr starts for
the first time), so the subsequent "ceph orchestrator set backend
deepsea" and "ceph deepsea config-set" commands will still fail with "no
valid command".
I've got two ways around this. One is to wait until mgr is reported as
being available:
# systemctl start ceph-mgr@$(hostname)
# while [ "$(ceph mgr dump | jq '.available')" != "true" ] ; \
do echo sleeping 1>&2 ; sleep 1 ; done
# ceph mgr module enable orchestrator_cli
# ceph mgr module enable deepsea
# ceph orchestrator set backend deepsea
# ceph deepsea config-set salt_api_url $URL
# ceph deepsea config-set salt_api_username $USERNAME
# ceph deepsea config-set salt_api_password $PASSWORD
This works fine (it takes about 4 seconds in my dev/test environment for
mgr to become available), but of course really needs to be tweaked to
break out of that loop if mgr never becomes available in a reasonable time.
Another option is to cheat a bit and force the modules to load, then
write config keys directly:
# ceph mgr module enable orchestrator_cli --force:
# ceph mgr module enable deepsea --force:
# ceph config-key set \
config/mgr/mgr/orchestrator_cli/orchestrator deepsea
# ceph config-key set config/mgr/mgr/deepsea/salt_api_url $URL
# ceph config-key set config/mgr/mgr/deepsea/salt_api_username $USERNAME
# ceph config-key set config/mgr/mgr/deepsea/salt_api_password $PASSWORD
This works too, no irritating loop is necessary, but then DeepSea, an
external deployment tool, suddenly knows internal details of both the
orchestrator CLI module and deepsea module (i.e. the config keys), so
those pieces are no longer black boxes anymore, and if they change in
Ceph upstream, DeepSea's deployment breaks.
What's preferable here? (I'm leaning towards the loop option) Am I
missing any other options?
Regards,
Tim
--
Tim Serong
Senior Clustering Engineer
SUSE
tserong@xxxxxxxx
