On 02/21/2019 01:37 AM, Sage Weil wrote: > Hi Tim, > > On Wed, 20 Feb 2019, Tim Serong wrote: >> Hi Sebastian, Juan, >> >> To follow up from the orchestrator call today, I've got DeepSea >> configuring the deepsea orchestrator module during cluster deployment, >> immediately after the mgr daemons themselves are started. The problem I >> had was that mgr doesn't know about the available modules yet until a >> few seconds after it starts up for the first time. >> >> DeepSea is (effectively) doing approximately this during deployment: >> >> # systemctl start ceph-mgr@$(hostname) >> # ceph mgr module enable orchestrator_cli >> # ceph mgr module enable deepsea >> # ceph orchestrator set backend deepsea >> # ceph deepsea config-set salt_api_url $URL >> # ceph deepsea config-set salt_api_username $USERNAME >> # ceph deepsea config-set salt_api_password $PASSWORD >> >> The `ceph mgr module enable` invocations would immediately fail with >> "all mgr daemons do not support module [...], pass --force to force >> enablement", and the subsequent module commands would fail because the >> modules weren't enabled. >> >> We can try using --force, e.g.: >> >> # ceph mgr module enable orchestrator_cli --force >> >> But then the module still isn't loaded quickly enough at this point >> (remember, it's only a fraction of a second after ceph-mgr starts for >> the first time), so the subsequent "ceph orchestrator set backend >> deepsea" and "ceph deepsea config-set" commands will still fail with "no >> valid command". >> >> I've got two ways around this. One is to wait until mgr is reported as >> being available: >> >> # systemctl start ceph-mgr@$(hostname) >> # while [ "$(ceph mgr dump | jq '.available')" != "true" ] ; \ >> do echo sleeping 1>&2 ; sleep 1 ; done >> # ceph mgr module enable orchestrator_cli >> # ceph mgr module enable deepsea >> # ceph orchestrator set backend deepsea >> # ceph deepsea config-set salt_api_url $URL >> # ceph deepsea config-set salt_api_username $USERNAME >> # ceph deepsea config-set salt_api_password $PASSWORD >> >> This works fine (it takes about 4 seconds in my dev/test environment for >> mgr to become available), but of course really needs to be tweaked to >> break out of that loop if mgr never becomes available in a reasonable time. >> >> Another option is to cheat a bit and force the modules to load, then >> write config keys directly: >> >> # ceph mgr module enable orchestrator_cli --force: >> # ceph mgr module enable deepsea --force: >> # ceph config-key set \ >> config/mgr/mgr/orchestrator_cli/orchestrator deepsea >> # ceph config-key set config/mgr/mgr/deepsea/salt_api_url $URL >> # ceph config-key set config/mgr/mgr/deepsea/salt_api_username $USERNAME >> # ceph config-key set config/mgr/mgr/deepsea/salt_api_password $PASSWORD >> >> This works too, no irritating loop is necessary, but then DeepSea, an >> external deployment tool, suddenly knows internal details of both the >> orchestrator CLI module and deepsea module (i.e. the config keys), so >> those pieces are no longer black boxes anymore, and if they change in >> Ceph upstream, DeepSea's deployment breaks. >> >> What's preferable here? (I'm leaning towards the loop option) Am I >> missing any other options? > > Jeff Layton ran into the same thing a few weeks back. We went with > the loop for the time being. Thanks Sage, I'll run with that too immediately. > I think the other two options are: > > 1. Make 'ceph config set ...' block if the mgr is not available and it is > a mgr option. This seems less than ideal (wouldn't expect that command to > fail) and is probably also racy, since the mgr restarts itself after teh > module enable command but the mon doesn't realize that right away. Yeah, that's possibly more trouble than it's worth. > 2. Make a built-in wait loop (ceph mgr wait-until-available [timeout]) > command that's coded into the CLI, so that every user doesn't have to do > this. That sounds good to me. Cheers, Tim -- Tim Serong Senior Clustering Engineer SUSE tserong@xxxxxxxx
