Hi Tim, On Wed, 20 Feb 2019, Tim Serong wrote: > Hi Sebastian, Juan, > > To follow up from the orchestrator call today, I've got DeepSea > configuring the deepsea orchestrator module during cluster deployment, > immediately after the mgr daemons themselves are started. The problem I > had was that mgr doesn't know about the available modules yet until a > few seconds after it starts up for the first time. > > DeepSea is (effectively) doing approximately this during deployment: > > # systemctl start ceph-mgr@$(hostname) > # ceph mgr module enable orchestrator_cli > # ceph mgr module enable deepsea > # ceph orchestrator set backend deepsea > # ceph deepsea config-set salt_api_url $URL > # ceph deepsea config-set salt_api_username $USERNAME > # ceph deepsea config-set salt_api_password $PASSWORD > > The `ceph mgr module enable` invocations would immediately fail with > "all mgr daemons do not support module [...], pass --force to force > enablement", and the subsequent module commands would fail because the > modules weren't enabled. > > We can try using --force, e.g.: > > # ceph mgr module enable orchestrator_cli --force > > But then the module still isn't loaded quickly enough at this point > (remember, it's only a fraction of a second after ceph-mgr starts for > the first time), so the subsequent "ceph orchestrator set backend > deepsea" and "ceph deepsea config-set" commands will still fail with "no > valid command". > > I've got two ways around this. One is to wait until mgr is reported as > being available: > > # systemctl start ceph-mgr@$(hostname) > # while [ "$(ceph mgr dump | jq '.available')" != "true" ] ; \ > do echo sleeping 1>&2 ; sleep 1 ; done > # ceph mgr module enable orchestrator_cli > # ceph mgr module enable deepsea > # ceph orchestrator set backend deepsea > # ceph deepsea config-set salt_api_url $URL > # ceph deepsea config-set salt_api_username $USERNAME > # ceph deepsea config-set salt_api_password $PASSWORD > > This works fine (it takes about 4 seconds in my dev/test environment for > mgr to become available), but of course really needs to be tweaked to > break out of that loop if mgr never becomes available in a reasonable time. > > Another option is to cheat a bit and force the modules to load, then > write config keys directly: > > # ceph mgr module enable orchestrator_cli --force: > # ceph mgr module enable deepsea --force: > # ceph config-key set \ > config/mgr/mgr/orchestrator_cli/orchestrator deepsea > # ceph config-key set config/mgr/mgr/deepsea/salt_api_url $URL > # ceph config-key set config/mgr/mgr/deepsea/salt_api_username $USERNAME > # ceph config-key set config/mgr/mgr/deepsea/salt_api_password $PASSWORD > > This works too, no irritating loop is necessary, but then DeepSea, an > external deployment tool, suddenly knows internal details of both the > orchestrator CLI module and deepsea module (i.e. the config keys), so > those pieces are no longer black boxes anymore, and if they change in > Ceph upstream, DeepSea's deployment breaks. > > What's preferable here? (I'm leaning towards the loop option) Am I > missing any other options? Jeff Layton ran into the same thing a few weeks back. We went with the loop for the time being. I think the other two options are: 1. Make 'ceph config set ...' block if the mgr is not available and it is a mgr option. This seems less than ideal (wouldn't expect that command to fail) and is probably also racy, since the mgr restarts itself after teh module enable command but the mon doesn't realize that right away. 2. Make a built-in wait loop (ceph mgr wait-until-available [timeout]) command that's coded into the CLI, so that every user doesn't have to do this. sage > > Regards, > > Tim > -- > Tim Serong > Senior Clustering Engineer > SUSE > tserong@xxxxxxxx > > >
