On Tue, Apr 5, 2016 at 12:56 PM, Owen Synge <osynge@xxxxxxxx> wrote:
> Dear all,
>
> This is in my opinion is clearly a bug, but I raise it in the mailing
> list as I expect all admins of ceph will strongly agree, that this makes
> ceph simpler, but developers may feel that since it requires changes to
> more than one repo its not worth doing.
>
> When ever you start the mon demon as a side effect the admin, osd, rgw
> and mds keys are created as a side effect if the mds keyring is not
> existing.
>
> In the systemV and systemd init scripts (at least) we have a side
> effect, that should be removed in my opinion, (or worse in my
> alternatively correctly documented.)
>
> This is a deployment layer violation, in my opinion, and it requires
> considerably more documentation, (and on my part also code) to keep this
> side effect than remove it.
>
> usecases for removing this are:
>
> (0) A ceph cluster should be able to be installed in any order. With the
> current behavior if the mds, rgw, or osd nodes are deployed first (along
> with the boot strap keyrings), the mon created must have all keys for
> the admin, mds-bootstrap, rgw-bootstrap, and osd-boostrap deployed in
> the correct path before the mon can safely be started, even if the
> cluster does not need the mds or rgw service's.
>
> (1) It is unfriendly to configuration being stored on the configuration
> server as the server needs to be updated with the values from the
> configured node keys, when people might want to store these keys centrally.
Is there anything stopping you from pre-generating all your keys
(including bootstrap etc) using ceph-authtool, and then including
those keys in the keyring file that you pass into mons when you create
them? That way, you could create your keys and store them in your
config management tool, all before starting any mons.
> (2) Assuming the admin, rgw-bootstrap, mds-bootstrap and osd-boostrap
> keys are always installed on all mon nodes is clearly increasing the
> distribution of keys where they might not be needed. Hence reducing
> security.
I think this is a bit of a subjective balance between security and
convenience. Currently we don't have to explain to folks how to get
their admin key into /etc/ceph, because it just pops out when the mon
starts. The security sacrifice is small because the keys are
remaining in the same domain (same host, same UID) as the mon's
/var/lib, which is already a master key.
For sites/systems that don't want to write out key files, it would be
straightforward to add a configuration option to ceph-create-keys so
that it would not write these out to disk.
Once you had that option, and were also pre-generating all your keys
with ceph-authtool, I think you'd have the behaviour you wanted
without anyone else having to update for a change to the mon's default
behaviour.
John
> (3) Using the current model adds an extra complication that these keys
> then need to be distributed to each node from the configured node, if
> generated by starting the mon, and not from the configuration server.
>
> (4) If you wish to use a more devops approach, and generate keys
> explicitly all the keys must be installed on all mon nodes before the
> mon nodes are started.
>
> (4.1) As a side effect we need to document why admins need the
> mds-bootstrap keyring when they dont want this service it is confusing,
> and requires an unnecessary process of migrating all keys to the
> explicitly desired keys.
>
> (5) I am developing a simple python library to configure ceph on each
> node independently of all others, (think of it as a parallelism version
> of ceph-deploy that can be called by any config management system) but
> with the current side effect behavior starting the mon needs to fail if
> the mds-bootstrap keyring is not created on the mon nodes before
> starting the mon, otherwise we get ordering complications.
>
> (5) The side effect is confusing, as no one expects this side effect,
> hence this leads to ceph seeming complex to a first time user.
>
> (6) I feel it is the responsibility of configuration management not the
> mon demon to request creating these keys.
>
> (7) I dont think this is clearly documented, hence this leads to ceph
> seeming complex to a first time user.
>
> (8) As more services like mds and rgw get added to ceph the problem gets
> multiplied.
>
> (9) Adding one more step to the by hand installation will clarify the
> authentication process. This extra step would simply be:
>
> /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>
> This is simpler and clearer than documenting the side effect.
>
> consequences:
>
> (1) Each configuration system which depends upon this behavior will need
> to be modified to call the single command on each mon:
>
> /usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>
> Here is a fix for ceph-deploy:
>
> https://github.com/SUSE/ceph-deploy/commit/58b030dbe0a964b32f1fbc9a3762e64dd74bf50c
>
> I assume other solutions will be easy to fix too.
>
> The systemd file in question, is
> "/usr/lib/systemd/system/ceph-create-keys@.service" and should be removed.
>
> This will simplify the salt configuration module documentation
> considerably, and if this is not done the salt module will need to add a
> requirement on the mds keyring being created before the mon can be created.
>
> the systemd file looks as follows:
>
> [Unit]
> Description=Ceph cluster key creator task
>
> # the last key created is the mds bootstrap key -- look for that.
> ConditionPathExists=!/var/lib/ceph/bootstrap-mds/ceph.keyring
>
> [Service]
> EnvironmentFile=-/etc/sysconfig/ceph
> Environment=CLUSTER=ceph
> ExecStart=/usr/sbin/ceph-create-keys --cluster ${CLUSTER} --id %i
>
> as you can see the side effect is blocked if the file
>
> /var/lib/ceph/bootstrap-mds/ceph.keyring
>
> already exists, which is just more to document.
>
> Hoping that you all agree
>
> Owen Synge
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
