On 2017-11-15T13:32:55, Sage Weil <sage@xxxxxxxxxxxx> wrote:
I am strongly in favor of moving the config to the MONs, and
depreciating ceph.conf - maybe a ceph-bootstrap.conf for connecting to
the MONs to get it, but that's it.
In a previous life, I helped design a Cluster Information Base to reduce
config drift - a central information store is vastly superior to files
copied around, whether that happens manually or from a config management
system.
It's always outdated *somewhere*, and Ceph already has the concept of
the MONs having maps and a concurrency/consistency algorithm for them
(beloved PAXOS), so it doesn't add any significant complexity.
So for once, I vote for building it in. Don't add etcd/consul. We want
strong consistency here, and can build on stuff already there. If Ceph
would need to invent this from scratch, sure, but thus we can build on
something existing that needs to work anyway or we're screwed.
> > > 1- a default value (compiled in)
> > > 2- a value from the mon
> > > 3- a value from ceph.conf
> > > 4- a value set via command line, 'ceph tell', 'ceph daemon ... config set
> > > ...', etc.
I'm opposed to 3 and 4.
I *can* see the need to override a value on a per-host or on a
per-daemon instance basis (including combinations thereof, e.g., all
OSDs on node X). (Back when, we also expected these to be way more
frequently needed; to this day, I can count on my fingers the times I
needed per-host overrides, I think; really the only use case where this
happens more often are debug flags.)
But if you want any sort of consistency, those modify the settings in
the respective map on the MON, and the daemon *then* gets that one from
the single authoritative source of truth.
> (We also have to allow local overrides to ensure that the mon config can't
> brick the cluster by setting the internal mon settings, like paxos_*, to
> some bad value.)
Valid point; but perhaps this could be solved by allowing the MONs to
start up in a "safe" mode, too?
> I think the reporting to mgr to make a distinction needs to be there,
> though, because (1) to make a transition we want to see the delta between
> what the daemon has running and what the mon wants, and (2) I don't think
> we should make things like 'ceph daemon ... config set ...' turn into a
> request to the monitor to set a config so that the daemon will get a
> corresponding config update. These are low-level commands that are
> important for debugging/fixing issues and I we shouldn't break them.
I'm not perfectly sure about this one, see above. I think having a
single channel through which config updates reach daemons might be worth
it.
> Or the UI can not implement those buttons at all and just show that there
> is a disaparity and leave it to the user to fix (or not)...?
... or make the disparity go away through a single source of truth.
Regards,
Lars
--
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
