On Thu, Nov 28, 2019 at 02:26:36PM +0000, Sage Weil wrote: --snip-- >> >I think it makes sense to focus on the out-of-the-box opinionated easy >> >scenario vs the DIY case, in general at least. But I have a few >> >questions... >> I think this focus will leave some users in the dust. Monitoring with prometheus >> can get complex, especially if it is to be fault tolerant (which imho is >> important for confidence in such a system). Also typically users don't want >> several monitoring systems in their environment. So let's keep the case of >> existing prometheus systems in mind please. > >That's what I want meant by 'vs' above... perhaps I should have said 'or'. >Either we deploy something simple and opinionated, or the user attaches to >their existing or self-configured setup. We don't probably need to worry >about the various points in the middle ground where we manage only part of >the metrics solution. I'm not sure we'll get off this easy. At the very least the prometheus mgr module is deployed by us. There is also an argument to be made for monitoring the things that we take control over, i.e. the containers we deploy (one node_exporter per container is a common setup) and maybe even the hosts that the orchestrator provisions. > >(Also, I'm trying to use 'metrics' to mean prometheus etc, vs 'monitoring' >which in my mind is nagios or pagerduty or whatever and presumably has a >level of HA required, and/or needs to be external instead of baked-in.) Not sure I understand that distinction. You mean metrics for the prometheus setup the orchestrator intents to install? (prometheus can certainly be a fully fledged monitoring stack). Jan > >sage > > >> > >> >- In the DIY case, does it makes sense to leave the node-exporter to the >> >reader too? Or might it make sense for us to help deploy the >> >node-exporter, but they run the external/existing prometheus instance? >> > >> >- Likewise, the alertmanager is going to have a bunch of ceph-specific >> >alerts configured, right? Might they want their own prom but we deploy >> >our alerts? (Is there any dependency in the dashboard on a particular set >> >of alerts in prometheus?) >> > >> >I'm guessing you think no in both these cases... >> >> What I'm missing from proposals I've seen so far is an interface to query the >> orchestrator for various prometheus bits. First and foremost the orchestrator >> should have a command that returns a prometheus file_sd_config of exporters that >> an external prometheus stack should scrape. Whether this is just the mgr >> exporter or also node_exporters (or others) depends on how far the orchestrator >> will take control. >> Alerts are currently handled as an rpm but could certainly be provided through a >> similar interface. >> >> At the very least, if the consensus will be that the orchestrator absolutely has >> to deploy everything itself, please at least provide an interface so that a >> federated setup is easily possible (an external prometheus scraping the >> orch-deployed prometheus) so that users don't have to care what the orchestrator >> does with monitoring (other then duplicating recorded metrics). See >> https://prometheus.io/docs/prometheus/latest/federation/#hierarchical-federation >> >> I'd really like to encourage the orchestrator team to carefully think this >> through. Monitoring is (at least for some users) a critical infrastructure >> component with its own inherent complexity. I'm worried that just doing this in >> a best-effort fashion and not offering an alternative path if going to weaken >> the ceph ecosystem. >> > >> >> > - Let's teach ceph-daemon how to do this, so that you do 'ceph-daemon >> >> > deploy --fsid ... --name prometheus.foo -i input.json'. ceph-daemon >> >> > has the framework for opening firewall ports etc now... just add ports >> >> > based on the daemon type. >> >> > >> >> >> >> TBH, I'd keep the monitoring containers away from the ceph daemons. They >> >> require different parameters, config files etc so why not keep them >> >> separate and keep the ceph logic clean. This also allows us to change >> >> monitoring without concerns over logic changes to normal ceph daemon >> >> management. >> > >> >Okay, but mgr/ssh is still going to be wired up to deploy these. And to do >> >so on a per-cluster, containerized basis... which means all of the infra >> >in ceph-daemon will still be useful. It seems easiest to just add it >> >there. >> > >> >Your points above seem to point toward simplifying the containers we >> >deploy to just two containers, one that's one-per-cluster for >> >prom+alertmanager+grafana, and one that's per-host for the node-exporter. >> >But I think making it fit in nicely with the other ceph containers (e.g., >> >/var/lib/ceph/$fsid/$thing) makes sense. Esp since we can just deploy >> >these during bootstrap by default (unless some --external-prometheus is >> >passed) and this all happens without the admin having to think about it. >> > >> >> > WDYT? >> >> > >> >> > >> >> I'm sure a lot of the above has already been discussed at length with the >> >> SuSE folks, so apologies for going over ground that you've already covered. >> > >> >Not yet! :) >> > >> >sage >> >_______________________________________________ >> >Dev mailing list -- dev@xxxxxxx >> >To unsubscribe send an email to dev-leave@xxxxxxx >> >> -- >> Jan Fajerski >> Senior Software Engineer Enterprise Storage >> SUSE Software Solutions Germany GmbH >> Maxfeldstr. 5, 90409 Nürnberg, Germany >> (HRB 36809, AG Nürnberg) >> Geschäftsführer: Felix Imendörffer >> _______________________________________________ >> Dev mailing list -- dev@xxxxxxx >> To unsubscribe send an email to dev-leave@xxxxxxx >> >> -- Jan Fajerski Senior Software Engineer Enterprise Storage SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
