On Wed, Jun 14, 2017 at 12:50 PM, Sage Weil <sweil@xxxxxxxxxx> wrote: > On Wed, 14 Jun 2017, Yehuda Sadeh-Weinraub wrote: >> On Sun, Jun 11, 2017 at 5:04 AM, John Spray <jspray@xxxxxxxxxx> wrote: >> > MgrClient instances (such as those in the daemons, and those in every >> > librados instance) open a session with ceph-mgr where they identify >> > themselves by entity name and type. ceph-mgr sends a MgrConfigure >> > message, which tells the client whether to both sending stats and how >> > often. ceph-mgr also keeps a copy of the metadata (a la "ceph osd >> > metadata...") for OSD and MDS daemons -- it loads all that up at >> > startup, and then also freshens it when it sees a daemon restart or >> > sees a new daemon. >> > >> > We would like to have something similar so that the mgr can be aware >> > of the existence of other services like RGW gateways, RBD mirror >> > services, perhaps also NFS gateways. >> > >> > The information about each daemon would at a minimum be its identity, >> > type, and some static metadata. It might also include some dynamic >> > state/health structure. The challenging part here is how to expose >> > that to the various daemons, given that things like RGW are not known >> > in advance to core Ceph and that they just consume the librados >> > interface. >> >> >> To start with, for radosgw process we could provide: > > It seems like these fall into two categories: > > 1/ static information about the daemon instance > >> - realm name (+ id) >> - zonegroup name (+ id) >> - zone name (+ id ) >> - instance id # (changes between runs) >> - listening ip:port >> - list of domains handled (maybe? can be quite large) >> - max number of concurrent requests > > This stuff would populate a 'running rgw daemons' type view, and could be > used by an agent that dynamically configures a load balancer (haproxy or > whatever). > > 2/ dynamic values that can be exposed as perfcounters > >> - current period >> - current # of requests (or max # of requests in the past X minutes) > - current bandwidth > - etc > > Is there anything that is updated at runtime (not startup) that > doesn't fit into a perfcounter? It not, then a single call to something > like rados_register_daemon(name, metadata_map) (and some mechanism > for cleaning out dead people) ought to suffice? > >> There is some other info that falls under different aspects of the rgw >> cluster, e.g., all the realm info / structure, the zonegroup config, >> zone config. This kind of data should be a category of its own, and >> more of a container to the specific radosgw process info. Does it make >> sense in the ceph-mgr context? > > This stuff any potential mgr module can pull out of the zone metadata pool > itself to display, right? rgw doesn't need to translate it and > report it? Right, that's another option. > >> > It doesn't feel like a particularly natural thing for librados, but >> > ultimately whatever we expose to rgw/rbd is de-facto librados, even if >> > we put it in a different library or whatever. >> > >> > So far I've got as far as thinking we should have an extra call just >> > in the C++ bindings that lets callers say "Hi, I'm a service not just >> > a client, and here's a map of metadata", that they call one time >> > between creating their RadosClient and connecting to the cluster. >> > >> >> >> It'd be great if the client itself could provide a flexible schema >> mapping of the info that it needs to expose. Or if there was some >> other generic way to do it. Was thinking something like the way you >> send a json to elasticsearch and it generates a doc out of it, but >> there's also a way to create a fixed mapping for the stored data. > > Hmm, instead of a map<string,string> it could just be a json blob that > the mgr stores that it is up to the user to parse and interpret. Or, if > there is info that doesn't fit into a simple dict, then one dict item > ("extra_stuff") can have a value consisting of encoded json. > Providing a json is probably easy and generic enough. Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html