On Thu, Nov 8, 2018 at 10:04 AM Erwan Velu <evelu@xxxxxxxxxx> wrote: > > Thanks Noah & Zack for your answers. > > In fact what I need here is being able, for a given version, to > anticipate for a given version what will be the json structure of a > given command. > > Let's consider "ceph osd metadata -f". > > When I'm doing this > (https://github.com/ErwanAliasr1/skydive/blob/1fa8c596823bcc53dd7fcecec8c9a529514a2a88/topology/probes/ceph/osd.go#L91), > on a 12.2.5, I get > https://github.com/ErwanAliasr1/skydive/blob/1fa8c596823bcc53dd7fcecec8c9a529514a2a88/topology/probes/ceph/osd.go#L39 > > For OSDs, that's pretty trivial but when you consider running a ceph -s > -f json, I end up with > https://github.com/ErwanAliasr1/skydive/blob/1fa8c596823bcc53dd7fcecec8c9a529514a2a88/topology/probes/ceph/cluster.go#L37 > and that's not still complete as I didn't run every part of ceph > component on this cluster. OSD metadata is an interesting example, because the content is completely unstructured -- the OSDs themselves are passing up a map of strings to strings. Similarly, the servicemap is basically freeform. > For a 3rd party tool like mine, but surely also for the manager or some > other ones, we should be able to anticipate what will be the expected > scheme for a release. I would paraphrase this as "it would be good if Ceph had a management API with versioned structures", and I would agree, albeit with the caveat that I'm not sure such an API needs to be quite as comprehensive as the CLI. ceph-mgr doesn't have to worry about this, because it's part of Ceph. If something changes in a dump() method, the consuming code in ceph-mgr can be updated at the same time -- this is one of the key motivations for implementing functionality inside ceph-mgr modules instead of trying to build things externally. We rely on testing to notice any breakage like this, but Noah's work could also make detecting such changes cleaner. John > If I understood correctly your PR, I'm not sure it does cover the output > of all those commands a user/3rd party can call. > > So I wonder If I'm using the right interface or the right way to collect > information about a ceph cluster. If I want to make a structured > representation of various ceph releases (and containers will generate > this situation), I wonder how to handle that :/ > > Le 07/11/2018 à 21:33, Noah Watkins a écrit : > > Hey Erwan, > > > > This sounds similar to something I started recently, but haven't been > > able to finish completely. Although, it's actually probably pretty > > close to being able to merge. Let me know if it seems like it might > > help out and we can work out what's needed to handle your case.. > > > > https://github.com/ceph/ceph/pull/23716 > > > > - Noah > > On Wed, Nov 7, 2018 at 7:11 AM Erwan Velu <evelu@xxxxxxxxxx> wrote: > >> Hi list, > >> > >> I'm working on a tool that reads the json output of several ceph commands. > >> > >> To ease the parsing, I've been choosing the JSON format which guarantee > >> a parseable output. > >> > >> I'm using the Unmarshall feature of golang to map this output in an > >> internal data structure so every member of this JSON output is easily > >> reachable from the code. > >> > >> That works pretty well except that I have to "anticipate" what will the > >> be members and their types. > >> > >> To do that, I've been transforming the sample output of my Ceph cluster > >> (luminous) into a data struct with https://mholt.github.io/json-to-go/ > >> > >> That works fine unless that I would be a complete output to get every > >> possible combination of the json output. > >> > >> > >> So instead of reversing the json output, and that per version of Ceph as > >> versions can change the format, how can I extract the complete JSON > >> schema from each command ? > >> > >> > >> Erwan, > >>
