On Tue, Aug 19, 2014 at 10:47:24AM -0600, Eric Blake wrote:
> On 08/19/2014 08:43 AM, Daniel P. Berrange wrote:
> > On Tue, Aug 19, 2014 at 03:14:19PM +0200, Peter Krempa wrote:
> >> I'd like to propose a (hopefully) fairly future-proof API to retrieve
> >> various statistics for domains.
> >>
> >> The motivation is that management layers that use libvirt usually poll
> >> libvirt for statistics using various split up APIs we currently provide.
> >> To get all the necessary stuff, the mgmt app need to issue Ndomains *
> >> Napis calls and cope with the various returned formats. The APIs I'm
> >> wanting to introduce here will:
> >>
> >> 1) Return data in a format that we can expand in the future and is
> >> hierarchical. For starters I'll use XML, with possible expansion to
> >> something like JSON if it will be favourable for a consumer (switchable
> >> by a flag)
> >
> > I'm not particularly a fan of using XML for this. As a guiding principal
> > we've used structures when we needed APIs for which efficiency is
> > important, and XML for APIs where efficiency is irrelevant. Even with
> > the ability to bulk list data from many VMs at once, I'd think that
> > efficiency is still a very important property of the API. Consider if
> > we have 1000 virtual machines on a host - the XML document is going to
> > get very large and frankly most XML parsers are terribly slow. Would
> > not surprise me if it too several seconds or more to parse an XML doc
> > that this proposed API would return, which I don't think is really
> > viable. JSON might be slightly better in this respect, but not by
> > as much as we might imagine.
>
> I'm also not a fan of XML or JSON for the all-domains-in-one-API call in
> this respect, because for a monolithic string, you can't return it
> unless you have the data for ALL domains, and you may be hitting RPC
> limits far sooner than you wish (it doesn't scale well to machines that
> can run thousands of VMs). I'm also not a fan of JSON in a public API
> since none of our existing APIs return JSON (well, libvirt-qemu.so
> returns JSON for arbitrary monitor commands, but that's an unsupported
> API and is merely exposing what qemu does) - adding JSON to a public API
> doubles the parsing support that a client has to use. Using XML for a
> single domain might be doable, though, but that's one XML document per
> domain, not one XML document for the overall API.
>
> I still think we're going to have to come up with some sort of iterative
> callback approach, where the callback is visited once per domain, rather
> than trying to return all results in a single call.
>
> >
> > So I'd rather think we need to figure out a way to map this into some
> > kind of struct, where reading any single statistic could be done in
> > approx constant time, or at least time that is independant of the
> > number of VMs.
> >
> > Much as I dislike the virTypedParameter struct, it might actually be
> > a reasonable fit here. Perhaps a struct
> >
> > struct virDomainRecord {
> > virDomainPtr dom;
> > size_t nparams;
> > virTypedParameter params;
> > };
> >
> > and the API returns an array of virDomainRecord elements.
> > For the keys in the parameters we could use dot separated
> > components. eg
> >
> > state=runing
> > cpu.online=8
> > cpu.0.state=running
> > cpu.0.time=10001231231
> > cpu.1.state=running
> > cpu.1.time=10001231231
> > ...
> > cpu.7.state=running
> > cpu.7.time=10001231231
>
> I'm not sure whether this is a net gain on memory usage. It seems like
> this approach would require that libvirt malloc's each parameter, and
> that the user must free them all at the end. At least with an XML
> document, there's only a single malloc'd chunk of memory, rather than
> lots of pieces, on the API side. On the other hand, if a user is given
> an XML document, but ends up parsing it into an in-memory
> representation, they may end up malloc'ing even more memory for that
> in-memory representation, so our parameters might be a net win after
> all, where it is just a tradeoff of libvirt instead of the xml parser
> doing all the mallocing of small chunks.
NB I'm not really thinking about mlloc overhead per se, but rather the
CPU overhead inherent in parsing a document format that is as complex
as XML is. Even so I'm confident that both building & parsing the XML
would have far more mallocs involved than a typed parameter based
approach.
Regards,
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list