On Wed, Jan 24, 2007 at 11:48:47PM +0000, Daniel P. Berrange wrote: > On Wed, Jan 24, 2007 at 02:17:31PM +0000, Richard W.M. Jones wrote: > > * Another proposal was to make all libvirt calls remote > > (http://people.redhat.com/berrange/libvirt/libvirt-arch-remote-3.png) > > but I don't think this is a going concern because (1) it requires > > a daemon always be run, which is another installation problem and > > another chance for sysadmins to give up, and (2) the perception will > > be that this is slow, whether or not that is actually true. > > I'd never compared performance of direct hypercalls vs libvirt_proxy > before, so I did a little test. The most commonly called method is > virt-manager is virDomainGetInfo for fetching current status of a > running domain - we call that once a second per guest. > > So I wrote a simple program in C which calls virDomainGetInfo 100,000 > times for 3 active guest VMs. I ran the test under a couple of different > libvirt backends. The results were: > > 1. As root, direct hypercalls -> 1.4 seconds > 2. As non-root, hypercalls via libvirt_proxy -> 9 seconds > 3. As non-root, via XenD -> 45 minutes [1] > > So although it is x10 slower than hypercalls, the libvirt_proxy is > actaully pretty damn fast - 9 seconds for 300,000 calls. Interesting figures, I has expected the proxy inter-process communication to slow things down more, I guess it works well because scheduling follows exactly the message passing so there is little latency in the RPC, was that on a uniprocessor machine ? > There are many reasons the XenD path is slow. Each operation makes > a new HTTP request. It spawns a new thread per request. It talks to > XenStore for every request which has very high I/O overhead. It uses > the old SEXPR protocol which requests far more info than we actually > need. It is written in Python. Now I'm sure we can improve performance > somewhat by switching to the new XML-RPC api, and getting persistent > connections running, but I doubt it'll ever be as fast as libvirt_proxy > let alone hypercalls. So as mentioned above, I'd like to take XenD > out of the loop for remote management just like we do for the local > case with libvirt_proxy, but with full authenticated read+write access. I love XML, but I doubt switching to XML-RPC will speed things up. Well maybe if the parser is written in C, but overall parsing an XML instance has a cost and I doubt you will get anyway close to the 300,000/s of a proxy like RPC, that would mean 600,000 XML instances parsing per second of just overhead and that's really not realistic. My only concern with an ad-hoc protocol like the proxy one is that it would make harder to build client side say in Java (since we don't have bindings and that would be a relatively nice way to do it as mixing C and Java always raises some resistance/problems). Though I really don't think this is a blocker. > > * If Xen upstream in the meantime come up with a secure remote access > > method then potentially this means clients could have to choose > > between the two, or run two services (libvirtd + Xen/remote). > > > > * There are issues with versioning the remote API. Do we allow > > different versions of libvirt/libvirtd to talk to each other? > > Do we provide backwards compatibility when we move to a new API? > > We can simply apply the same rules as we do for public API. No changes > to existing calls, only additions are allowed. A simple protocol > version number can allow the client & server to negotiate the mutually > supported feature set. Agreed, actually we already do that with the protocol to the proxy: .... struct _virProxyPacket { unsigned short version; /* version of the proxy protocol */ unsigned short command; /* command number a virProxyCommand */ .... If we stick to adding only that should work in general c.f. HTTP 1.0 and HTTP 1.1 long term cohabitation. > > * Do we allow more than one client to talk to a libvirtd daemon > > (no | multiple readers one writer | multiple readers & writers). > > The latter - since we have things modifiying domains via XenD, or the > HV indirectly updating domain state, we defacto have multiple writers > already. From my work in the qemu daemon, I didn't encounter any major > problems with allowing multiple writers - by using a poll() based > single-thread event loop approach, I avoided any nasty multi-thread > problems associated with multiple connections. Provided each request > can be completed in a short amount of time, there should be no need to > go fully threaded. And if needed we can revamp the design internally later without breaking any compatibility since we operate in stricly synchronous RPC mode at the connection level (true from the C API down to the wire). > > * What's the right level to make a remote API? Should we batch > > calls up together? > > We may be already constrained by the client side API - all calls in > the libvirt public API are synchronous so from a single client thread > there is nothing available to batch. Yup let's avoid the asynch support, that killed so many protocol that I'm really cautious about this. If that mean we need to add higher level APIs, why not but this really should come as a user induced evolution, i.e. field reports :-) > > I've been investigating RPC mechanisms and there seem to be two > > reasonable possibilities, SunRPC and XMLRPC. (Both would need to > > run over some sort of secure connection, so there is a layer below > > both). My analysis of those is here: > > > > http://et.redhat.com/~rjones/secure_rpc/ > > SunRPC would handle our current APIs fine. We've talked every now & then > about providing asynchronous callbacks into the API - eg, so the client > can be notified of VM state changes without having to poll the > virDomainGetInfo api every second. The RPC wire protocol certainly > supports that, but its not clear the C apis do. Callbacks are hairy, somehow I would prefer to allow piggybacking extra payload on an RPC return than initiating one from the server, this simplifies things both on the client and server code, but also integration on the client event loop (please no threads !) > The XDR wire formating rules are very nicely defined - another option is > to use XDR as the wire encoding for our existing prototype impl in the > qemud. > > For XML-RPC I'd like to do a proof of concept of the virDomainGetInfo > method impl to see how much overhead it adds. Hopefully it would be > acceptable, although I'm sure its a fair bit more than XDR / SunRPC. > We would need persistent connections for XML-RPC to be viable, particularly > with TLS enabled. Since XML-RPC doesn't really impose any firm C API > I imagine we could get a-synchronous notifications from the server > working without much trouble. I'm a bit afraid of the XML-RPC overhead, parsing 10,000 small instances per second with libxml2 is doable with a bit of tuning but that's still more than an order of magnitude slower than the ad-hoc protocol and without the marshalling/demarshalling from the strings values. For SunRPC, well if you do some testing that will be fantastic. > [1] It didn't actually finish after 45 seconds. I just got bored of waiting. s/seconds/minutes/ I guess, and you checked CPU was at 100% not a bad deadlock somewhere, right ;-) ? Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@xxxxxxxxxx | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
