On Mon, Jan 20, 2014 at 11:27 AM, Daniel P. Berrange <berrange@xxxxxxxxxx> wrote:
On Fri, Dec 20, 2013 at 11:03:53PM -0500, Adam Walters wrote:I vaguely recall something being mentioned in the past, but not the
> This patchset adds a driver named 'config' that allows access to configuration
> data, such as secret and storage definitions. This is a pre-requisite for my
> next patchset which resolves the race condition on libvirtd startup and the
> circular dependencies between QEMU and the storage driver.
details. Can you explain the details of the circular dependancy
problem that you're currently facing ?
We have a handful of places in our code where we call out to public
> The basic rationale behind this idea is that there exist circumstances under
> which a driver may need to access things such as secrets during a time at
> which there is no active connection to a hypervisor. Without a connection,
> the data can't be accessed currently. I felt that this was a much simpler
> solution to the problem that building new APIs that do not require a connection
> to operate.
APIs to implement some piece of functionality. I've never been all
that happy about these scenarios. If we call the public API directly,
they cause havoc with error reporting because we end up dispatching
and resetting errors in the middle of an nested API call. Calling the
driver methods directly by doing conn->driver->invokeMethod() is
somewhat tedious & error prone because we're then skipping various
sanity tests that the public APIs do. With ACL checking now implemented
we also have the slightly odd situation that a public API check which
is documented as requiring permission 'xxxx' may also require permissions
'yyyy' and 'zzzz' to deal with other public APIs we invoke secretly.
I think there is a fairly strong argument that our internal implementations
could be decoupled from the public APIs, so we can call methods internally
without having to go via the public APIs at all.
On the flip side though, there's also a long term desire to separate
the different drivers into separate daemons, eg so the secrets driver
might move into a libvirt-secretsd daemon, which might explicitly
require use to go via the public APIs.
Mulling this over last night, I think there may be an alternative implementation
that would be sustainable long-term. You mentioned a desire to split the drivers
into separate daemons eventually... It might seem silly today, but what if I went
ahead and implemented a connection URI for each of the existing drivers? This
would result in a 'network:///', 'secrets:///', 'storage:///', etc. Once complete, the
existing code base could slowly be updated to utilize connections to those new
URIs in preparation for splitting the code out into daemons. This would end up
with the current libvirtd process eventually becoming a connection broker, with
a compatibility shim to allow access to the API as it is currently defined for
backwards compatibility.
In effect, today nothing would change, other than the storage driver would use
'secrets:///' to open a connection for its backends. Eventually, though, all drivers
would use the new URIs (the actual APIs implemented would not need to change),
allowing an easier split of the daemons. The compatibility shim I mentioned would
just be some code to open a connection to the proper URI, make the request,
close the connection, and return the data. The one thing that would be needed
at some point (which I'm not sure how to really implement today), would be a
restriction on which API functions could be called. Basically, if you opened a
connection to 'secrets:///', you should only be able to access secrets. If you tried
to access network information, an error would need to be thrown in order to prevent
all of the drivers having a connection to all of the other drivers, burning up sockets
on the host system. I'm guessing that this could be implemented using the ACL
framework that is already in place, but I haven't dug through the code to verify this yet.
In this long-term desire to split libvirt into multiple daemons, is there a plan to also
create a libvirt-domaind? Currently, the various hypervisors don't actually know how
to read the domain XML fully, so a daemon like that would probably be desired. As a
bonus, it could also allow a single connection to view all running domains, regardless
of which hypervisor driver actually created it, though that isn't strictly needed.
The reason I ask is that currently, it is nigh impossible to determine driver inter-dependencies
automatically. If strict controls were in place to prevent cross-URI calls (in effect making
libvirt as a single process act as though it were already split), it should become
easier to determine a driver's dependencies at compile (or possibly even run) time.
Having that data would allow for a true dependency-based driver load order. While
I think implementation of a dependency-aware driver loader is probably beyond my
personal programming skills, it would be the best method in the long run. Though,
in a multi-process model, it may be simpler to define a new connection function that
waits (optionally until a timeout expires) for the connection to succeed, performing
automatic retries in that time. This would allow the various daemons to start in any order,
yet automatically initialize in the proper dependency order. Any circular dependency
would, of course, cause a long hang during startup, but a timeout could detect that and
fail the startup of libvirt as a whole. The same timeout-based connection function could
also be used in the single-process model, of course, removing the need for driver load
order to be changed at all if all existing code used the proposed new URIs. In effect, the
driver load order change would be needed today, but once all existing code is updated,
the change could be reverted to the current free-for-all model, as there would be implied
control over that by virtue of the fact that the drivers would wait for their dependencies to
finish connecting.
One big concern I have about this approach is the fact that once we add
> This driver is technically what one may call a hypervisor driver, but it
> does not implement any domain operations. It simply exists to handle requests
> by drivers for access to informatino that would otherwise require a connection.
> The URI used for this driver is 'config:///' and has been tested working on 4
> different machines of mine, running three different distributions of Linux
> (Archlinux, Gentoo, and CentOS). Being a very simple driver, I would expect
> it to work pretty much anywhere.
>
> I would love to hear any comments and suggestions you may have about this
> driver. At the very least this plus my next patchset resolves the startup
> race condition on my machine. If a more robust setup (likely a new internal
> API) is in the works, this driver could act as a band-aid to allow access
> to this type of data in the interim if a better resolution is a ways off.
this 'config:///' driver, it is very hard for us to ever remove it again,
since this concept leaks out into the public API. So we're going to want
to be very sure that this is something we wish to support long term, and
I don't really have such confidence myself.
I'd like to understand a bit more about the dependancy issue you're facing
to see if there's something else we can do to address it.
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list