On Tue, Sep 27, 2011 at 10:10:00AM +0100, Stefan Hajnoczi wrote: > Libguestfs provides very nice functionality for applications that need > to work with disk images. The includes provisioning applications that > set up or customize disk images. It also includes backup applications > that want to look inside disk image snapshots - both at the block and > file level. > > What's missing for libguestfs to fill this role is integration that > allows libvirt and libguestfs to work together with the same > network-transparent remote API approach. > > In the past we have discussed remoting libguestfs and Richard > presented possible approaches: > https://www.redhat.com/archives/libguestfs/2011-May/msg00093.html > > Could libvirt could provide a secure channel over which the libguestfs > client that an application is linked against talks to the remote > guestfsd? > > Would libguestfs need some new parameters to connect to a remote > libvirtd and create its appliance VM? > > In terms of application use cases, I'm thinking along the lines of > using libvirt to enumerate storage volumes and then switching to > libguestfs to actually access the storage volume remotely. Is there > enough information exposed by libvirt today to switch over to > libguestfs and point it at the storage volume/image file? IMHO, the overall goal for integration is that anytime you have a libvirt connection, you should be able to use libguestfs to access storage volumes or guests without further remote configuration required. The primary way to achieve this is if all communication takes place over the existing libvirt data transport, and not any out of band transport (NBD, NFS, SSH, whatever) The next obvious question is which side of the libvirt connection should the guestfs daemon/appliance be running. There are 3 main scenarios wrt guests 1. Guest is running and has a guestfs channel already present 2. Guest is running and does not have a guestfs channel present 3. Guest is not running. In case 1, obviously the daemon is running server side in the main guest OS. In case 2, you could either boot the guestfs appliance readonly, or hotplug a virtio channel for guestfs into the running guest to get readwrite access (assumes the guest OS has neccessary magic to auto-launch the daemon). The latter is obviously server side again, the former could be client or server side. In case 3, you need to boot the appliance. This could be client or server side. To run the appliance server side, will require that libvirtd gains the ability to spawn a guest with the appliance, and tunnel the host side of the virtio channel back over the libvirt connection to the client. On the client side, either libguestfs needs to have a set of pluggable I/O functions, which we can redirect to use the virStreamPtr APIs, or libvirt would have to turn its stream back into a UNIX socket. The latter is doable without any more libguestfs changes, but introduces extra I/O copying, while the former is most effecient, but requires libguestfs changes. To run the appliance client side, will require that libvirtd gains the ability to tunnel disk access over the libvirt connection to the client, which then runs the appliance somehow. On the client side, either QEMU needs a block driver with a set of pluggable I/O functions, which we can redirect to use libvirt APIs, or libvirt would have to turn its stream back into a UNIX socket running NBD protocol, or a create a userspace block device (FUSE but for block devs). Again the latter is doable without any QEMU changes, but has extra I/O copying, whjile the former is most efficient. Finally, there are some other things worth considering where libguestfs currently has functionality gaps wrt libvirt. - Integration with the libvirt lock manager infrastructure to prevent concurrent R/W guest disk access - Integration with the libvirt secret manager infrastructure to allow access to encrypted QCow2 disks - Integration with sVirt to ensure the appliance runs in a strictly confined context - Hotplug to allow extra disks to be inserted/removed to/from a running libguestfs appliance Running the appliance server side, spawned by libvirt would allow all of those points to be satisfied. There is a final problem in that not all hypervisors feature libvirtd, eg VMWare ESX, Hyper V and VirtualBox. Some of them might expose guest disks via HTTP or some other protocols that QEMU might be able to access directly. Others though would require that you setup some kind of shared filesystem (eg mount NFS) to access them. Others might let you SSH in (with suitable credentials) which lets you use FUSE SSHFS. With such hypervisors it is more or less impossible to satisfy my initial requirement that 'any time you have a libvirt connection you can run libguestfs without further admin configuration'. Probably the best you can do here is to ensure that there is one API you use for access guest disks. One other point worth mentioning is that libguestfs.so does not want to directly link to libvirt.so, and vica-verca, to ensure we both avoid pulling major new dependancy chains for all users. Similarly, if at all possible, any existing libguestfs application would like to be able to 'just work' with any libvirt integration without further code changes. Now what do I think we should do. Personally I would really like to have libguestfs be integrating with the lock manager, secret manager and sVirt infrastructure in libvirt. The only real practical way for this to happen is if the appliance runs server side, spawned via the normal QEMU driver guest startup code in libvirt. So I discount any idea of running the appliance client side & tunnelling block devices over libvirt. Also note that different solutions may be required for hypervisors without libvirt. I am ignoring such hypervisors for now. If I were ignoring the requirement that libguestfs does not link to libvirt, then you could quite likely make all this happen with only a simple additional API in libvirt. We need an API to let a client open a connection to a <channel> device, using the virStreamPtr API. If the guests were not running, libguestfs would use virDomainCreate to spawn a transient, auto-detroy guest, with a custom kernel/initrd that runs the appliance, and an additional <channel> device, but with all other parts of the guest XML unchanged. This would ensure all the lock manager, sVirt and secret stuff 'just works'. If the guest is already running, libguestfs would just query the XML to find the <channel> device configuration. Then it could just use a new API like virDomainOpenChannel(virStreamPtr, const char *channelid) to get a stream to talk to the guestfs daemon with. If libguestfs were just wanting to use a random disk device, not yet associated with any guest, then it again would use virDOmainCreate to spawn a transient, auto-destroy guest, with an XML config it fully creates. This would all make it possible for any guestfs based application to work with libvirt without any app code changes. I assume it is still the case, however, that libguestfs does *not* want to link directly to libvirt and use it for guest creation. Thus my idea is to find an alternative solution that gets as close to that "ideal" setup as possible. To do this I would create what I call a bridging library, to be named 'libvirt-guestfs.so'. This would have a handful of API calls for handling the initial creation/startup of the appliance & access to the vmchannel device only, delegating everything else to normal libguestfs APIs. - int virDomainStartGuestFS(virDomainPtr dom, int flags) If the guest 'dom' is not already running, boot the guest pointing it at the libguestfs kernel/initrd + vmchannel device. - guestfs_h *virDomainOpenGuestFS(virDomainPtr dom) Server side, find the vmchannel device for the guest, open a stream for it. On the client side the stream would somehow be proxied to a UNIX domain socket. We then call the libguestfs APIs to neccessary to attach the external UNIX domain socket, create the guestfs_h handle and return it. With those two APIs (possibly 1 or 2 more), an application wanting to use an integrated libguestfs+libvirt, would use libvirt-guestfs.so to obtain their guestfs_h * handle, and then use all the other libguestfs.so APIs as normal. This avoids having to wrap every single libguestfs API in libvirt. For apps like virt-manager this would easily be workable, other existing libguestfs apps would however need to have some small changes in their initial connection setup code to optionally use libvirt-guestfs.so Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list