On Tue, Sep 19, 2017 at 11:48:10AM -0400, Chuck Lever wrote: > > > On Sep 19, 2017, at 11:10 AM, Daniel P. Berrange <berrange@xxxxxxxxxx> wrote: > > > > On Tue, Sep 19, 2017 at 10:35:49AM -0400, Chuck Lever wrote: > >> > >>> On Sep 19, 2017, at 5:31 AM, Daniel P. Berrange <berrange@xxxxxxxxxx> wrote: > >>> > >>> On Mon, Sep 18, 2017 at 07:09:27PM +0100, Stefan Hajnoczi wrote: > >>>> There are 2 main use cases: > >>>> > >>>> 1. Easy file sharing between host & guest > >>>> > >>>> It's true that a disk image can be used but that's often inconvenient > >>>> when the data comes in individual files. Making throwaway ISO or > >>>> disk image from those files requires extra disk space, is slow, etc. > >>> > >>> More critically, it cannot be easily live-updated for a running guest. > >>> Not all of the setup data that the hypervisor wants to share with the > >>> guest is boot-time only - some may be access repeatedly post boot & > >>> have a need to update it dynamically. Currently OpenStack can only > >>> satisfy this if using its network based metadata REST service, but > >>> many cloud operators refuse to deploy this because they are not happy > >>> with the guest and host sharing a LAN, leaving only the virtual disk > >>> option which can not support dynamic update. > >> > >> Hi Daniel- > >> > >> OK, but why can't the REST service run on VSOCK, for instance? > > > > That is a possibility, though cloud-init/OpenStack maintainers are > > reluctant to add support for new features for the metadata REST > > service, because the spec being followed is defined by Amazon (as > > part of EC2), not by OpenStack. So adding new features would be > > effectively forking the spec by adding stuff Amazon doesn't (yet) > > support - this is why its IPv4 only, with no IPv6 support too, > > as Amazon has not defined a standardized IPv6 address for the > > metadata service at this time. > > You guys are asking the NFS community for a similar kind of > specification change here. We would prefer that you seek that change > with the relevant authority (the IETF) first before trying to merge > an implementation of it. > > As a first step we have to define RPC operation on VSOCK transports. > That's the smallest piece of it. Dealing with some of the NFS issues > (like, what happens to filehandles and lock state during a live > guest migration) is an even larger challenge. > > Sorry, but you can't avoid one interoperability problem (Amazon) > by introducing another (NFS). Agreed, I can't argue with that. It does feel overdue to get NFS-over-VSOCK defined as a formal spec, especially since it was already implemented in the NFS-ganesha userspace server. > >> How is VSOCK different than guests and hypervisor sharing a LAN? > > > > VSOCK requires no guest configuration, it won't be broken accidentally > > by NetworkManager (or equivalent), it won't be mistakenly blocked by > > guest admin/OS adding "deny all" default firewall policy. Similar > > applies on the host side, and since there's separation from IP networking, > > there is no possibility of the guest ever getting a channel out to the > > LAN, even if the host is mis-configurated. > > We don't seem to have configuration fragility problems with other > deployments that scale horizontally. > > IMO you should focus on making IP reliable rather than trying to > move familiar IP-based services to other network fabrics. I don't see that ever happening, except in a scenario where a single org is in tight control of the whole stack (host & guest), which is not the case for cloud in general - only some on-site clouds. > Or, look at existing transports: > > - We have an RPC-over-named-pipes transport that could be used Could that transport be used with a serial port rather than a named pipe I wonder ? If so, it could potentially be made to work with the virtio-serial device model. Is this named pipe transport already working with the NFS server, or is this just describing a possible strategy yet to be implemented ? > - NFS/RDMA with a virtual RoCE adapter would also work, and > could perform better than something based on TCP streams I don't know enough about RDMA to answer this, but AFAIK there is no RDMA device emulation for KVM yet, so that would be another device model to be created & supported. > >> Would it be OK if the hypervisor and each guest shared a virtual > >> point-to-point IP network? > > > > No - per above / below text > > > >> Can you elaborate on "they are not happy with the guests and host > >> sharing a LAN" ? > > > > The security of the host management LAN is so critical to the cloud, > > that they're not willing to allow any guest network interface to have > > an IP visible to/from the host, even if it were locked down with > > firewall rules. It is just one administrative mis-configuration > > away from disaster. > > For the general case of sharing NFS files, I'm not suggesting that > the guest and hypervisor share a management LAN for NFS. Rather, > the suggestion is to have a separate point-to-point storage area > network that has narrow trust relationships > > >>> If the admin takes any live snapshots of the guest, then this throwaway > >>> disk image has to be kept around for the lifetime of the snapshot too. > >>> We cannot just throw it away & re-generate it later when restoring the > >>> snapshot, because we canot guarantee the newly generated image would be > >>> byte-for-byte identical to the original one we generated due to possible > >>> changes in mkfs related tools. > >> > >> Seems like you could create a loopback mount of a small file to > >> store configuration data. That would consume very little local > >> storage. I've done this already in the fedfs-utils-server package, > >> which creates small loopback mounted filesystems to contain FedFS > >> domain root directories, for example. > >> > >> Sharing the disk serially is a little awkward, but not difficult. > >> You could use an automounter in the guest to grab that filesystem > >> when needed, then release it after a period of not being used. > > > > > > With QEMU's previous 9p-over-virtio filesystem support people have > > built tools which run virtual machines where the root FS is directly > > running against a 9p share from the host filesystem. It isn't possible > > to share the host filesystem's /dev/sda (or whatever) to the guest > > because its a holding a non-cluster filesystem so can't be mounted > > twice. Likewise you don't want to copy the host filesystems entire > > contents into a block device and mount that, as its simply impratical > > > > With 9p-over-virtio, or NFS-over-VSOCK, we can execute commands > > present in the host's filesystem, sandboxed inside a QEMU guest > > by simply sharing the host's '/' FS to the guest and have the > > guest mount that as its own / (typically it would be read-only, > > and then a further FS share would be added for writeable areas). > > For this to be reliable we can't use host IP networking because > > there's too many ways for that to fail, and if spawning the sandbox > > as non-root we can't influence the host networking setup at all. > > Currently it uses 9p-over-virtio for this reason, which works > > great, except that distros hate the idea of supporting a 9p > > filesystem driver in the kernel - a NFS driver capable of > > running over virtio is a much smaller incremental support > > burden. > > OK, so this is not the use case that has been described before. > This is talking about more than just configuration information. > > Let's call this use case "NFSROOT without IP networking". > > Still, the practice I was aware of is that OpenStack would > provide a copy of a golden root filesystem to the hypervisor, > who would then hydrate that on its local storage, and provide > it to a new guest as a virtual disk. The original OpenStack guest storage model is disk based, and in a simple deployment, you would indeed copy the root filesystem disk image from the image repository, over to the local virt host and expose that to the guest. These days though, the more popular option avoids using virt host local storage, and instead has the root disk image in an RBD volume that QEMU connects to directly. There's a further OpenStack project though called Manilla whose goal is to expose filesystem shares to guests, rather than block storage. This would typically be used in addition to the block based storage. eg the guest would have block dev for its root filesystem, and then a manilla filesystem share for application data storage. This is where the interest for NFS-over-VSOCK is coming from in the context of OpenStack. > >>>> From a user perspective it's much nicer to point to a directory and > >>>> have it shared with the guest. > >>>> > >>>> 2. Using NFS over AF_VSOCK as an interface for a distributed file system > >>>> like Ceph or Gluster. > >>>> > >>>> Hosting providers don't necessarily want to expose their distributed > >>>> file system directly to the guest. An NFS frontend presents an NFS > >>>> file system to the guest. The guest doesn't have access to the > >>>> distributed file system configuration details or network access. The > >>>> hosting provider can even switch backend file systems without > >>>> requiring guest configuration changes. > >> > >> Notably, NFS can already support hypervisor file sharing and > >> gateway-ing to Ceph and Gluster. We agree that those are useful. > >> However VSOCK is not a pre-requisite for either of those use > >> cases. > > > > This again requires that the NFS server which runs on the management LAN > > be visible to the guest network. So this hits the same problem above with > > cloud providers wanting those networks completely separate. > > No, it doesn't require using the management LAN at all. The Ceph server is on the management LAN, and guest which has to perform the NFS client mount is on the guest LAN. So some component must be visible on both LANs - either its the Ceph server or the NFS server, neither of which is desired. > > The desire from OpenStack is to have an NFS server on the compute host, > > which exposes the Ceph filesystem to the guest over VSOCK > > The use of VSOCK is entirely separate from the existence of an > NFS service on the hypervisor, since NFS can already gateway > Ceph. Let's just say that the requirement is access to an NFS > service without IP networking. FWIW, the Ceph devs have done a proof of concept already where they use an NFS server running in the host exporting the volume over VSOCK. They used NFS-ganesha userspace server which already merged patches to support the VSOCK protocol with NFS. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
