> On May 18, 2017, at 11:08 AM, J. Bruce Fields <bfields@xxxxxxxxxx> wrote: > > On Thu, May 18, 2017 at 03:04:50PM +0000, Trond Myklebust wrote: >> On Thu, 2017-05-18 at 10:28 -0400, Chuck Lever wrote: >>>> On May 18, 2017, at 9:34 AM, Stefan Hajnoczi <stefanha@xxxxxxxxxx> >>>> wrote: >>>> >>>> On Tue, May 16, 2017 at 09:11:42AM -0400, J. Bruce Fields wrote: >>>>> I think you explained this before, perhaps you could just offer a >>>>> pointer: remind us what your requirements or use cases are >>>>> especially >>>>> for VM migration? >>>> >>>> The NFS over AF_VSOCK configuration is: >>>> >>>> A guest running on host mounts an NFS export from the host. The >>>> NFS >>>> server may be kernel nfsd or an NFS frontend to a distributed >>>> storage >>>> system like Ceph. A little more about these cases below. >>>> >>>> Kernel nfsd is useful for sharing files. For example, the guest >>>> may >>>> read some files from the host when it launches and/or it may write >>>> out >>>> result files to the host when it shuts down. The user may also >>>> wish to >>>> share their home directory between the guest and the host. >>>> >>>> NFS frontends are a different use case. They hide distributed >>>> storage >>>> systems from guests in cloud environments. This way guests don't >>>> see >>>> the details of the Ceph, Gluster, etc nodes. Besides benefiting >>>> security it also allows NFS-capable guests to run without >>>> installing >>>> specific drivers for the distributed storage system. This use case >>>> is >>>> "filesystem as a service". >>>> >>>> The reason for using AF_VSOCK instead of TCP/IP is that traditional >>>> networking configuration is fragile. Automatically adding a >>>> dedicated >>>> NIC to the guest and choosing an IP subnet has a high chance of >>>> conflicts (subnet collisions, network interface naming, firewall >>>> rules, >>>> network management tools). AF_VSOCK is a zero-configuration >>>> communications channel so it avoids these problems. >>>> >>>> On to migration. For the most part, guests can be live migrated >>>> between >>>> hosts without significant downtime or manual steps. PCI >>>> passthrough is >>>> an example of a feature that makes it very hard to live migrate. I >>>> hope >>>> we can allow migration with NFS, although some limitations may be >>>> necessary to make it feasible. >>>> >>>> There are two NFS over AF_VSOCK migration scenarios: >>>> >>>> 1. The files live on host H1 and host H2 cannot access the files >>>> directly. There is no way for an NFS server on H2 to access >>>> those >>>> same files unless the directory is copied along with the guest or >>>> H2 >>>> proxies to the NFS server on H1. >>> >>> Having managed (and shared) storage on the physical host is >>> awkward. I know some cloud providers might do this today by >>> copying guest disk images down to the host's local disk, but >>> generally it's not a flexible primary deployment choice. >>> >>> There's no good way to expand or replicate this pool of >>> storage. A backup scheme would need to access all physical >>> hosts. And the files are visible only on specific hosts. >>> >>> IMO you want to treat local storage on each physical host as >>> a cache tier rather than as a back-end tier. >>> >>> >>>> 2. The files are accessible from both host H1 and host H2 because >>>> they >>>> are on shared storage or distributed storage system. Here the >>>> problem is "just" migrating the state from H1's NFS server to H2 >>>> so >>>> that file handles remain valid. >>> >>> Essentially this is the re-export case, and this makes a lot >>> more sense to me from a storage administration point of view. >>> >>> The pool of administered storage is not local to the physical >>> hosts running the guests, which is how I think cloud providers >>> would prefer to operate. >>> >>> User storage would be accessible via an NFS share, but managed >>> in a Ceph object (with redundancy, a common high throughput >>> backup facility, and secure central management of user >>> identities). >>> >>> Each host's NFS server could be configured to expose only the >>> the cloud storage resources for the tenants on that host. The >>> back-end storage (ie, Ceph) could operate on a private storage >>> area network for better security. >>> >>> The only missing piece here is support in Linux-based NFS >>> servers for transparent state migration. >> >> Not really. In a containerised world, we're going to see more and more >> cases where just a single process/application gets migrated from one >> NFS client to another (and yes, a re-exporter/proxy of NFS is just >> another client as far as the original server is concerned). >> IOW: I think we want to allow a client to migrate some parts of its >> lock state to another client, without necessarily requiring every >> process being migrated to have its own clientid. > > It wouldn't have to be every process, it'd be every container, right? > What's the disadvantage of per-container clientids? I guess you lose > the chance to share delegations and caches. Can't each container have it's own net namespace, and each net namespace have its own client ID? (I agree, btw, this class of problems should be considered in the new nfsv4 WG charter. Thanks for doing that, Trond). -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
