On 08/04/2019 12:36, Joao Martins wrote: > On 4/8/19 7:44 AM, Juergen Gross wrote: >> On 12/03/2019 18:14, Joao Martins wrote: >>> On 2/22/19 4:59 PM, Paolo Bonzini wrote: >>>> On 21/02/19 12:45, Joao Martins wrote: >>>>> On 2/20/19 9:09 PM, Paolo Bonzini wrote: >>>>>> On 20/02/19 21:15, Joao Martins wrote: >>>>>>> 2. PV Driver support (patches 17 - 39) >>>>>>> >>>>>>> We start by redirecting hypercalls from the backend to routines >>>>>>> which emulate the behaviour that PV backends expect i.e. grant >>>>>>> table and interdomain events. Next, we add support for late >>>>>>> initialization of xenbus, followed by implementing >>>>>>> frontend/backend communication mechanisms (i.e. grant tables and >>>>>>> interdomain event channels). Finally, introduce xen-shim.ko, >>>>>>> which will setup a limited Xen environment. This uses the added >>>>>>> functionality of Xen specific shared memory (grant tables) and >>>>>>> notifications (event channels). >>>>>> >>>>>> I am a bit worried by the last patches, they seem really brittle and >>>>>> prone to breakage. I don't know Xen well enough to understand if the >>>>>> lack of support for GNTMAP_host_map is fixable, but if not, you have to >>>>>> define a completely different hypercall. >>>>>> >>>>> I guess Ankur already answered this; so just to stack this on top of his comment. >>>>> >>>>> The xen_shim_domain() is only meant to handle the case where the backend >>>>> has/can-have full access to guest memory [i.e. netback and blkback would work >>>>> with similar assumptions as vhost?]. For the normal case, where a backend *in a >>>>> guest* maps and unmaps other guest memory, this is not applicable and these >>>>> changes don't affect that case. >>>>> >>>>> IOW, the PV backend here sits on the hypervisor, and the hypercalls aren't >>>>> actual hypercalls but rather invoking shim_hypercall(). The call chain would go >>>>> more or less like: >>>>> >>>>> <netback|blkback|scsiback> >>>>> gnttab_map_refs(map_ops, pages) >>>>> HYPERVISOR_grant_table_op(GNTTABOP_map_grant_ref,...) >>>>> shim_hypercall() >>>>> shim_hcall_gntmap() >>>>> >>>>> Our reasoning was that given we are already in KVM, why mapping a page if the >>>>> user (i.e. the kernel PV backend) is himself? The lack of GNTMAP_host_map is how >>>>> the shim determines its user doesn't want to map the page. Also, there's another >>>>> issue where PV backends always need a struct page to reference the device >>>>> inflight data as Ankur pointed out. >>>> >>>> Ultimately it's up to the Xen people. It does make their API uglier, >>>> especially the in/out change for the parameter. If you can at least >>>> avoid that, it would alleviate my concerns quite a bit. >>> >>> In my view, we have two options overall: >>> >>> 1) Make it explicit, the changes the PV drivers we have to make in >>> order to support xen_shim_domain(). This could mean e.g. a) add a callback >>> argument to gnttab_map_refs() that is invoked for every page that gets looked up >>> successfully, and inside this callback the PV driver may update it's tracking >>> page. Here we no longer have this in/out parameter in gnttab_map_refs, and all >>> shim_domain specific bits would be a little more abstracted from Xen PV >>> backends. See netback example below the scissors mark. Or b) have sort of a >>> translate_gref() and put_gref() API that Xen PV drivers use which make it even >>> more explicit that there's no grant ops involved. The latter is more invasive. >>> >>> 2) The second option is to support guest grant mapping/unmapping [*] to allow >>> hosting PV backends inside the guest. This would remove the Xen changes in this >>> series completely. But it would require another guest being used >>> as netback/blkback/xenstored, and less performance than 1) (though, in theory, >>> it would be equivalent to what does Xen with grants/events). The only changes in >>> Linux Xen code is adding xenstored domain support, but that is useful on its own >>> outside the scope of this work. >>> >>> I think there's value on both; 1) is probably more familiar for KVM users >>> perhaps (as it is similar to what vhost does?) while 2) equates to implementing >>> Xen disagregation capabilities in KVM. >>> >>> Thoughts? Xen maintainers what's your take on this? >> >> What I'd like best would be a new handle (e.g. xenhost_t *) used as an >> abstraction layer for this kind of stuff. It should be passed to the >> backends and those would pass it on to low-level Xen drivers (xenbus, >> event channels, grant table, ...). >> > So if IIRC backends would use the xenhost layer to access grants or frames > referenced by grants, and that would grok into some of this. IOW, you would have > two implementors of xenhost: one for nested remote/local events+grants and > another for this "shim domain" ? As I'd need that for nested Xen I guess that would make it 3 variants. Probably the xen-shim variant would need more hooks, but that should be no problem. >> I was planning to do that (the xenhost_t * stuff) soon in order to add >> support for nested Xen using PV devices (you need two Xenstores for that >> as the nested dom0 is acting as Xen backend server, while using PV >> frontends for accessing the "real" world outside). >> >> The xenhost_t should be used for: >> >> - accessing Xenstore >> - issuing and receiving events >> - doing hypercalls >> - grant table operations >> > > In the text above, I sort of suggested a slice of this on 1.b) with a > translate_gref() and put_gref() API -- to get the page from a gref. This was > because of the flags|host_addr hurdle we depicted above wrt to using using grant > maps/unmaps. You think some of the xenhost layer would be ammenable to support > this case? I think so, yes. > >> So exactly the kind of stuff you want to do, too. >> > Cool idea! In the end you might make my life easier for nested Xen. :-) Do you want to have a try with that idea or should I do that? I might be able to start working on that in about a month. Juergen
