Chris Wright wrote:
> Yup. Just noting that API without clear users is the type of thing that
> is regularly rejected from Linux.
>
Yes. It is becoming clear from feedback from you and Andi that there
are things in the API that are unnecessary for Linux. But keep in mind,
they may be necessary for other operating systems. I think we should
probably drop the Linux changes to issue things like RDTSC and such via
VMI call wrappers. It does simplify the Linux interface.
But I still think they should be part of the spec - an optional part of
the spec, that need not be implemented by Linux or even by the
hypervisor. If some vendor or kernel combination finds that they are a
performance concern, as they readily could become, they can drop in the
functionality when and if they need it. No reason to complicate things
on either end, but also no reason to purposely add asymmetry to the spec
just because the current set of calls is sufficient for the currently
known fast paths.
>
>>> Many of these will look the same on x86-64, but the API is not
>>> 64-bit clean so has to be duplicated.
>>>
>> Yes, register pressure forces the PAE API to be slightly different from
>> the long mode API. But long mode has different register calling
>> conventions anyway, so it is not a big deal. The important thing is,
>> once the MMU mess is sorted out, the same interface can be used from C
>> code for both platforms, and the details about which lock primitives are
>> used can be hidden. The cost of which lock primitives to use differs on
>> 32-bit and 64-bit platforms, across vendor, and the style of the
>> hypervisor implementation (direct / writable / shadowed page tables).
>>
>
> My mistake, it makes perfect sense from ABI point of view.
>
>
>>> Is this the batching, multi-call analog?
>>>
>> Yes. This interface needs to be documented in a much better fashion.
>> But the idea is that VMI calls are mapped into Xen multicalls by
>> allowing deferred completion of certain classes of operations. That
>> same mode of deferred operation is used to batch PTE updates in our
>> implementation (although Xen uses writable page tables now, this used to
>> provide the same support facility in Xen as well). To complement this,
>> there is an explicit flush - and it turns out this maps very nicely,
>> getting rid of a lot of the XenoLinux changes around mmu_context.h.
>>
>
> Are these valid differences? Or did I misunderstand the batching
> mechanism?
>
> 1) can't use stack based args, so have to allocate each data structure,
> which could conceivably fail unless it's some fixed buffer.
>
We use a fixed buffer that is private to our VMI layer. It's a per-cpu
packing struct for hypercalls. Dynamically allocating from the kernel
inside the interface layer is a really great way to get into a whole lot
of trouble.
> 2) complicates the rom implementation slightly where implementation of
> each deferrable part of the API needs to have switch (am I deferred or
> not) to then build the batch, or make direct hypercall.
>
This is an overhead that is easily absorbed by the gain. The direct
hypercalls are mostly either always direct, or always queued. The page
table updates already have conditional logic to do the right thing, and
Xen doesn't require the queueing of these anymore anyways. And the
flush happens at an explicit point. The best approach can still be fine
tuned. You could have separate VMI calls for queued vs. non-queued
operation. But that greatly bloats the interface and doesn't make sense
for everything. I believe the best solution is to annotate this in the
VMI call itself. Consider the VMI call number, not as an integer, but
as an identifier tuple. Perhaps I'm going overboard here. Perhaps not.
31--------24-23---------16-15--------8-7-----------0
| family | call number | reserved | annotation |
---------------------------------------------------
Now, you have multiple families of calls -
0x00 legacy
0x01 CPU
0x02 Segmentation
0x03 MMU
0xFF reserved for experimentation
And each family has children:
0x03 MMU:
0x00 SetPTE
0x01 SetLongPTE
0x02 FlushTLB
Now, lets say I add a new feature, and I don't want to redefine part of
the interface. Lets say that feature is queuing of hypercalls. I have
this private, annotation field as part of the identifier for each
hypercall - in effect, really just the hypercall number.
And I don't want to break binary compatibility of the interface. So
what I do is I define a new annotation that is specific to the affected
calls.
0x00 SetPTE
0x00 - no annotation
0x01 - may be queued !
Now, the hypercall isn't any different. Hypervisors which are unware of
the annotation treat it no differently. But hypervisors that support
PTE queuing recognize it as a hint and use it appropriately.
Queuing is a common enough optimization that it might even make sense to
have a bit set aside in the call ID for it. Having this type of static
annotation allows you to get rid of the dynamic concerns you have.
The really nice thing about defining your interface this way is you have
a hierarchy of different classes of the interface, with the ability to
add new classes, new calls within a class, and new annotations
(upgrades, if you will) or those calls.
And it provides a natural way to query for supported families of support
- do you support a virtual event channel? Should I do some extra work
to give you MMU hints or not? And you can add extra, optional
functionality on to existing call sites. Something vary useful, if you
say, realize that you want to add a hint field to one of your calls
without breaking the old interface or forcing another vendor into
complicating the ir hypervisor. Which is most of what
paravirtualization is anyway. Extra, optionally used hints about how
things are being used that allow the hypervisor implementation to avoid
making costly assumptions to ensure correctness under unknown constraints.
Is this worth threshing out more? I think so, since it does provide a
nice value proposition as well as overcoming the rather clumsy top level
versioning scheme.
Thanks again for your feedback,
Zach
