On 06/07/2010 01:42 PM, Avi Kivity wrote:
On 06/07/2010 08:04 PM, Anthony Liguori wrote:
I think we could also move the local APIC.
I'm not even sure we can safely move the ioapic/pic (mostly due to
churn). But the local APIC is so heavily accessed by the guest that
it's impossible to move it. Run an ftrace one day, especially on an
smp guest. Every IPI requires several APIC accesses. Before a halt a
tickless kernel sets the wakeup timer. EOIs.
To optimize device models, we've tended to put the full device model
in the kernel whereas the hardware vendors have tended to put only
the fast paths of the devices models in hardware.
For instance, we could introduce a userspace interface similar to
vapic support whereas a shared page that mapped the APIC's layout was
used with a mask to select which registers trapped on read/write.
That leads to very problematic interfaces. When you separate along a
device boundary, you have a spec that defines the software
interfaces. When you separate along a boundary that you define, it's
up to you to get everything right.
In fact with the ioapic/pic/lapic one of the problems is that the
interconnection between the devices that is not well defined, and
that's where we have bugs.
That said, I can understand an argument that the local APIC is part
of the CPU state since it's a very special type of device.
A better example would be a generic counter kernel mechanism. I can
envision such a device as doing nothing more than providing a
read-only view of a counter with a userspace configurable divider and
width. Any write to the counter or read of any other byte outside
the counter register would result in a trap to userspace.
What about latches? byte access to word registers? There will be as
many special cases as there are timers.
If the kernel supported a bytecode/jit facility I'd happily use that
to download portions of the device model into the kernel.
That should allow both the PIT and the HPET to be accelerated with
minimal effort in the kernel.
IMO it's probably more effort than porting HPET to the kernel. Try
outlining an interface that supports PIT, HPET, RTC, and ACPI PMTIMER.
I was referring specifically to time sources, not time events.
An accelerated counter for HPET is pretty trivial. It's a 32-bit
register that's actually a nanosecond value in qemu. We need to be able
to set an offset from the host wall clock time, a means to stop it, and
a means to start it.
The PIT is latched so the kernel needs to know enough about how to
decode the PIT state to understand the latching. There's very little
state associated with latching though so I don't think this is a huge
problem. It's a fixed value write to a fixed register followed by a
read to a fixed register. The act of latching doesn't effect the state
beyond the fact that you need to save the latched value in the event
that you have a live migration before reading the latched value.
The PMTIMER is also pretty straight forward. It's a variable port
address (that's fixed during execution).
Even if we require three separate interfaces, the interfaces are so
simply that it seems like an obvious win.
I'd be in favor of a straight port to userspace. We already have the
interfaces to communicate with an external device model for these
devices so let's just take the kernel code and stick it into
dedicated threads in userspace.
Currently we support an all-or-nothing approach. I don't think local
APIC in userspace is worthwhile. Esp. as it will slow down vhost and
assigned devices significantly - interrupts will have to be mediated
by userspace.
Yeah, as I said, I can understand the arguments for keeping the lapic in
the kernel.
I think it's easier to then work to merge the two bits of code in the
same tree than it is to try and take out-of-tree code and merge it
incrementally.
Are you talking about qemu.git/qemu-kvm.git? That's the least of my
concerns, I'm worried about kvm.git.
qemu.git.
5. Risk
We may find out after all this is implemented that performance is
not acceptable and all the work will have to be dropped.
That's another advantage to a straight port to userspace. We can
collect performance data with only a modest amount of engineering
effort.
Port what exactly? We have a userspace irqchip implementation. What
we don't have is just the ioapic/pic/pit in userspace, and the only
way to try it out is to implement the whole thing.
If you take the kernel code and do a pretty straight port: switching
kernel functions to libc functions and maintaining all the existing
locking via pthreads, you could then implement a very simple MMIO/PIO
dispatch mechanism in the kvm code that shortcutted those devices before
we ever hit the qemu_mutex and the traditional qemu code paths. It
should be a relatively easy conversion and it gives a proper vehicle for
doing experimentations.
In fact, you could pretty quickly determine viability by porting the PIT
to userspace and implementing a vpit interface in the kernel that
allowed the channel 0 counters to be latched and read within lightweight
exits.
Regards,
Anthony Liguori
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
