Hi Daniel:
Really appreciate the reply! :) I just read the code again. And we will prepare the PoC patch based on the new ideas! :)
-----Original Message-----
From: Daniel Vetter [mailto:daniel.vetter@xxxxxxxx] On Behalf Of Daniel Vetter
Sent: Wednesday, September 09, 2015 4:40 AM
To: Wang, Zhi A
Cc: intel-gfx@xxxxxxxxxxxxxxxxxxxxx; Vetter, Daniel
Subject: Re: About GVT-g Interrupt in i915 Host
On Tue, Sep 08, 2015 at 04:03:02PM +0000, Wang, Zhi A wrote:
> BACKGROUND
>
> Under Intel GVT-g environment, HW interrupts are shared among different VMs.
>
> Because of the sharing, to enable or disable a HW interrupt becomes a bit
> complicated. Considered that a virtual machine may request to disable a
> interrupt which may be using by other virtual machines. The GEN IRQ register
> value has to be re-calculated before it gets updated to HW.
>
> i.e.
>
> - Virtual machine(VM) A wants to *disable* a PIPE VBLANK interrupt.
> - Virtual machine B wants to *enable* a PIPE VBLANK interrupt.
>
> The final result is to *enable* the HW VBLANK interrupt, so that we can:
> - Get VBLANK interrupt from HW for virtual machine B.
>
> Meanwhile, we choose not to inject VBLANK interrupt into virtual machine A,
> which actually doesn't want VBLANK interrupt at this time.
>
> Current Implementation - Interrupt Injection in GVT-g
>
> +-------------------------------+ +------------------+
> | Linux Guest | | Linux Guest |
> | I915 (Guest Mode) *Upstreamed | | i915 (Guest Mode)|
> +--------------------^----------+ +----^-------------+
> | |
> INJECT! | | INJECT!
> | |
> +---------------------|-----------------|------------+
> | +------------+ +-------------------------------+ |
> | |i915 * | | GVT-g | |
> | |(Host Mode) | +-+-----------------------------+ |
> | +-----^------+ | Host Linux Kernel |
> +-------|-----------|----^---------------------------+
> +-----------+ |
> INJECT! | Interrupts
> +------------------------|---------------------------+
> | Hypervisor |
> +------------------------|---------------------------+
> +------------------------|---------------------------+
> | GPU Hardware |
> +----------------------------------------------------+
>
> Current implementation treats i915 host(DOM0) like a VM, all the interrupt
> register access will be trapped and emulated by GVT-g.
>
> The most obvious advantange is minimum i915 host code change:
>
> - add some hooks in I915_READ()/I915_WRITE() to trap host common register access
> - add some hooks in i915 irq initilization path to trap host interrupt
> - add some hooks in i915_gem_gtt.c to trap host GTT mmio access.
>
> But it brings some architectural conflict. so we purpose another approach.
>
> EXPECTED ARCHITECTURE
>
> +-------------------------------+ +------------------+
> | Linux Guest | | Linux Guest |
> | I915 (Guest Mode) *Upstreamed | | i915 (Guest Mode)|
> +--------------------^----------+ +--------^---------+
> | |
> INJECT! +------------------+ | INJECT!
> | |
> +----------------------------------------|--|---------+
> | +-------------------------+ +|--|-------+ |
> | |i915 +-----------> GVT-g | |
> | |(Host Mode) | DISPATCH! +-----------+ |
> | +-------------------^-----+ |
> +---------------------|-------------------------------+
> | Interrupt
> +---------------------|-------------------------------+
> | Hypervisor |
> +-----------------------------------------------------+
> +---------------------|-------------------------------+
> | GPU Hardware |
> +-----------------------------------------------------+
>
> [Host i915 manages all interrupts]
>
> DESIGN
>
> To achieve such kind of an approach, an ideal design in i915 is:
>
> Introduce two IRQ register caches into i915, one for host and one for GVT-g.
>
> Then we provide an central IRQ register mainpulation interface to other
> components of host i915, so we can log all interrupt register values of
> host i915.
>
> Meanwhile, GVT-g will calculate the IRQ register values according to all
> virtual machines and update GVT-g IRQ register cache inside i915 host.
>
> When i915 host itself wants to update IRQ registers or GVT-g requests i915 host
> to update IRQ registers, all related IRQ registers of i915 host and GVT-g
> will be re-calculated and combined together and written into HW later.
>
> This figure shows the details of the design above
>
> +----------------------------------------------------+
> | DRM / i915 components IRQ manipulation clients |
> | drm_irq_{install,uninstall}... |
> +--------------------------^-------------------------+
> | +-------+
> +--------------------------|-------------------------+ | |
> | i915: i915_irq.c | | +----Guest |
> | +---------------------v-----------------------+ | COMBINE | | |
> | | Central IRQ register manipulation interface | | ALL GUEST | +-------+
> | +---------------------------------------------+ | vIRQ |
> | | REGISTERS! |
> | +--------------------+ +--------------------+ | +-------+ | +-------+
> | | i915 host | | GVT-g | | | | | | |
> | | IRQ Register Cache | | IRQ Register Cache <-----GVT-g <--+----Guest |
> | | | | | | | | | |
> | +--------+-----------+ +---------+----------+ | +-------+ +-------+
> | | COMBINE BOTH! | |
> | +------------|------------+ |
> | | |
> | +------------v------------+ *Cache |
> | | *HW IRQ Register Cache | combined |
> | +------------+----------- + values |
> +--------------------------|-------------------------+
> |
> +--------------------------v-------------------------+
> | GPU Hardware |
> +----------------------------------------------------+
> [ Another approach without trapping i915 host ]
>
> A SIMPLER DESIGN
>
> We have been cooked some PoC patches based on above ideal design. After some discussions with Joonas,
> we thought it was still a bit complicated. As in the design, we have to record i915 host irq register
> values, we have to introduce a framework to support the interrupt register value logging. And the
> major disadvantage is it will affect the native code path, mean while LOC is also a problem.
>
> So we think over and over again to see if it's possible for GVT-g to require less interrupt from i915
> host.
>
> As we are planning to do full display virtualization. For guest, all display related interrupts
> are emulated. So GVT-g doesn't need i915 host pass it these interrupts. For now, it seems GVT-g only
> needs i915 host forwards some GT interrupts used to emulate the status of guest workload execustion:
>
> - PIPE_CONTROL_NOTIFY used by Windows guest
> - MI_USER_INTERRUPT used by Linux guest
>
> Let these two interrupt always be enabled under i915 host mode.
>
> - CONTEXT SWITCH interrupt
>
> This interrupt has been always enablined under i915 host mode. So no extra modification is needed.
>
> Then call vgt interrupt event forwarding handler at the end of gen8_irq_handler.
>
> Maybe this should be simplest design.
>
> Welcome to discuss!
Imo the simplest approach is to just refcount users for each interrupt.
And then GVT-g can simply grab a reference for each guest, like i915 host
code already does. An example would be the ring MI_USER_NOTIFY interrupt.
For that we have two main functions:
intel_ring_buffer->irq_get() and intel_ring_buffer->irq_put(). Currently
there's only one user for this in __i915_wait_request, but note that this
one is concurrent (you can call this function without holding
dev->struct_mutex). The ->irq_get/put() implementations have internally a
refcount to make sure the interrupt itself will get enabled for the first
user and only disabled once the last user stops.
For vblanks we have something similar in drm_irq.c with
drm_vblank_get/put(). That's imo how interrupt enabling should be
coordinated between gvt and i915 host code - gvt code should just use the
correct irq_get/put() functions for each client vm.
The other part is handling such interrupts, and for those I think the
simplest approach is to insert a callback into gvt code from the
i915_irq.c irq handler. For MI_NOTIFY_USER that would be notify_ring
(which is btw violating our naming convention that all interrupt handlers
should have an _irq_handler suffic). And for vblanks it would be
intel_pipe_vblank_handle. For other interrupts there's similar things and
for some we might need to create new infrastructure right away.
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
ohttp://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@xxxxxxxxxxxxxxxxxxxxx
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
