On Tue, Sep 22, 2015 at 11:09:14PM +0100, Marc Zyngier wrote: > On Tue, 4 Aug 2015 06:52:01 +0100 > Bhushan Bharat <Bharat.Bhushan@xxxxxxxxxxxxx> wrote: > > > > > > > > -----Original Message----- > > > From: Pranavkumar Sawargaonkar [mailto:pranavkumar@xxxxxxxxxx] > > > Sent: Tuesday, August 04, 2015 11:18 AM > > > To: Bhushan Bharat-R65777 > > > Cc: kvm@xxxxxxxxxxxxxxx; Alex Williamson; kvmarm@xxxxxxxxxxxxxxxxxxxxx; > > > linux-arm-kernel@xxxxxxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; > > > christoffer.dall@xxxxxxxxxx; marc.zyngier@xxxxxxx; will.deacon@xxxxxxx; > > > bhelgaas@xxxxxxxxxx; arnd@xxxxxxxx; rob.herring@xxxxxxxxxx; > > > eric.auger@xxxxxxxxxx; patches@xxxxxxx; Yoder Stuart-B08248 > > > Subject: Re: [RFC 0/2] VFIO: Add virtual MSI doorbell support. > > > > > > Hi Bharat, > > > > > > On 28 July 2015 at 23:28, Alex Williamson <alex.williamson@xxxxxxxxxx> > > > wrote: > > > > On Tue, 2015-07-28 at 17:23 +0000, Bhushan Bharat wrote: > > > >> Hi Alex, > > > >> > > > >> > -----Original Message----- > > > >> > From: Alex Williamson [mailto:alex.williamson@xxxxxxxxxx] > > > >> > Sent: Tuesday, July 28, 2015 9:52 PM > > > >> > To: Pranavkumar Sawargaonkar > > > >> > Cc: kvm@xxxxxxxxxxxxxxx; kvmarm@xxxxxxxxxxxxxxxxxxxxx; linux-arm- > > > >> > kernel@xxxxxxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; > > > >> > christoffer.dall@xxxxxxxxxx; marc.zyngier@xxxxxxx; > > > >> > will.deacon@xxxxxxx; bhelgaas@xxxxxxxxxx; arnd@xxxxxxxx; > > > >> > rob.herring@xxxxxxxxxx; eric.auger@xxxxxxxxxx; patches@xxxxxxx; > > > >> > Bhushan Bharat-R65777; Yoder > > > >> > Stuart-B08248 > > > >> > Subject: Re: [RFC 0/2] VFIO: Add virtual MSI doorbell support. > > > >> > > > > >> > On Fri, 2015-07-24 at 14:33 +0530, Pranavkumar Sawargaonkar wrote: > > > >> > > In current VFIO MSI/MSI-X implementation, linux host kernel > > > >> > > allocates MSI/MSI-X vectors when userspace requests through vfio > > > ioctls. > > > >> > > Vfio creates irqfd mappings to notify MSI/MSI-X interrupts to the > > > >> > > userspace when raised. > > > >> > > Guest OS will see emulated MSI/MSI-X controller and receives an > > > >> > > interrupt when kernel notifies the same via irqfd. > > > >> > > > > > >> > > Host kernel allocates MSI/MSI-X using standard linux routines > > > >> > > like > > > >> > > pci_enable_msix_range() and pci_enable_msi_range(). > > > >> > > These routines along with requset_irq() in host kernel sets up > > > >> > > MSI/MSI-X vectors with Physical MSI/MSI-X addresses provided by > > > >> > > interrupt controller driver in host kernel. > > > >> > > > > > >> > > This means when a device is assigned with the guest OS, MSI/MSI-X > > > >> > > addresses present in PCIe EP are the PAs programmed by the host > > > >> > > linux > > > >> > kernel. > > > >> > > > > > >> > > In x86 MSI/MSI-X physical address range is reserved and iommu is > > > >> > > aware about these addreses and transalation is bypassed for these > > > address range. > > > >> > > > > > >> > > Unlike x86, ARM/ARM64 does not reserve MSI/MSI-X Physical address > > > >> > > range and all the transactions including MSI go through > > > >> > > iommu/smmu > > > >> > without bypass. > > > >> > > This requires extending current vfio MSI layer with additional > > > >> > > functionality for ARM/ARM64 by 1. Programing IOVA (referred as a > > > >> > > MSI virtual doorbell address) > > > >> > > in device's MSI vector as a MSI address. > > > >> > > This IOVA will be provided by the userspace based on the > > > >> > > MSI/MSI-X addresses reserved for the guest. > > > >> > > 2. Create an IOMMU mapping between this IOVA and > > > >> > > Physical address (PA) assigned to the MSI vector. > > > >> > > > > > >> > > This RFC is proposing a solution for MSI/MSI-X passthrough for > > > >> > ARM/ARM64. > > > >> > > > > >> > > > > >> > Hi Pranavkumar, > > > >> > > > > >> > Freescale has the same, or very similar, need, so any solution in > > > >> > this space will need to work for both ARM and powerpc. I'm not a > > > >> > big fan of this approach as it seems to require the user to > > > >> > configure MSI/X via ioctl and then call a separate ioctl mapping > > > >> > the doorbells. That's more code for the user, more code to get > > > >> > wrong and potentially a gap between configuring MSI/X and enabling > > > mappings where we could see IOMMU faults. > > > >> > > > > >> > If we know that doorbell mappings are required, why can't we set > > > >> > aside a bank of IOVA space and have them mapped automatically as > > > >> > MSI/X is being configured? Then the user's need for special > > > >> > knowledge and handling of this case is limited to setup. The IOVA > > > >> > space will be mapped and used as needed, we only need the user to > > > >> > specify the IOVA space reserved for this. Thanks, > > > >> > > > >> We probably need a mix of both to support Freescale PowerPC and ARM > > > >> based machines. > > > >> In this mix mode kernel vfio driver will reserve some IOVA for > > > >> mapping MSI page/s. > > > > > > > > If vfio is reserving pages independently from the user, this becomes > > > > what Marc called "shaping" the VM and what x86 effectively does. An > > > > interface extension should expose these implicit regions so the user > > > > can avoid them for DMA memory mapping. > > > > > > > >> If any other iova mapping will overlap with this then it will return > > > >> error and user-space. Ideally this should be choosen in such a way > > > >> that it never overlap, which is easy on some systems but can be > > > >> tricky on some other system like Freescale PowerPC. This is not > > > >> sufficient for at-least Freescale PowerPC based SOC. This is because > > > >> of hardware limitation, where we need to fit this reserved iova > > > >> address within aperture decided by user-space. So if we allow > > > >> user-space to change this reserved iova address to a value decided by > > > >> user-spece itself then we can support both ARM/PowerPC based > > > solutions. > > > > > > > > Yes, that's my intention, to allow userspace to specify the reserved > > > > region. I believe you have some additional restrictions on the number > > > > of MSI banks available and whether MSI banks can be shared, but I > > > > would hope that doesn't preclude a shared interface with ARM. > > > > > > > >> I have some implementation ready/tested with this approach and if > > > >> this approach looks good then I can submit a RFC patch. > > > > > > > > Yes, please post. Thanks, > > > > > > Could you please share a tentative timeline by which you will be posting your > > > patches ? > > > > I have not touched that code for a while, I am planning to send the > > patch in couple of weeks. > > Have we made any progress on this subject? It looks like a lot of time > has passed, but I haven't seen anything. Did I miss it? > Pranav is going to respin his series, because we are clarly not making progress on this front. Thanks, -Christoffer -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
