> -----Original Message----- > From: Pranavkumar Sawargaonkar [mailto:pranavkumar@xxxxxxxxxx] > Sent: Tuesday, August 04, 2015 11:18 AM > To: Bhushan Bharat-R65777 > Cc: kvm@xxxxxxxxxxxxxxx; Alex Williamson; kvmarm@xxxxxxxxxxxxxxxxxxxxx; > linux-arm-kernel@xxxxxxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; > christoffer.dall@xxxxxxxxxx; marc.zyngier@xxxxxxx; will.deacon@xxxxxxx; > bhelgaas@xxxxxxxxxx; arnd@xxxxxxxx; rob.herring@xxxxxxxxxx; > eric.auger@xxxxxxxxxx; patches@xxxxxxx; Yoder Stuart-B08248 > Subject: Re: [RFC 0/2] VFIO: Add virtual MSI doorbell support. > > Hi Bharat, > > On 28 July 2015 at 23:28, Alex Williamson <alex.williamson@xxxxxxxxxx> > wrote: > > On Tue, 2015-07-28 at 17:23 +0000, Bhushan Bharat wrote: > >> Hi Alex, > >> > >> > -----Original Message----- > >> > From: Alex Williamson [mailto:alex.williamson@xxxxxxxxxx] > >> > Sent: Tuesday, July 28, 2015 9:52 PM > >> > To: Pranavkumar Sawargaonkar > >> > Cc: kvm@xxxxxxxxxxxxxxx; kvmarm@xxxxxxxxxxxxxxxxxxxxx; linux-arm- > >> > kernel@xxxxxxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; > >> > christoffer.dall@xxxxxxxxxx; marc.zyngier@xxxxxxx; > >> > will.deacon@xxxxxxx; bhelgaas@xxxxxxxxxx; arnd@xxxxxxxx; > >> > rob.herring@xxxxxxxxxx; eric.auger@xxxxxxxxxx; patches@xxxxxxx; > >> > Bhushan Bharat-R65777; Yoder > >> > Stuart-B08248 > >> > Subject: Re: [RFC 0/2] VFIO: Add virtual MSI doorbell support. > >> > > >> > On Fri, 2015-07-24 at 14:33 +0530, Pranavkumar Sawargaonkar wrote: > >> > > In current VFIO MSI/MSI-X implementation, linux host kernel > >> > > allocates MSI/MSI-X vectors when userspace requests through vfio > ioctls. > >> > > Vfio creates irqfd mappings to notify MSI/MSI-X interrupts to the > >> > > userspace when raised. > >> > > Guest OS will see emulated MSI/MSI-X controller and receives an > >> > > interrupt when kernel notifies the same via irqfd. > >> > > > >> > > Host kernel allocates MSI/MSI-X using standard linux routines > >> > > like > >> > > pci_enable_msix_range() and pci_enable_msi_range(). > >> > > These routines along with requset_irq() in host kernel sets up > >> > > MSI/MSI-X vectors with Physical MSI/MSI-X addresses provided by > >> > > interrupt controller driver in host kernel. > >> > > > >> > > This means when a device is assigned with the guest OS, MSI/MSI-X > >> > > addresses present in PCIe EP are the PAs programmed by the host > >> > > linux > >> > kernel. > >> > > > >> > > In x86 MSI/MSI-X physical address range is reserved and iommu is > >> > > aware about these addreses and transalation is bypassed for these > address range. > >> > > > >> > > Unlike x86, ARM/ARM64 does not reserve MSI/MSI-X Physical address > >> > > range and all the transactions including MSI go through > >> > > iommu/smmu > >> > without bypass. > >> > > This requires extending current vfio MSI layer with additional > >> > > functionality for ARM/ARM64 by 1. Programing IOVA (referred as a > >> > > MSI virtual doorbell address) > >> > > in device's MSI vector as a MSI address. > >> > > This IOVA will be provided by the userspace based on the > >> > > MSI/MSI-X addresses reserved for the guest. > >> > > 2. Create an IOMMU mapping between this IOVA and > >> > > Physical address (PA) assigned to the MSI vector. > >> > > > >> > > This RFC is proposing a solution for MSI/MSI-X passthrough for > >> > ARM/ARM64. > >> > > >> > > >> > Hi Pranavkumar, > >> > > >> > Freescale has the same, or very similar, need, so any solution in > >> > this space will need to work for both ARM and powerpc. I'm not a > >> > big fan of this approach as it seems to require the user to > >> > configure MSI/X via ioctl and then call a separate ioctl mapping > >> > the doorbells. That's more code for the user, more code to get > >> > wrong and potentially a gap between configuring MSI/X and enabling > mappings where we could see IOMMU faults. > >> > > >> > If we know that doorbell mappings are required, why can't we set > >> > aside a bank of IOVA space and have them mapped automatically as > >> > MSI/X is being configured? Then the user's need for special > >> > knowledge and handling of this case is limited to setup. The IOVA > >> > space will be mapped and used as needed, we only need the user to > >> > specify the IOVA space reserved for this. Thanks, > >> > >> We probably need a mix of both to support Freescale PowerPC and ARM > >> based machines. > >> In this mix mode kernel vfio driver will reserve some IOVA for > >> mapping MSI page/s. > > > > If vfio is reserving pages independently from the user, this becomes > > what Marc called "shaping" the VM and what x86 effectively does. An > > interface extension should expose these implicit regions so the user > > can avoid them for DMA memory mapping. > > > >> If any other iova mapping will overlap with this then it will return > >> error and user-space. Ideally this should be choosen in such a way > >> that it never overlap, which is easy on some systems but can be > >> tricky on some other system like Freescale PowerPC. This is not > >> sufficient for at-least Freescale PowerPC based SOC. This is because > >> of hardware limitation, where we need to fit this reserved iova > >> address within aperture decided by user-space. So if we allow > >> user-space to change this reserved iova address to a value decided by > >> user-spece itself then we can support both ARM/PowerPC based > solutions. > > > > Yes, that's my intention, to allow userspace to specify the reserved > > region. I believe you have some additional restrictions on the number > > of MSI banks available and whether MSI banks can be shared, but I > > would hope that doesn't preclude a shared interface with ARM. > > > >> I have some implementation ready/tested with this approach and if > >> this approach looks good then I can submit a RFC patch. > > > > Yes, please post. Thanks, > > Could you please share a tentative timeline by which you will be posting your > patches ? I have not touched that code for a while, I am planning to send the patch in couple of weeks. > Also are you planning to post counterpart patches for qemu or kvmtool ? I will send only QEMU side changes. Thanks -Bharat > > Thanks, > Pranav ��.n��������+%������w��{.n�����o�^n�r������&��z�ޗ�zf���h���~����������_��+v���)ߣ�
