RE: [PATCH V9 00/11] IOMMU probe deferral support

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Sricharan,

> -----Original Message-----
> From: Sricharan R [mailto:sricharan@xxxxxxxxxxxxxx]
> Sent: Friday, March 24, 2017 7:10 AM
> To: Wangzhou (B); robin.murphy@xxxxxxx; will.deacon@xxxxxxx;
> joro@xxxxxxxxxx; lorenzo.pieralisi@xxxxxxx; iommu@lists.linux-
> foundation.org; linux-arm-kernel@xxxxxxxxxxxxxxxxxxx; linux-arm-
> msm@xxxxxxxxxxxxxxx; m.szyprowski@xxxxxxxxxxx;
> bhelgaas@xxxxxxxxxx; linux-pci@xxxxxxxxxxxxxxx; linux-
> acpi@xxxxxxxxxxxxxxx; tn@xxxxxxxxxxxx; hanjun.guo@xxxxxxxxxx;
> okaya@xxxxxxxxxxxxxx
> Cc: Shameerali Kolothum Thodi
> Subject: Re: [PATCH V9 00/11] IOMMU probe deferral support
> 
> Hi Zhou,
> 
> On 3/24/2017 9:23 AM, Zhou Wang wrote:
> > On 2017/3/10 3:00, Sricharan R wrote:
> >> This series calls the dma ops configuration for the devices at a
> >> generic place so that it works for all busses.
> >> The dma_configure_ops for a device is now called during the
> >> device_attach callback just before the probe of the bus/driver is
> >> called. Similarly dma_deconfigure is called during
> >> device/driver_detach path.
> >>
> >> pci_bus_add_devices    (platform/amba)(_device_create/driver_register)
> >>        |                         |
> >> pci_bus_add_device     (device_add/driver_register)
> >>        |                         |
> >> device_attach           device_initial_probe
> >>        |                         |
> >> __device_attach_driver    __device_attach_driver
> >>        |
> >> driver_probe_device
> >>        |
> >> really_probe
> >>        |
> >> dma_configure
> >>
> >> Similarly on the device/driver_unregister path
> >> __device_release_driver is called which inturn calls dma_deconfigure.
> >>
> >> Rebased the series against mainline 4.11-rc1. Applies and builds
> >> cleanly against mainline and linux-next. There is a conflict with
> >> patch#9 against iommu-next, but that should go away eventually as
> >> iommu-next is rebased against 4.11-rc1.
> >>
> >> * Tested with platform and pci devices for probe deferral
> >>   and reprobe on arm64 based platform.
> >
> > Hi Sricharan,
> >
> > I applied this series on v4.11-rc1 to test PCIe pass through in
> > HiSilicon
> > D05 board(with Intel 82599 networking card). It failed.
> >
> > After I used:
> >
> > echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override
> > echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
> > echo 0002:81:10.0 > /sys/bus/pci/drivers_probe
> >
> > to bind vfio-pci driver to Intel 82599 networking card VF.
> >
> > I got log in host:
> > [...]
> > [  414.275818] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual
> > Function Network Driver - version 3.2.2-k [  414.275824] ixgbevf: Copyright
> (c) 2009 - 2015 Intel Corporation.
> > [  414.276647] ixgbe 0002:81:00.0 eth12: SR-IOV enabled with 1 VFs [
> > 414.342252] pcieport 0002:80:00.0: can't derive routing for PCI INT A
> > [  414.342255] ixgbe 0002:81:00.0: PCI INT A: no GSI [  414.343348]
> > ixgbe 0002:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue
> > count = 4 [  414.448135] pci 0002:81:10.0: [8086:10ed] type 00 class
> > 0x020000 [  414.448713] iommu: Adding device 0002:81:10.0 to group 4 [
> > 414.449798] ixgbevf 0002:81:10.0: enabling device (0000 -> 0002) [
> > 414.451101] ixgbevf 0002:81:10.0: PF still in reset state.  Is the PF interface
> up?
> > [  414.451103] ixgbevf 0002:81:10.0: Assigning random MAC address [
> > 414.451414] ixgbevf 0002:81:10.0: be:30:8f:ed:f8:02 [  414.451417]
> > ixgbevf 0002:81:10.0: MAC: 1 [  414.451418] ixgbevf 0002:81:10.0:
> > Intel(R) 82599 Virtual Function [  414.464271] VFIO - User Level
> > meta-driver version: 0.3 [  414.570074] ixgbe 0002:81:00.0: registered
> > PHC device on eth12
> > [  414.700493] specified DMA range outside IOMMU capability
> <-- error here
> > [  414.700496] Failed to set up IOMMU for device 0002:81:10.0; retaining
> platform DMA ops        <-- error here
> 
> Looks like this triggers the start of the bug.
> So the below check in iommu_dma_init_domain fails,
> 
>          if (domain->geometry.force_aperture) {
>                  if (base > domain->geometry.aperture_end ||
>                      base + size <= domain->geometry.aperture_start) {
> 
> and the rest goes out of sync after that. Can you print out the base,
> aperture_start and end values to see why the check fails ?

dev_info(dev, "0x%llx 0x%llx, 0x%llx 0x%llx, 0x%llx 0x%llx\n", base, size, domain->geometry.aperture_start, domain->geometry.aperture_end, *dev->dma_mask, dev->coherent_dma_mask);

[  183.752100] ixgbevf 0000:81:10.0: 0x0 0x100000000, 0x0 0xffffffffffff, 0xffffffff 0xffffffff
.....
[  319.508037] vfio-pci 0000:81:10.0: 0x0 0x0, 0x0 0xffffffffffff, 0xffffffffffffffff 0xffffffffffffffff

Yes, size seems to be the problem here. When the VF  device gets attached to vfio-pci,
somehow the dev->coherent_dma_mask is set to 64 bits and size become zero.

@@ -107,7 +107,7 @@ int of_dma_configure(struct device *dev, struct device_node *np)
  	ret = of_dma_get_range(np, &dma_addr, &paddr, &size);
  	if (ret < 0) {
  		dma_addr = offset = 0;
 -		size = dev->coherent_dma_mask + 1;
 +		size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);

@@ -1386,7 +1387,8 @@ int acpi_dma_configure(struct device *dev, enum dev_dma_attr attr)
  	 * Assume dma valid range starts at 0 and covers the whole
  	 * coherent_dma_mask.
  	 */
 -	arch_setup_dma_ops(dev, 0, dev->coherent_dma_mask + 1, iommu,
 +	size = max(dev->coherent_dma_mask, dev->coherent_dma_mask + 1);
 +	arch_setup_dma_ops(dev, 0, size, iommu,
  			   attr == DEV_DMA_COHERENT);

With the above fixes, DT boot works fine. But we still get the below crash on ACPI

> > [  402.581445] kernel BUG at drivers/iommu/arm-smmu-v3.c:1064!
> > [  402.587007] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
> > [  402.592479] Modules linked in: vfio_iommu_type1 vfio_pci irqbypass
> vfio_virqfd vfio ixgbevf ixgb

> The change that this series does is trying to add the dma/iommu ops to the
> device after the iommu is actually probed.
> So in your working case, does the device initially gets hooked to iommu_ops
> and the above same check passes in working case ?

I believe so. Because didn't notice the "specified DMA range outside IOMMU capability"
in the working case.
 
Thanks,
Shameer



[Index of Archives]     [DMA Engine]     [Linux Coverity]     [Linux USB]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [Greybus]

  Powered by Linux