On Fri, 9 Feb 2018 10:31:57 -0800 Ravi Kerur <rkerur@xxxxxxxxx> wrote: > Hi, > > I am running into an issue when DPDK is started with intel_iommu=on > via GRUB command. Problem is not seen with regular kernel driver, > error messages show when DPDK is started and happens for both PF and > VF interfaces. Discussing with DPDK folks it is not a DPDK issue hence > sending an email to this list. > > Workaround is to use "iommu=pt" but I want iommu enabled in my setup. > > My understanding is that 'Address width' is reported as '46' on host > and '39' on guest and it could be causing the problem but not certain > so kindly let me know how to resolve this issue. > > I tried influencing guest address width with 'host-phys-bits=true and > phys-bits=46' properties when instantiating a guest but it still ends > with address width 39. > > I have tried 'x-aw-bits=48', still see DMAR errors. > > Following are the details > > (1) Linux kernel 4.9 (host and guest), Qemu 2.11 > > (2) DPDK 17.05 > > (3) IXGBE details > ethtool -i enp4s0f0 (PF driver) > driver: ixgbe > version: 5.3.3 > firmware-version: 0x800007b8, 1.1018.0 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > ethtool -i enp4s16f2 (VF driver) > driver: ixgbevf > version: 4.3.2 > firmware-version: > bus-info: 0000:04:10.2 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: no > supports-register-dump: yes > supports-priv-flags: no > > Bus info Device Class Description > ========================================================= > pci@0000:01:00.0 ens11f0 network 82599ES 10-Gigabit > SFI/SFP+ Network Connection > pci@0000:01:00.1 ens11f1 network 82599ES 10-Gigabit > SFI/SFP+ Network Connection > pci@0000:04:00.0 enp4s0f0 network 82599ES 10-Gigabit > SFI/SFP+ Network Connection > pci@0000:04:00.1 enp4s0f1 network 82599ES 10-Gigabit > SFI/SFP+ Network Connection > pci@0000:04:10.0 enp4s16 network Illegal Vendor ID > pci@0000:04:10.2 enp4s16f2 network Illegal Vendor ID > > (5) Kernel dmesg on host > > dmesg | grep -e DMAR -e IOMMU > [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I > 00000001 INTL 20091013) > [ 0.000000] DMAR: IOMMU enabled > [ 0.519368] DMAR: Host address width 46 > [ 0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > [ 0.538073] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap > d2078c106f0466 ecap f020df > [ 0.554253] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 > [ 0.565074] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap > d2078c106f0466 ecap f020df > [ 0.581259] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff > [ 0.593980] DMAR: ATSR flags: 0x0 > [ 0.600802] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0 > [ 0.613532] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1 > [ 0.626265] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 IOMMU 0 > [ 0.639177] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 IOMMU 1 > [ 0.652089] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 IOMMU 1 > [ 0.664996] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 > [ 0.675984] DMAR-IR: Queued invalidation will be enabled to support > x2apic and Intr-remapping. > [ 0.694475] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 9.637093] DMAR: dmar1: Using Queued invalidation > [ 9.646945] DMAR: Setting RMRR: > [ 9.653942] DMAR: Setting identity map for device 0000:00:1d.0 > [0x7bbc6000 - 0x7bbd4fff] > [ 9.670513] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 9.681656] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 > - 0xffffff] > [ 9.696630] DMAR: Intel(R) Virtualization Technology for Directed I/O > [ 2605.450811] DMAR: DRHD: handling fault status reg 2 > [ 2605.450814] DMAR: [DMA Read] Request device [04:10.0] fault addr > 33a128000 [fault reason 06] PTE Read access is not set > [ 2607.450907] DMAR: DRHD: handling fault status reg 102 > [ 2607.450910] DMAR: [DMA Read] Request device [04:10.0] fault addr > 33a128000 [fault reason 06] PTE Read access is not set > [ 4539.597735] DMAR: DRHD: handling fault status reg 202 > [ 4539.597737] DMAR: [DMA Read] Request device [04:10.0] fault addr > 33a128000 [fault reason 06] PTE Read access is not set It's a read fault to address 0x33a128000, just below 13G. I don't see why that would have anything to do with the emulated VT-d address width, we're well under 39bits. Is the VM configured with enough memory that this would be a valid guest physical address? Is the VM configured with more than 39bits of physical address space (1TB)? When you say that iommu=pt is a workaround, is that done in the host or guest? I don't see why it would particularly matter for either since the device is removed from the passthrough domain for assignment to the guest and for use in guest userspace with vfio/dpdk, so long as you're not somehow using no-iommu in the guest even though you have an iommu. What makes the DPDK folks confident that this isn't a driver bug? AFAICT, a stray read from the driver would generate exactly this sort of log. It's also possible that VT-d emulation hit a bug and didn't map this page correctly, but what's so unique about this page? (Cc PeterX). Thanks, Alex > (6) dmesg on guest > # dmesg | grep -e DMAR -e IOMMU > [ 0.000000] ACPI: DMAR 0x000000007FFE201D 000050 (v01 BOCHS > BXPCDMAR 00000001 BXPC 00000001) > [ 0.000000] DMAR: IOMMU enabled > [ 1.387988] DMAR: Host address width 39 > [ 1.389203] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 > [ 1.390692] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap > 12008c22260286 ecap f00f5e > [ 1.393099] DMAR: ATSR flags: 0x1 > [ 1.394257] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0 > [ 1.395891] DMAR-IR: Queued invalidation will be enabled to support > x2apic and Intr-remapping. > [ 1.400856] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 3.719211] DMAR: No RMRR found > [ 3.729983] DMAR: dmar0: Using Queued invalidation > [ 3.731395] DMAR: Setting RMRR: > [ 3.732467] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 3.734099] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 > - 0xffffff] > [ 4.802391] DMAR: Intel(R) Virtualization Technology for Directed I/O > > Thanks.
