> -----Original Message----- > From: kvm-owner@xxxxxxxxxxxxxxx [mailto:kvm-owner@xxxxxxxxxxxxxxx] On Behalf > Of Ravi Kerur > Sent: Saturday, February 10, 2018 2:32 AM > To: kvm@xxxxxxxxxxxxxxx > Subject: DMAR errors with VT-d and vfio/vfio-pci on IXGBE vf interface assigned to > guest > > Hi, > > I am running into an issue when DPDK is started with intel_iommu=on via GRUB > command. Problem is not seen with regular kernel driver, error messages show > when DPDK is started and happens for both PF and VF interfaces. Discussing with > DPDK folks it is not a DPDK issue hence sending an email to this list. > > Workaround is to use "iommu=pt" but I want iommu enabled in my setup. iommu=pt is actually 1:1 mapping mimicked by iommu driver, such 1:1 mapping is programmed to host through VFIO. Whereas, if iommu=pt is not set, then each mapping should be dynamically programmed to host through vfio. At this moment, I intend to suspect if DPDK missed to request iova mapping from guest iommu driver or somehow unmapped the iova mapping while further operation still use the stale iova. May be helpful to check the DPDK's behavior here. > My understanding is that 'Address width' is reported as '46' on host and '39' on guest > and it could be causing the problem but not certain so kindly let me know how to > resolve this issue. > > I tried influencing guest address width with 'host-phys-bits=true and phys-bits=46' > properties when instantiating a guest but it still ends with address width 39. So far, AW=39 is a default setting for vIOMMU emulation. it can be configured with 'x-aw-bits' option. As long as AW is configured smaller than the AW reported on host, it is should be fine. If it is AW related issue, it may report another DMAR fault which would be " Invalid programming of a context-entry". Thanks, Yi L > I have tried 'x-aw-bits=48', still see DMAR errors. > > Following are the details > > (1) Linux kernel 4.9 (host and guest), Qemu 2.11 > > (2) DPDK 17.05 > > (3) IXGBE details > ethtool -i enp4s0f0 (PF driver) > driver: ixgbe > version: 5.3.3 > firmware-version: 0x800007b8, 1.1018.0 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > ethtool -i enp4s16f2 (VF driver) > driver: ixgbevf > version: 4.3.2 > firmware-version: > bus-info: 0000:04:10.2 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: no > supports-register-dump: yes > supports-priv-flags: no > > Bus info Device Class Description > ========================================================= > pci@0000:01:00.0 ens11f0 network 82599ES 10-Gigabit > SFI/SFP+ Network Connection > pci@0000:01:00.1 ens11f1 network 82599ES 10-Gigabit > SFI/SFP+ Network Connection > pci@0000:04:00.0 enp4s0f0 network 82599ES 10-Gigabit > SFI/SFP+ Network Connection > pci@0000:04:00.1 enp4s0f1 network 82599ES 10-Gigabit > SFI/SFP+ Network Connection > pci@0000:04:10.0 enp4s16 network Illegal Vendor ID > pci@0000:04:10.2 enp4s16f2 network Illegal Vendor ID > > (5) Kernel dmesg on host > > dmesg | grep -e DMAR -e IOMMU > [ 0.000000] ACPI: DMAR 0x000000007999BAD0 0000E0 (v01 ALASKA A M I > 00000001 INTL 20091013) > [ 0.000000] DMAR: IOMMU enabled > [ 0.519368] DMAR: Host address width 46 > [ 0.527243] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > [ 0.538073] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap > d2078c106f0466 ecap f020df > [ 0.554253] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 > [ 0.565074] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap > d2078c106f0466 ecap f020df > [ 0.581259] DMAR: RMRR base: 0x0000007bbc6000 end: 0x0000007bbd4fff > [ 0.593980] DMAR: ATSR flags: 0x0 > [ 0.600802] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0 > [ 0.613532] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1 > [ 0.626265] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 IOMMU 0 > [ 0.639177] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 IOMMU 1 > [ 0.652089] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 IOMMU 1 > [ 0.664996] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 > [ 0.675984] DMAR-IR: Queued invalidation will be enabled to support > x2apic and Intr-remapping. > [ 0.694475] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 9.637093] DMAR: dmar1: Using Queued invalidation > [ 9.646945] DMAR: Setting RMRR: > [ 9.653942] DMAR: Setting identity map for device 0000:00:1d.0 > [0x7bbc6000 - 0x7bbd4fff] > [ 9.670513] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 9.681656] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 > - 0xffffff] > [ 9.696630] DMAR: Intel(R) Virtualization Technology for Directed I/O > [ 2605.450811] DMAR: DRHD: handling fault status reg 2 [ 2605.450814] DMAR: > [DMA Read] Request device [04:10.0] fault addr > 33a128000 [fault reason 06] PTE Read access is not set [ 2607.450907] DMAR: DRHD: > handling fault status reg 102 [ 2607.450910] DMAR: [DMA Read] Request device > [04:10.0] fault addr > 33a128000 [fault reason 06] PTE Read access is not set [ 4539.597735] DMAR: DRHD: > handling fault status reg 202 [ 4539.597737] DMAR: [DMA Read] Request device > [04:10.0] fault addr > 33a128000 [fault reason 06] PTE Read access is not set > > > (6) dmesg on guest > # dmesg | grep -e DMAR -e IOMMU > [ 0.000000] ACPI: DMAR 0x000000007FFE201D 000050 (v01 BOCHS > BXPCDMAR 00000001 BXPC 00000001) > [ 0.000000] DMAR: IOMMU enabled > [ 1.387988] DMAR: Host address width 39 > [ 1.389203] DMAR: DRHD base: 0x000000fed90000 flags: 0x1 > [ 1.390692] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap > 12008c22260286 ecap f00f5e > [ 1.393099] DMAR: ATSR flags: 0x1 > [ 1.394257] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0 > [ 1.395891] DMAR-IR: Queued invalidation will be enabled to support > x2apic and Intr-remapping. > [ 1.400856] DMAR-IR: Enabled IRQ remapping in x2apic mode > [ 3.719211] DMAR: No RMRR found > [ 3.729983] DMAR: dmar0: Using Queued invalidation > [ 3.731395] DMAR: Setting RMRR: > [ 3.732467] DMAR: Prepare 0-16MiB unity mapping for LPC > [ 3.734099] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 > - 0xffffff] > [ 4.802391] DMAR: Intel(R) Virtualization Technology for Directed I/O > > Thanks.
