Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting for once, to make this easily accessible to everyone. Gwan-gyeong Mun, was this regression ever addressed? Doesn't look like it from here, but I might have missed something. Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr If I did something stupid, please tell me, as explained on that page. #regzbot poke On 07.07.23 16:16, Gwan-gyeong Mun wrote: > > > On 7/6/23 1:01 AM, Bjorn Helgaas wrote: >> On Mon, Jul 03, 2023 at 01:37:42PM +0300, Gwan-gyeong Mun wrote: >>> Since Linux 6.2 kernel (same happens in Linux 6.4.1), loading vfio-pci >>> driver to a specific HW (Intel DG2 A770) target does not work properly. >>> (It works fine on Linux 6.1 kernel with the same HW). >> >> Thank you very much for the report! >> >> Does this problem only happen with vfio-pci? d8d2b65a940b appeared in >> v6.2-rc1 (Dec 25, 2022), so I would think somebody would have used DG2 >> on a v6.2 or newer kernel. >> > Hi Bjorn, > > Yes, the problem only occurred when I set DG2 to vfio-pci as shown below > in the settings [1]. > (The reason for setting DG2 to vfio-pci is to use dg2 as a qemu pci > paththru device). > If you don't set DG2 to vfio-pci, you won't see any logs of the problem. > > >> Can you please collect the complete "sudo lspci -vv" output (not just >> the DG2 items)? We need info about the switch ports and all the >> capabilities, since d8d2b65a940b has to do with switch ports, AER, and >> MSI. >> >> Also, please collect the complete dmesg log with v6.4.1 (which does >> not work) and v6.4.1 with d8d2b65a940b reverted (which should work). >> > > I've filed this issue with kernel bugzilla[2] and added the dmesg and > lspci information you asked about as attachments. > I've also added direct links to the relevant logs below. > > 1. complete dmesg log with v6.4.1 with d8d2b65a940b reverted.[3] > 2. lspci -vv with v6.4.1 with d8d2b65a940b reverted [4] > 3. complete dmesg log with v6.4.1 [5] > 4. lspci -vv with v6.4.1 [6] > > [1] > $ cat /etc/modprobe.d/vfio.conf > > options vfio-pci ids=8086:56a0,8086:4f90 > softdep drm pre: vfio-pci > > [2] https://bugzilla.kernel.org/show_bug.cgi?id=217641 > [3] https://bugzilla.kernel.org/attachment.cgi?id=304560 > [4] https://bugzilla.kernel.org/attachment.cgi?id=304561 > [5] https://bugzilla.kernel.org/attachment.cgi?id=304562 > [6] https://bugzilla.kernel.org/attachment.cgi?id=304563 > > >> I know you said that on v6.4.1 with d8d2b65a940b reverted, the system >> boots but there's still a problem with suspend. I'm intentionally >> ignoring this problem for now. After we figure out the boot-time >> problem with the DG2 being left in D3cold, we can come back to the >> suspend problem. > Yes, I understand, and I agree. > > Br, > > G.G. >> >> Bjorn >> >>> The configuration and hardware information used is described in [1]. >>> >>> Starting with the Linux 6.2 kernel, the following log is output to dmesg >>> when a problem occurs. >>> ... >>> [ 15.049948] pcieport 0000:00:01.0: Data Link Layer Link Active not >>> set in >>> 1000 msec >>> [ 15.050024] pcieport 0000:01:00.0: Unable to change power state from >>> D3cold >>> to D0, device inaccessible >>> [ 15.051067] pcieport 0000:02:01.0: Unable to change power state from >>> D3cold >>> to D0, device inaccessible >>> [ 15.052141] pcieport 0000:02:04.0: Unable to change power state from >>> D3cold >>> to D0, device inaccessible >>> [ 17.286554] vfio-pci 0000:03:00.0: not ready 1023ms after resume; >>> giving up >>> [ 17.286553] vfio-pci 0000:04:00.0: not ready 1023ms after resume; >>> giving up >>> [ 17.286576] vfio-pci 0000:03:00.0: Unable to change power state from >>> D3cold >>> to D0, device inaccessible >>> [ 17.286578] vfio-pci 0000:04:00.0: Unable to change power state from >>> D3cold >>> to D0, device inaccessible >>> ... >>> >>> And if you check the DG2 hardware using the "lspci -nnv" command, you >>> will >>> see that "Flags:" is displayed as "!!! Unknown header type 7f" as shown >>> below. [2] >>> The normal output log looks like [3]. >>> >>> This issue has been occurring since the patch below was applied. [4] >>> >>> d8d2b65a940bb497749d66bdab59b530901d3854 is the first bad commit >>> commit d8d2b65a940bb497749d66bdab59b530901d3854 >>> Author: Bjorn Helgaas <bhelgaas@xxxxxxxxxx> >>> Date: Fri Dec 9 11:01:00 2022 -0600 >>> >>> PCI/portdrv: Allow AER service only for Root Ports & RCECs >>> >>> >>> Rolling back the [4] patch makes it work on boot with the latest >>> version of >>> the kernel, but the same problem still occurs after "suspend to s2idle". >>> This problem existed even before applying [4]. >>> >>> Suspend has been tested with the following command. >>> $ systemctl suspend -i >>> >>> $ cat /sys/power/mem_sleep >>> [s2idle] deep >>> >>> >>> Here is the log that is issued when testing suspend to s2idle. [5] >>> >>> >>> Br, >>> >>> G.G. >>> >>> >>> [1] Env: >>> >>> NUC: intel-nuc-13-extreme-kit-nuc13rngi7 >>> (https://ark.intel.com/content/www/us/en/ark/products/229784/intel-nuc-13-extreme-kit-nuc13rngi7.html) >>> (MB: Z690, CPU: RPL-S i13700k) >>> >>> PCIE Card: Intel A770 GPU >>> >>> Add boot parameter: intel_iommu=on iommu=pt >>> >>> $ lspci -nn |grep DG2 >>> 03:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc >>> A770] >>> [8086:56a0] (rev 08) >>> 04:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller >>> [8086:4f90] >>> >>> >>> $ cat /etc/modprobe.d/vfio.conf >>> >>> options vfio-pci ids=8086:56a0,8086:4f90 >>> softdep drm pre: vfio-pci >>> >>> [2] >>> 03:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc >>> A770] >>> [8086:56a0] (rev 08) (prog-if 00 [VGA controller]) >>> Subsystem: Intel Corporation Device [8086:1020] >>> !!! Unknown header type 7f >>> Memory at 93000000 (64-bit, non-prefetchable) [size=16M] >>> Memory at 6000000000 (64-bit, prefetchable) [size=16G] >>> Expansion ROM at 94000000 [disabled] [size=2M] >>> Kernel driver in use: vfio-pci >>> Kernel modules: i915 >>> >>> 04:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller >>> [8086:4f90] >>> Subsystem: Intel Corporation Device [8086:1020] >>> !!! Unknown header type 7f >>> Memory at 94300000 (64-bit, non-prefetchable) [size=16K] >>> Kernel driver in use: vfio-pci >>> Kernel modules: snd_hda_intel >>> >>> >>> [3] >>> 03:00.0 VGA compatible controller [0300]: Intel Corporation DG2 [Arc >>> A770] >>> [8086:56a0] (rev 08) (prog-if 00 [VGA controller]) >>> Subsystem: Intel Corporation Device [8086:1020] >>> Flags: bus master, fast devsel, latency 0, IOMMU group 19 >>> Memory at 93000000 (64-bit, non-prefetchable) [size=16M] >>> Memory at 6000000000 (64-bit, prefetchable) [size=16G] >>> Expansion ROM at 94000000 [disabled] [size=2M] >>> Capabilities: <access denied> >>> Kernel driver in use: vfio-pci >>> Kernel modules: i915 >>> >>> 04:00.0 Audio device [0403]: Intel Corporation DG2 Audio Controller >>> [8086:4f90] >>> Subsystem: Intel Corporation Device [8086:1020] >>> Flags: fast devsel, IOMMU group 20 >>> Memory at 94300000 (64-bit, non-prefetchable) [disabled] [size=16K] >>> Capabilities: <access denied> >>> Kernel driver in use: vfio-pci >>> Kernel modules: snd_hda_intel >>> >>> >>> [4] >>> commit d8d2b65a940bb497749d66bdab59b530901d3854 >>> Author: Bjorn Helgaas <bhelgaas@xxxxxxxxxx> >>> Date: Fri Dec 9 11:01:00 2022 -0600 >>> >>> PCI/portdrv: Allow AER service only for Root Ports & RCECs >>> >>> Previously portdrv allowed the AER service for any device with >>> an AER >>> capability (assuming Linux had control of AER) even though the AER >>> service >>> driver only attaches to Root Port and RCECs. >>> >>> Because get_port_device_capability() included AER for non-RP, >>> non-RCEC >>> devices, we tried to initialize the AER IRQ even though these >>> devices >>> don't generate AER interrupts. >>> >>> Intel DG1 and DG2 discrete graphics cards contain a switch >>> leading to a >>> GPU. The switch supports AER but not MSI, so initializing an >>> AER IRQ >>> failed, and portdrv failed to claim the switch port at all. The >>> GPU >>> itself >>> could be suspended, but the switch could not be put in a >>> low-power state >>> because it had no driver. >>> >>> Don't allow the AER service on non-Root Port, non-Root Complex >>> Event >>> Collector devices. This means we won't enable Bus Mastering if the >>> device >>> doesn't require MSI, the AER service will not appear in sysfs, >>> and the >>> AER >>> service driver will not bind to the device. >>> >>> Link: >>> https://lore.kernel.org/r/20221207084105.84947-1-mika.westerberg@xxxxxxxxxxxxxxx >>> Link: >>> https://lore.kernel.org/r/20221210002922.1749403-1-helgaas@xxxxxxxxxx >>> Based-on-patch-by: Mika Westerberg >>> <mika.westerberg@xxxxxxxxxxxxxxx> >>> Signed-off-by: Bjorn Helgaas <bhelgaas@xxxxxxxxxx> >>> Reviewed-by: Kuppuswamy Sathyanarayanan >>> <sathyanarayanan.kuppuswamy@xxxxxxxxxxxxxxx> >>> >>> diff --git a/drivers/pci/pcie/portdrv.c b/drivers/pci/pcie/portdrv.c >>> index a6c4225505d5..8b16e96ec15c 100644 >>> --- a/drivers/pci/pcie/portdrv.c >>> +++ b/drivers/pci/pcie/portdrv.c >>> @@ -232,7 +232,9 @@ static int get_port_device_capability(struct pci_dev >>> *dev) >>> } >>> >>> #ifdef CONFIG_PCIEAER >>> - if (dev->aer_cap && pci_aer_available() && >>> + if ((pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT || >>> + pci_pcie_type(dev) == PCI_EXP_TYPE_RC_EC) && >>> + dev->aer_cap && pci_aer_available() && >>> (pcie_ports_native || host->native_aer)) >>> services |= PCIE_PORT_SERVICE_AER; >>> #endif >>> >>> >>> [5] >>> [ 71.995824] PM: suspend entry (s2idle) >>> [ 72.000793] Filesystems sync: 0.004 seconds >>> [ 72.153926] Freezing user space processes >>> [ 72.156234] Freezing user space processes completed (elapsed 0.002 >>> seconds) >>> [ 72.156244] OOM killer disabled. >>> [ 72.156246] Freezing remaining freezable tasks >>> [ 72.157616] Freezing remaining freezable tasks completed (elapsed >>> 0.001 >>> seconds) >>> [ 72.157619] printk: Suspending console(s) (use no_console_suspend to >>> debug) >>> [ 73.756457] ACPI: EC: interrupt blocked >>> [ 75.103988] ucsi_acpi USBC000:00: ucsi_handle_connector_change: >>> GET_CONNECTOR_STATUS failed (-5) >>> [ 84.052478] ACPI: EC: interrupt unblocked >>> [ 86.085388] pcieport 0000:00:01.0: Data Link Layer Link Active not >>> set in >>> 1000 msec >>> [ 86.085523] pcieport 0000:01:00.0: Unable to change power state from >>> D3cold to D0, device inaccessible >>> [ 86.086984] pci 0000:02:01.0: Unable to change power state from >>> D3cold to >>> D0, device inaccessible >>> [ 86.087005] pci 0000:02:04.0: Unable to change power state from >>> D3cold to >>> D0, device inaccessible >>> [ 88.335403] vfio-pci 0000:04:00.0: not ready 1023ms after resume; >>> waiting >>> [ 88.335427] vfio-pci 0000:03:00.0: not ready 1023ms after resume; >>> waiting >>> [ 89.375444] vfio-pci 0000:04:00.0: not ready 2047ms after resume; >>> waiting >>> [ 89.375471] vfio-pci 0000:03:00.0: not ready 2047ms after resume; >>> waiting >>> [ 91.615418] vfio-pci 0000:04:00.0: not ready 4095ms after resume; >>> waiting >>> [ 91.615439] vfio-pci 0000:03:00.0: not ready 4095ms after resume; >>> waiting >>> [ 95.882059] vfio-pci 0000:04:00.0: not ready 8191ms after resume; >>> waiting >>> [ 95.882081] vfio-pci 0000:03:00.0: not ready 8191ms after resume; >>> waiting >>> [ 104.202062] vfio-pci 0000:04:00.0: not ready 16383ms after resume; >>> waiting >>> [ 104.202066] vfio-pci 0000:03:00.0: not ready 16383ms after resume; >>> waiting >>> [ 121.482058] vfio-pci 0000:04:00.0: not ready 32767ms after resume; >>> waiting >>> [ 121.482067] vfio-pci 0000:03:00.0: not ready 32767ms after resume; >>> waiting >>> [ 155.615409] vfio-pci 0000:04:00.0: not ready 65535ms after resume; >>> giving >>> up >>> [ 155.615412] vfio-pci 0000:03:00.0: not ready 65535ms after resume; >>> giving >>> up >>> [ 155.633757] i915 0000:00:02.0: [drm] GT0: GuC firmware >>> i915/tgl_guc_70.bin version 70.5.1 >>> [ 155.633761] i915 0000:00:02.0: [drm] GT0: HuC firmware >>> i915/tgl_huc.bin >>> version 7.9.3 >>> [ 155.636177] i915 0000:00:02.0: [drm] GT0: HuC: authenticated! >>> [ 155.636860] i915 0000:00:02.0: [drm] GT0: GUC: submission enabled >>> [ 155.636860] i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled >>> [ 155.637228] i915 0000:00:02.0: [drm] GT0: GUC: RC enabled >>> [ 155.661583] nvme nvme0: Shutdown timeout set to 10 seconds >>> [ 155.663188] nvme nvme0: 24/0/0 default/read/poll queues >>> [ 155.674267] iwlwifi 0000:00:14.3: WRT: Invalid buffer destination >>> [ 155.823379] ucsi_acpi USBC000:00: possible UCSI driver bug 1 >>> [ 155.823390] ucsi_acpi USBC000:00: failed to re-enable >>> notifications (-22) >>> [ 155.833326] iwlwifi 0000:00:14.3: WFPM_UMAC_PD_NOTIFICATION: 0x1f >>> [ 155.833358] iwlwifi 0000:00:14.3: WFPM_LMAC2_PD_NOTIFICATION: 0x0 >>> [ 155.833367] iwlwifi 0000:00:14.3: WFPM_AUTH_KEY_0: 0x90 >>> [ 155.833377] iwlwifi 0000:00:14.3: CNVI_SCU_SEQ_DATA_DW9: 0x960 >>> [ 155.942363] ata6: SATA link down (SStatus 4 SControl 300) >>> [ 155.942537] ata5: SATA link down (SStatus 4 SControl 300) >>> [ 156.030241] mei_hdcp >>> 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: >>> bound 0000:00:02.0 (ops i915_hdcp_ops [i915]) >>> [ 156.030830] OOM killer enabled. >>> [ 156.030831] Restarting tasks ... >>> [ 156.030894] mei_pxp >>> 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: >>> bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915]) >>> [ 156.031827] done. >>> [ 156.031837] random: crng reseeded on system resumption >>> [ 156.036058] PM: suspend exit >>> [ 158.962881] wlp0s20f3: authenticate with 4c:ed:fb:a0:7f:6c >>> [ 158.966647] wlp0s20f3: send auth to 4c:ed:fb:a0:7f:6c (try 1/3) >>> [ 159.001337] wlp0s20f3: authenticated >>> [ 159.001858] wlp0s20f3: associate with 4c:ed:fb:a0:7f:6c (try 1/3) >>> [ 159.002882] wlp0s20f3: RX AssocResp from 4c:ed:fb:a0:7f:6c >>> (capab=0x11 >>> status=0 aid=1) >>> [ 159.010807] wlp0s20f3: associated >>> [ 159.159528] IPv6: ADDRCONF(NETDEV_CHANGE): wlp0s20f3: link becomes >>> ready >>> [ 287.875205] vfio-pci 0000:04:00.0: Unable to change power state from >>> D3cold to D0, device inaccessible >>> [ 287.936500] vfio-pci 0000:04:00.0: Unable to change power state from >>> D3cold to D0, device inaccessible >>> [ 289.414087] vfio-pci 0000:03:00.0: Unable to change power state from >>> D3cold to D0, device inaccessible >>> [ 289.475297] vfio-pci 0000:03:00.0: Unable to change power state from >>> D3cold to D0, device inaccessible
