[AMD Official Use Only] Hi James, Could you provide the pp_dpm_* values in sysfs with and without the patch? Also, could you try forcing PCIE to gen3 (through pp_dpm_pcie) if it's not in gen3 when the issue happens? For details on pp_dpm_*, please check https://dri.freedesktop.org/docs/drm/gpu/amdgpu.html Thanks, Lijo -----Original Message----- From: James Turner <linuxkernel.foss@xxxxxxxxxxxxxxxxxxxxxx> Sent: Saturday, January 22, 2022 6:21 AM To: Alex Deucher <alexdeucher@xxxxxxxxx> Cc: Thorsten Leemhuis <regressions@xxxxxxxxxxxxx>; Deucher, Alexander <Alexander.Deucher@xxxxxxx>; Lazar, Lijo <Lijo.Lazar@xxxxxxx>; regressions@xxxxxxxxxxxxxxx; kvm@xxxxxxxxxxxxxxx; Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx>; Pan, Xinhui <Xinhui.Pan@xxxxxxx>; LKML <linux-kernel@xxxxxxxxxxxxxxx>; amd-gfx@xxxxxxxxxxxxxxxxxxxxx; Alex Williamson <alex.williamson@xxxxxxxxxx>; Koenig, Christian <Christian.Koenig@xxxxxxx> Subject: Re: [REGRESSION] Too-low frequency limit for AMD GPU PCI-passed-through to Windows VM > Are you ever loading the amdgpu driver in your tests? Yes, although I'm binding the `vfio-pci` driver to the AMD GPU's PCI devices via the kernel command line. (See my initial email.) My understanding is that `vfio-pci` is supposed to keep other drivers, such as `amdgpu`, from interacting with the GPU, although that's clearly not what's happening. I've been testing with `amdgpu` included in the `MODULES` list in `/etc/mkinitcpio.conf` (which Arch Linux uses to generate the initramfs). However, I ran some more tests today (results below), this time without `i915` or `amdgpu` in the `MODULES` list. The `amdgpu` kernel module still gets loaded. (I think udev loads it automatically?) Your comment gave me the idea to blacklist the `amdgpu` kernel module. That does serve as a workaround on my machine – it fixes the behavior for f9b7f3703ff9 ("drm/amdgpu/acpi: make ATPX/ATCS structures global (v2)") and for the current Arch Linux prebuilt kernel (5.16.2-arch1-1). That's an acceptable workaround for my machine only because the separate GPU used by the host is an Intel integrated GPU. That workaround wouldn't work well for someone with two AMD GPUs. # New test results The following tests are set up the same way as in my initial email, with the following exceptions: - I've updated libvirt to 1:8.0.0-1. - I've removed `i915` and `amdgpu` from the `MODULES` list in `/etc/mkinitcpio.conf`. For all three of these tests, `lspci` said the following: % lspci -nnk -d 1002:6981 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 3200] [1002:6981] Subsystem: Dell Device [1028:0926] Kernel driver in use: vfio-pci Kernel modules: amdgpu % lspci -nnk -d 1002:aae0 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0] Subsystem: Dell Device [1028:0926] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel ## Version f1688bd69ec4 ("drm/amd/amdgpu:save psp ring wptr to avoid attack") This is the commit immediately preceding the one which introduced the issue. % sudo dmesg | grep -i amdgpu [ 15.840160] [drm] amdgpu kernel modesetting enabled. [ 15.840884] amdgpu: CRAT table not found [ 15.840885] amdgpu: Virtual CRAT table created for CPU [ 15.840893] amdgpu: Topology: Add CPU node % lsmod | grep amdgpu amdgpu 7450624 0 gpu_sched 49152 1 amdgpu drm_ttm_helper 16384 1 amdgpu ttm 77824 2 amdgpu,drm_ttm_helper i2c_algo_bit 16384 2 amdgpu,i915 drm_kms_helper 303104 2 amdgpu,i915 drm 581632 11 gpu_sched,drm_kms_helper,amdgpu,drm_ttm_helper,i915,ttm The passed-through GPU worked properly in the VM. ## Version f9b7f3703ff9 ("drm/amdgpu/acpi: make ATPX/ATCS structures global (v2)") This is the commit which introduced the issue. % sudo dmesg | grep -i amdgpu [ 15.319023] [drm] amdgpu kernel modesetting enabled. [ 15.329468] amdgpu: CRAT table not found [ 15.329470] amdgpu: Virtual CRAT table created for CPU [ 15.329482] amdgpu: Topology: Add CPU node % lsmod | grep amdgpu amdgpu 7450624 0 gpu_sched 49152 1 amdgpu drm_ttm_helper 16384 1 amdgpu ttm 77824 2 amdgpu,drm_ttm_helper i2c_algo_bit 16384 2 amdgpu,i915 drm_kms_helper 303104 2 amdgpu,i915 drm 581632 11 gpu_sched,drm_kms_helper,amdgpu,drm_ttm_helper,i915,ttm The passed-through GPU did not run above 501 MHz in the VM. ## Blacklisted `amdgpu`, version f9b7f3703ff9 ("drm/amdgpu/acpi: make ATPX/ATCS structures global (v2)") For this test, I added `module_blacklist=amdgpu` to kernel command line to blacklist the `amdgpu` module. % sudo dmesg | grep -i amdgpu [ 14.591576] Module amdgpu is blacklisted % lsmod | grep amdgpu The passed-through GPU worked properly in the VM. James
