On 1/4/2023 12:24 PM, Baolu Lu wrote: > On 2023/1/4 2:55, Matt Fagnani wrote: >> I reproduced the problem with 6.2-rc1 in a Fedora 37 installation with early >> kdump enabled as described >> athttps://fedoraproject.org/wiki/How_to_use_kdump_to_debug_kernel_crashes ; >> https://github.com/k-hagio/fedora-kexec-tools/blob/master/early-kdump-howto.txt ; >> I panicked the kernel with sysrq+alt+c. The dmesg saved with the kdump showed >> warnings at drivers/pci/ats.c:251 pci_disable_pri+0x75/0x80 and at >> drivers/pci/ats.c:419 pci_disable_pasid+0x45/0x50 involving AMD IOMMU and >> amdgpu functions in the trace. Since those warnings' were >> if (WARN_ON(!pdev->pri_enabled)) and if (WARN_ON(!pdev->pasid_enabled)), >> pci_disable_pri and pci_disable_pasid looked like they were called when >> pdev->pri_enabled and pdev->pasid_enabled were both false. >> A null pointer dereference occurred right after that which made amdgpu crash. >> >> [ 13.132368] [drm] amdgpu kernel modesetting enabled. >> [ 13.133766] amdgpu: Topology: Add APU node [0x0:0x0] >> [ 13.137596] Console: switching to colour dummy device 80x25 >> [ 13.143717] amdgpu 0000:00:01.0: vgaarb: deactivate vga console >> [ 13.143970] [drm] initializing kernel modesetting (CARRIZO 0x1002:0x9874 >> 0x103C:0x8332 0xCA). >> [ 13.144205] [drm] register mmio base: 0xF0400000 >> [ 13.144209] [drm] register mmio size: 262144 >> [ 13.144310] [drm] add ip block number 0 <vi_common> >> [ 13.144316] [drm] add ip block number 1 <gmc_v8_0> >> [ 13.144320] [drm] add ip block number 2 <cz_ih> >> [ 13.144324] [drm] add ip block number 3 <gfx_v8_0> >> [ 13.144328] [drm] add ip block number 4 <sdma_v3_0> >> [ 13.144332] [drm] add ip block number 5 <powerplay> >> [ 13.144336] [drm] add ip block number 6 <dm> >> [ 13.144340] [drm] add ip block number 7 <uvd_v6_0> >> [ 13.144343] [drm] add ip block number 8 <vce_v3_0> >> [ 13.144347] [drm] add ip block number 9 <acp_ip> >> [ 13.144388] amdgpu 0000:00:01.0: amdgpu: Fetched VBIOS from VFCT >> [ 13.144397] amdgpu: ATOM BIOS: 113-C75100-031 >> [ 13.144425] [drm] UVD is enabled in physical mode >> [ 13.144431] [drm] VCE enabled in physical mode >> [ 13.144435] amdgpu 0000:00:01.0: amdgpu: Trusted Memory Zone (TMZ) feature >> not supported >> [ 13.144491] [drm] vm size is 64 GB, 2 levels, block size is 10-bit, >> fragment size is 9-bit >> [ 13.144503] amdgpu 0000:00:01.0: amdgpu: VRAM: 512M 0x000000F400000000 - >> 0x000000F41FFFFFFF (512M used) >> [ 13.144511] amdgpu 0000:00:01.0: amdgpu: GART: 1024M 0x000000FF00000000 - >> 0x000000FF3FFFFFFF >> [ 13.144524] [drm] Detected VRAM RAM=512M, BAR=512M >> [ 13.144529] [drm] RAM width 64bits UNKNOWN >> [ 13.144623] [drm] amdgpu: 512M of VRAM memory ready >> [ 13.144630] [drm] amdgpu: 3572M of GTT memory ready. >> [ 13.144653] [drm] GART: num cpu pages 262144, num gpu pages 262144 >> [ 13.144705] [drm] PCIE GART of 1024M enabled (table at 0x000000F400600000). >> [ 13.158820] amdgpu: hwmgr_sw_init smu backed is smu8_smu >> [ 13.175036] [drm] Found UVD firmware Version: 1.91 Family ID: 11 >> [ 13.175097] [drm] UVD ENC is disabled >> [ 13.186675] [drm] Found VCE firmware Version: 52.4 Binary ID: 3 >> [ 13.187879] amdgpu: smu version 27.18.00 >> [ 13.193760] [drm] DM_PPLIB: values for Engine clock >> [ 13.193773] [drm] DM_PPLIB: 300000 >> [ 13.193776] [drm] DM_PPLIB: 480000 >> [ 13.193779] [drm] DM_PPLIB: 533340 >> [ 13.193781] [drm] DM_PPLIB: 576000 >> [ 13.193784] [drm] DM_PPLIB: 626090 >> [ 13.193786] [drm] DM_PPLIB: 685720 >> [ 13.193788] [drm] DM_PPLIB: 720000 >> [ 13.193791] [drm] DM_PPLIB: 757900 >> [ 13.193793] [drm] DM_PPLIB: Validation clocks: >> [ 13.193796] [drm] DM_PPLIB: engine_max_clock: 75790 >> [ 13.193799] [drm] DM_PPLIB: memory_max_clock: 93300 >> [ 13.193802] [drm] DM_PPLIB: level : 8 >> [ 13.193806] [drm] DM_PPLIB: values for Display clock >> [ 13.193809] [drm] DM_PPLIB: 300000 >> [ 13.193811] [drm] DM_PPLIB: 400000 >> [ 13.193814] [drm] DM_PPLIB: 496560 >> [ 13.193816] [drm] DM_PPLIB: 626090 >> [ 13.193819] [drm] DM_PPLIB: 685720 >> [ 13.193821] [drm] DM_PPLIB: 757900 >> [ 13.193823] [drm] DM_PPLIB: 800000 >> [ 13.193825] [drm] DM_PPLIB: 847060 >> [ 13.193828] [drm] DM_PPLIB: Validation clocks: >> [ 13.193830] [drm] DM_PPLIB: engine_max_clock: 75790 >> [ 13.193833] [drm] DM_PPLIB: memory_max_clock: 93300 >> [ 13.193836] [drm] DM_PPLIB: level : 8 >> [ 13.193839] [drm] DM_PPLIB: values for Memory clock >> [ 13.193842] [drm] DM_PPLIB: 667000 >> [ 13.193844] [drm] DM_PPLIB: 933000 >> [ 13.193847] [drm] DM_PPLIB: Validation clocks: >> [ 13.193849] [drm] DM_PPLIB: engine_max_clock: 75790 >> [ 13.193852] [drm] DM_PPLIB: memory_max_clock: 93300 >> [ 13.193854] [drm] DM_PPLIB: level : 8 >> [ 13.193973] [drm] Display Core initialized with v3.2.215! >> [ 13.309967] [drm] UVD initialized successfully. >> [ 13.511031] [drm] VCE initialized successfully. >> [ 13.515217] kfd kfd: amdgpu: Allocated 3969056 bytes on gart >> [ 13.515442] amdgpu: sdma_bitmap: f >> [ 13.515549] ------------[ cut here ]------------ >> [ 13.515555] WARNING: CPU: 0 PID: 477 at drivers/pci/ats.c:251 >> pci_disable_pri+0x75/0x80 >> [ 13.515571] Modules linked in: amdgpu(+) drm_ttm_helper ttm iommu_v2 >> hid_logitech_hidpp crct10dif_pclmul drm_buddy crc32_pclmul gpu_sched >> crc32c_intel polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 >> drm_display_helper wdat_wdt serio_raw hid_multitouch sp5100_tco >> hid_logitech_dj r8169 cec video wmi scsi_dh_rdac scsi_dh_emc scsi_dh_alua fuse >> dm_multipath >> [ 13.515620] CPU: 0 PID: 477 Comm: systemd-udevd Kdump: loaded Not tainted >> 6.2.0-0.rc1.14.fc38.x86_64 #1 >> [ 13.515628] Hardware name: HP HP Laptop 15-bw0xx/8332, BIOS F.52 12/03/2019 >> [ 13.515634] RIP: 0010:pci_disable_pri+0x75/0x80 >> [ 13.515642] Code: 54 24 06 89 ee 48 89 df 83 e2 fe 66 89 54 24 06 0f b7 d2 >> e8 1d e1 fc ff 80 a3 4b 08 00 00 fd 48 83 c4 08 5b 5d e9 2b 8b 69 00 <0f> 0b >> eb b6 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 >> [ 13.515651] RSP: 0018:ffffbaf4407ab8e8 EFLAGS: 00010046 >> [ 13.515658] RAX: 0000000000000000 RBX: ffff90aa00ac4000 RCX: 0000000000000009 >> [ 13.515663] RDX: 0000000000000000 RSI: 0000000000000014 RDI: ffff90aa00ac4000 >> [ 13.515668] RBP: ffff90aa0e0c3810 R08: 0000000000000002 R09: 0000000000000000 >> [ 13.515673] R10: 0000000000000000 R11: ffffffffade4e430 R12: ffff90aa011a8800 >> [ 13.515678] R13: ffff90aa0e0c3800 R14: ffff90aa011a8800 R15: ffff90aa0e0c3960 >> [ 13.515683] FS: 00007fabd67feb40(0000) GS:ffff90aaf7400000(0000) >> knlGS:0000000000000000 >> [ 13.515689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [ 13.515695] CR2: 00007f5689ff54c0 CR3: 0000000100f16000 CR4: 00000000001506f0 >> [ 13.515700] Call Trace: >> [ 13.515704] <TASK> >> [ 13.515710] amd_iommu_attach_device+0x2e0/0x300 >> [ 13.515719] __iommu_attach_device+0x1b/0x90 >> [ 13.515727] iommu_attach_group+0x65/0xa0 >> [ 13.515735] amd_iommu_init_device+0x16b/0x250 [iommu_v2] >> [ 13.515747] kfd_iommu_resume+0x4c/0x1a0 [amdgpu] >> [ 13.517094] kgd2kfd_resume_iommu+0x12/0x30 [amdgpu] >> [ 13.518419] kgd2kfd_device_init.cold+0x346/0x49a [amdgpu] >> [ 13.519699] amdgpu_amdkfd_device_init+0x142/0x1d0 [amdgpu] >> [ 13.520877] amdgpu_device_init.cold+0x19f5/0x1e21 [amdgpu] >> [ 13.522118] ? _raw_spin_lock_irqsave+0x23/0x50 >> [ 13.522126] amdgpu_driver_load_kms+0x15/0x110 [amdgpu] >> [ 13.523386] amdgpu_pci_probe+0x161/0x370 [amdgpu] >> [ 13.524516] local_pci_probe+0x41/0x80 >> [ 13.524525] pci_device_probe+0xb3/0x220 >> [ 13.524533] really_probe+0xde/0x380 >> [ 13.524540] ? pm_runtime_barrier+0x50/0x90 >> [ 13.524546] __driver_probe_device+0x78/0x170 >> [ 13.524555] driver_probe_device+0x1f/0x90 >> [ 13.524560] __driver_attach+0xce/0x1c0 >> [ 13.524565] ? __pfx___driver_attach+0x10/0x10 >> [ 13.524570] bus_for_each_dev+0x73/0xa0 >> [ 13.524575] bus_add_driver+0x1ae/0x200 >> [ 13.524580] driver_register+0x89/0xe0 >> [ 13.524586] ? __pfx_init_module+0x10/0x10 [amdgpu] >> [ 13.525819] do_one_initcall+0x59/0x230 >> [ 13.525828] do_init_module+0x4a/0x200 >> [ 13.525834] __do_sys_init_module+0x157/0x180 >> [ 13.525839] do_syscall_64+0x5b/0x80 >> [ 13.525845] ? handle_mm_fault+0xff/0x2f0 >> [ 13.525850] ? do_user_addr_fault+0x1ef/0x690 >> [ 13.525856] ? exc_page_fault+0x70/0x170 >> [ 13.525860] entry_SYSCALL_64_after_hwframe+0x72/0xdc >> [ 13.525867] RIP: 0033:0x7fabd66cde4e >> [ 13.525872] Code: 48 8b 0d e5 5f 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e >> 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 af 00 00 00 0f 05 <48> 3d >> 01 f0 ff ff 73 01 c3 48 8b 0d b2 5f 0c 00 f7 d8 64 89 01 48 >> [ 13.525878] RSP: 002b:00007ffdd89bc6a8 EFLAGS: 00000246 ORIG_RAX: >> 00000000000000af >> [ 13.525884] RAX: ffffffffffffffda RBX: 0000563e4d23f0a0 RCX: 00007fabd66cde4e >> [ 13.525887] RDX: 00007fabd6817453 RSI: 000000000174fb66 RDI: 00007fabd3bd4010 >> [ 13.525890] RBP: 00007fabd6817453 R08: 0000563e4d237c70 R09: 00007fabd672f900 >> [ 13.525893] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000020000 >> [ 13.525896] R13: 0000563e4d239060 R14: 0000000000000000 R15: 0000563e4d23e450 >> [ 13.525900] </TASK> >> [ 13.525902] ---[ end trace 0000000000000000 ]--- >> [ 13.525964] ------------[ cut here ]------------ > > This (including the following) kernel traces are triggered by the > following code. > > 1698 static int pdev_pri_ats_enable(struct pci_dev *pdev) > 1699 { > 1700 int ret; > 1701 > 1702 /* Only allow access to user-accessible pages */ > 1703 ret = pci_enable_pasid(pdev, 0); > 1704 if (ret) > 1705 goto out_err; > > [--cut for short--] > > 1724 out_err: > 1725 pci_disable_pri(pdev); > 1726 pci_disable_pasid(pdev); > 1727 > 1728 return ret; > 1729 } > > pci_disable_pri() and pci_disable_pasid() are called with PCI PASID and > PRI not enabled. There are WARN_ON()s in the pci code for such cases. Yeah. Error path needs to be fixed. > > This happens in the domain attach device path. I haven't figured out why > the failure of PASID or PRI enabling will cause the domain attach device > to fail. And also why pci_pasid_features() and pci_pri_supported() are > not called before pci_enable_pasid/pri(). PASID/PRI support is verified in amd_iommu_device_info(). For AMD GPUs (PASID/PRI supported devices) - We allocate new domain called V2 domain and then attach device(s). amd_iommu_init_device() - > iommu_attach_group() In attach devices path : amd_iommu_attach_device() -> attach_device() If domain is v2 domain and device is PASID/PRI capable, then we try to enable PASID/PRI. This is where we are hitting WARN_ON. I think if attach device fails then we should put the device/group back to default domain so that we don't hit these warnings. Matt, Can you please test below patch? (its not a fix to original issue, but to avoid kernel warnings/traces). > > commit 201007ef707a ("PCI: Enable PASID only when ACS RR & UF enabled on > upstream path") requires ACS P2P Request Redirect and Upstream > Forwarding are enabled for the path leading to the device when enabling > PASID because PCIe fabric routes Memory Requests based on the TLP > address, ignoring any PASID. I guess this is the reason why > pci_enable_pasid() returns failure and discovers above buggy code. Can we get the lspci -vvv output. It will tell whether ACS request support. -Vasant ----- diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c index cbeaab55c0db..f81ab787eee1 100644 --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -1702,27 +1702,26 @@ static int pdev_pri_ats_enable(struct pci_dev *pdev) /* Only allow access to user-accessible pages */ ret = pci_enable_pasid(pdev, 0); if (ret) - goto out_err; + return ret; /* First reset the PRI state of the device */ ret = pci_reset_pri(pdev); if (ret) - goto out_err; + goto out_pasid; /* Enable PRI */ /* FIXME: Hardcode number of outstanding requests for now */ ret = pci_enable_pri(pdev, 32); if (ret) - goto out_err; + goto out_pasid; ret = pci_enable_ats(pdev, PAGE_SHIFT); - if (ret) - goto out_err; - - return 0; + if (!ret) + return 0; -out_err: pci_disable_pri(pdev); + +out_pasid: pci_disable_pasid(pdev); return ret; diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c index 864e4ffb6aa9..4228e44b4950 100644 --- a/drivers/iommu/amd/iommu_v2.c +++ b/drivers/iommu/amd/iommu_v2.c @@ -815,6 +815,7 @@ int amd_iommu_init_device(struct pci_dev *pdev, int pasids) return 0; out_drop_group: + iommu_detach_group(dev_state->domain, group); iommu_group_put(group); out_free_domain:
