2025年1月3日 14:19，Chen, Xiaogang <xiaogang.chen@xxxxxxx> 写道：
On 1/2/2025 11:55 PM, Gerry Liu wrote:
2025年1月3日 13:44，Chen, Xiaogang <xiaogang.chen@xxxxxxx> 写道：
On 1/2/2025 8:22 PM, Gerry Liu wrote:
2025年1月3日 07:08，Chen, Xiaogang <xiaogang.chen@xxxxxxx> 写道：
On 1/1/2025 11:36 PM, Jiang Liu wrote:
On error recover path during device probe, it may trigger invalid memory access as below: 024-12-25 12:00:53 [ 2703.773040] general protection fault, probably for non-canonical address 0x52445f4749464e4f: 0000 [#1] SMP NOPTI 2024-12-25 12:00:53 [ 2703.785199] CPU: 157 PID: 151951 Comm: rmmod Kdump: loaded Tainted: G W OE 5.10.134-17.2.al8.x86_64 #1 2024-12-25 12:00:53 [ 2703.797515] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 3.0.ES.AL.P.087.05 04/07/2024 2024-12-25 12:00:53 [ 2703.809188] RIP: 0010:kgd2kfd_device_exit+0x6/0x60 [amdgpu] 2024-12-25 12:00:53 [ 2703.816136] Code: ff 48 c7 83 38 01 00 00 80 45 e4 c0 c7 83 40 01 00 00 08 0f 00 00 e9 cd fa ff ff 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 <80> bf 28 01 00 00 00 48 89 fd 75 09 48 89 ef 5d e9 65 df 9d f4 8b 2024-12-25 12:00:54 [ 2703.838622] RSP: 0018:ffffb5313df07e10 EFLAGS: 00010202 2024-12-25 12:00:54 [ 2703.845216] RAX: 0000000000000000 RBX: ffff97ad689a3ff0 RCX: 0000000080400014 2024-12-25 12:00:54 [ 2703.853935] RDX: 0000000080400015 RSI: ffff97ad627e93d8 RDI: 52445f4749464e4f 2024-12-25 12:00:54 [ 2703.862652] RBP: ffff97ad689a3ff0 R08: 0000000000000000 R09: ffffffffb5814c00 2024-12-25 12:00:54 [ 2703.871368] R10: ffff97ad627e9280 R11: 0000000000000001 R12: ffffb5313df07e98 2024-12-25 12:00:54 [ 2703.880068] R13: ffff97ad689a1810 R14: 0000000000000001 R15: 0000000000000000 2024-12-25 12:00:54 [ 2703.888768] FS: 00007fa4db81e740(0000) GS:ffff98a93ec80000(0000) knlGS:0000000000000000 2024-12-25 12:00:54 [ 2703.898547] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2024-12-25 12:00:54 [ 2703.905684] CR2: 00007f4502deca00 CR3: 000001008fc50001 CR4: 0000000002770ee0 2024-12-25 12:00:54 [ 2703.914382] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2024-12-25 12:00:54 [ 2703.923066] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400 2024-12-25 12:00:54 [ 2703.931746] PKRU: 55555554 2024-12-25 12:00:54 [ 2703.935444] Call Trace: 2024-12-25 12:00:54 [ 2703.938962] amdgpu_amdkfd_device_fini_sw+0x1a/0x40 [amdgpu] 2024-12-25 12:00:54 [ 2703.946080] amdgpu_device_ip_fini.isra.0+0x3d/0x1b0 [amdgpu] 2024-12-25 12:00:54 [ 2703.953278] amdgpu_device_fini_sw+0x2a/0x240 [amdgpu] 2024-12-25 12:00:54 [ 2703.959789] amdgpu_driver_release_kms+0x12/0x30 [amdgpu] 2024-12-25 12:00:54 [ 2703.966501] devm_drm_dev_init_release+0x42/0x70 [drm] 2024-12-25 12:00:54 [ 2703.972891] release_nodes+0x6e/0xb0 2024-12-25 12:00:54 [ 2703.977522] amdgpu_xcp_drv_release+0x38/0x80 [amdxcp] 2024-12-25 12:00:54 [ 2703.983906] __do_sys_delete_module.constprop.0+0x138/0x2a0 2024-12-25 12:00:54 [ 2703.990775] ? exit_to_user_mode_loop+0xd6/0x120 2024-12-25 12:00:54 [ 2703.996563] do_syscall_64+0x2e/0x50 2024-12-25 12:00:54 [ 2704.001166] entry_SYSCALL_64_after_hwframe+0x62/0xc7 2024-12-25 12:00:54 [ 2704.007432] RIP: 0033:0x7fa4db2620cb 2024-12-25 12:00:54 [ 2704.012025] Code: 73 01 c3 48 8b 0d a5 6d 19 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 75 6d 19 00 f7 d8 64 89 01 48 Signed-off-by: Jiang Liu <gerry@xxxxxxxxxxxxxxxxx> --- drivers/gpu/drm/amd/amdkfd/kfd_device.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c index b6c5ffd4630b..58c1b5fcf785 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c @@ -663,6 +663,8 @@ static void kfd_cleanup_nodes(struct kfd_dev *kfd, unsigned int num_nodes) for (i = 0; i < num_nodes; i++) { knode = kfd->nodes[i]; + if (!knode) + continue;I wonder how knode can be null here? kfd_cleanup_nodes is called by kgd2kfd_device_exit under condition if (kfd->init_complete). kfd->init_complete is true only after all kfd node got initialized at kgd2kfd_device_init. If kfd driver init fail kfd->init_complete would be false, then kfd_cleanup_node would not get called.
Hi Xiaogang,It may get triggered on error handling path of ‘kid2kfd_device_init()`, when it jump to label `node_alloc_error` andthen call `kfd_cleanup_nodes()`.
If it was the case kzalloc failed. That means there is no memory available even to allocate struct kfd_node. From the backtrace the general protection fault happened at
RIP: 0010:kgd2kfd_device_exit+0x6/0x60 [amdgpu] It happened during driver module got released, not init time. I do not see how the patch is related to the issue you talked.Aha, it’s caused by another bug which got fixed by "[PATCH 4/6] amdgpu: fix use after free bug related to amdgpu_driver_release_kms()”.Without the fix in "[PATCH 4/6] amdgpu: fix use after free bug related to amdgpu_driver_release_kms()”, kgd2kfd_device_exit() will got calledtwice in case of device probe failure. I caused me some time to figure out the double free issue.Actually we should reset kfd->init_completed to false in function kgd2kfd_device_exit().We can set kfd->init_completed = false at end of kgd2kfd_device_exit, but how kgd2kfd_device_exit was called two times? is there another bug caused that?
I guess it caused by another bug related to the way amdgpu cooperates with the amdgpu_xcp driver. It would be better to enhance amdgpu_xcp driver either.
kfd driver has considered which kfd nodes got initialized and release them accordingly. From what saw here seems you may mix different issues or not target the real issue. Let's have backtrace match the changes.
Regards
Xiaogang
Regards
Xiaogang
Regards Xiaogang
Thanks,Gerry
Regards
Xiaogang
device_queue_manager_uninit(knode->dqm); kfd_interrupt_exit(knode); kfd_topology_remove_device(knode); @@ -954,7 +956,10 @@ void kgd2kfd_device_exit(struct kfd_dev *kfd) kfd_doorbell_fini(kfd); ida_destroy(&kfd->doorbell_ida); kfd_gtt_sa_fini(kfd); - amdgpu_amdkfd_free_gtt_mem(kfd->adev, &kfd->gtt_mem); + if (kfd->gtt_mem) { + amdgpu_amdkfd_free_gtt_mem(kfd->adev, &kfd->gtt_mem); + kfd->gtt_mem = NULL; + } } kfree(kfd);
