Circular lock dependency chain between pm.mutex and topology_lock

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Evan and Mukul,

You both made changes recently involving the pm.mutex and the topology_lock respectively. I'm now seeing a circular lock dependency between those locks (see below) that could potentially lead to a deadlock. This cycle also involves the mmap_lock and some file-system related code through the SMU firmware loading. A summary of the dump below is:

 * Thread A can take mmap_lock while holding i_mutex_dir_key
 * Thread B can take i_mutex_dir_key while holding the pm.mutex
 * Thread C can take pm.mutex while holding topology_lock
 * Thread D can take topology_lock while holding mmap_lock

The backtraces below illustrate each of these scenarios. If four such threads run concurrently and try to enter these critical sections around the same time, you may get a deadlock between them. I see two potential ways to break this cycle within our driver:

1. Avoid holding the pm.mutex while loading SMU firmware
2. Avoid holding the topology lock while getting mem-info in the KFD
   topology code

Can you determine, which one of those is the better/easier solution and implement a fix?

Thank you,
  Felix


[  168.544078] ======================================================
[  168.550309] WARNING: possible circular locking dependency detected
[  168.556523] 5.16.0-kfd-fkuehlin #148 Tainted: G            E
[  168.562558] ------------------------------------------------------
[  168.568764] kfdtest/3479 is trying to acquire lock:
[  168.573672] ffffffffc0927a70 (&topology_lock){++++}-{3:3}, at: kfd_topology_device_by_id+0x16/0x60 [amdgpu]
[  168.583663]
               but task is already holding lock:
[  168.589529] ffff97d303dee668 (&mm->mmap_lock#2){++++}-{3:3}, at: vm_mmap_pgoff+0xa9/0x180
[  168.597755]
               which lock already depends on the new lock.

[  168.605970]
               the existing dependency chain (in reverse order) is:
[  168.613487]
               -> #3 (&mm->mmap_lock#2){++++}-{3:3}:
[  168.619700]        lock_acquire+0xca/0x2e0
[  168.623814]        down_read+0x3e/0x140
[  168.627676]        do_user_addr_fault+0x40d/0x690
[  168.632399]        exc_page_fault+0x6f/0x270
[  168.636692]        asm_exc_page_fault+0x1e/0x30
[  168.641249]        filldir64+0xc8/0x1e0
[  168.645115]        call_filldir+0x7c/0x110
[  168.649238]        ext4_readdir+0x58e/0x940
[  168.653442]        iterate_dir+0x16a/0x1b0
[  168.657558]        __x64_sys_getdents64+0x83/0x140
[  168.662375]        do_syscall_64+0x35/0x80
[  168.666492]        entry_SYSCALL_64_after_hwframe+0x44/0xae
[  168.672095]
               -> #2 (&type->i_mutex_dir_key#6){++++}-{3:3}:
[  168.679008]        lock_acquire+0xca/0x2e0
[  168.683122]        down_read+0x3e/0x140
[  168.686982]        path_openat+0x5b2/0xa50
[  168.691095]        do_file_open_root+0xfc/0x190
[  168.695652]        file_open_root+0xd8/0x1b0
[  168.702010]        kernel_read_file_from_path_initns+0xc4/0x140
[  168.709542]        _request_firmware+0x2e9/0x5e0
[  168.715741]        request_firmware+0x32/0x50
[  168.721667]        amdgpu_cgs_get_firmware_info+0x370/0xdd0 [amdgpu]
[  168.730060]        smu7_upload_smu_firmware_image+0x53/0x190 [amdgpu]
[  168.738414]        fiji_start_smu+0xcf/0x4e0 [amdgpu]
[  168.745539]        pp_dpm_load_fw+0x21/0x30 [amdgpu]
[  168.752503]        amdgpu_pm_load_smu_firmware+0x4b/0x80 [amdgpu]
[  168.760698]        amdgpu_device_fw_loading+0xb8/0x140 [amdgpu]
[  168.768412]        amdgpu_device_init.cold+0xdf6/0x1716 [amdgpu]
[  168.776285]        amdgpu_driver_load_kms+0x15/0x120 [amdgpu]
[  168.784034]        amdgpu_pci_probe+0x19b/0x3a0 [amdgpu]
[  168.791161]        local_pci_probe+0x40/0x80
[  168.797027]        work_for_cpu_fn+0x10/0x20
[  168.802839]        process_one_work+0x273/0x5b0
[  168.808903]        worker_thread+0x20f/0x3d0
[  168.814700]        kthread+0x176/0x1a0
[  168.819968]        ret_from_fork+0x1f/0x30
[  168.825563]
               -> #1 (&adev->pm.mutex){+.+.}-{3:3}:
[  168.834721]        lock_acquire+0xca/0x2e0
[  168.840364]        __mutex_lock+0xa2/0x930
[  168.846020]        amdgpu_dpm_get_mclk+0x37/0x60 [amdgpu]
[  168.853257]        amdgpu_amdkfd_get_local_mem_info+0xba/0xe0 [amdgpu]
[  168.861547]        kfd_create_vcrat_image_gpu+0x1b1/0xbb0 [amdgpu]
[  168.869478]        kfd_create_crat_image_virtual+0x447/0x510 [amdgpu]
[  168.877884]        kfd_topology_add_device+0x5c8/0x6f0 [amdgpu]
[  168.885556]        kgd2kfd_device_init.cold+0x385/0x4c5 [amdgpu]
[  168.893347]        amdgpu_amdkfd_device_init+0x138/0x180 [amdgpu]
[  168.901177]        amdgpu_device_init.cold+0x141b/0x1716 [amdgpu]
[  168.909025]        amdgpu_driver_load_kms+0x15/0x120 [amdgpu]
[  168.916458]        amdgpu_pci_probe+0x19b/0x3a0 [amdgpu]
[  168.923442]        local_pci_probe+0x40/0x80
[  168.929249]        work_for_cpu_fn+0x10/0x20
[  168.935008]        process_one_work+0x273/0x5b0
[  168.940944]        worker_thread+0x20f/0x3d0
[  168.946623]        kthread+0x176/0x1a0
[  168.951765]        ret_from_fork+0x1f/0x30
[  168.957277]
               -> #0 (&topology_lock){++++}-{3:3}:
[  168.965993]        check_prev_add+0x8f/0xbf0
[  168.971613]        __lock_acquire+0x1299/0x1ca0
[  168.977485]        lock_acquire+0xca/0x2e0
[  168.982877]        down_read+0x3e/0x140
[  168.987975]        kfd_topology_device_by_id+0x16/0x60 [amdgpu]
[  168.995583]        kfd_device_by_id+0xa/0x20 [amdgpu]
[  169.002180]        kfd_mmap+0x95/0x200 [amdgpu]
[  169.008293]        mmap_region+0x337/0x5a0
[  169.013679]        do_mmap+0x3aa/0x540
[  169.018678]        vm_mmap_pgoff+0xdc/0x180
[  169.024095]        ksys_mmap_pgoff+0x186/0x1f0
[  169.029734]        do_syscall_64+0x35/0x80
[  169.035005]        entry_SYSCALL_64_after_hwframe+0x44/0xae
[  169.041754]
               other info that might help us debug this:

[  169.053276] Chain exists of:
                 &topology_lock --> &type->i_mutex_dir_key#6 --> &mm->mmap_lock#2

[  169.068389]  Possible unsafe locking scenario:

[  169.076661]        CPU0                    CPU1
[  169.082383]        ----                    ----
[  169.088087]   lock(&mm->mmap_lock#2);
[  169.092922]                                lock(&type->i_mutex_dir_key#6);
[  169.100975]                                lock(&mm->mmap_lock#2);
[  169.108320]   lock(&topology_lock);
[  169.112957]
                *** DEADLOCK ***

[  169.122265] 1 lock held by kfdtest/3479:
[  169.127286]  #0: ffff97d303dee668 (&mm->mmap_lock#2){++++}-{3:3}, at: vm_mmap_pgoff+0xa9/0x180
[  169.137033]
               stack backtrace:
[  169.143579] CPU: 18 PID: 3479 Comm: kfdtest Tainted: G            E     5.16.0-kfd-fkuehlin #148
[  169.153480] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./EPYCD8-2T, BIOS P2.60 04/10/2020
[  169.164175] Call Trace:
[  169.167761]  <TASK>
[  169.170988]  dump_stack_lvl+0x45/0x59
[  169.175795]  check_noncircular+0xfe/0x110
[  169.180947]  check_prev_add+0x8f/0xbf0
[  169.185828]  __lock_acquire+0x1299/0x1ca0
[  169.190966]  lock_acquire+0xca/0x2e0
[  169.195677]  ? kfd_topology_device_by_id+0x16/0x60 [amdgpu]
[  169.202808]  down_read+0x3e/0x140
[  169.207333]  ? kfd_topology_device_by_id+0x16/0x60 [amdgpu]
[  169.214332]  kfd_topology_device_by_id+0x16/0x60 [amdgpu]
[  169.221123]  kfd_device_by_id+0xa/0x20 [amdgpu]
[  169.227071]  kfd_mmap+0x95/0x200 [amdgpu]
[  169.232521]  mmap_region+0x337/0x5a0
[  169.237305]  do_mmap+0x3aa/0x540
[  169.241713]  vm_mmap_pgoff+0xdc/0x180
[  169.246529]  ksys_mmap_pgoff+0x186/0x1f0
[  169.251603]  do_syscall_64+0x35/0x80
[  169.256335]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  169.262536] RIP: 0033:0x7f828e0d1916
[  169.267261] Code: 00 00 00 00 f3 0f 1e fa 41 f7 c1 ff 0f 00 00 75 2b 55 48 89 fd 53 89 cb 48 85 ff 74 37 41 89 da 48 89 ef b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 62 5b 5d c3 0f 1f 80 00 00 00 00 48 8b 05 41
[  169.288448] RSP: 002b:00007ffe9bc5e7c8 EFLAGS: 00000206 ORIG_RAX: 0000000000000009
[  169.297252] RAX: ffffffffffffffda RBX: 0000000000000011 RCX: 00007f828e0d1916
[  169.305628] RDX: 0000000000000003 RSI: 0000000000001000 RDI: 0000000000201000
[  169.313993] RBP: 0000000000201000 R08: 0000000000000003 R09: 04b4000000000000
[  169.322358] R10: 0000000000000011 R11: 0000000000000206 R12: 000055571c9c5a40
[  169.330733] R13: 0000000000000000 R14: 0000000000000011 R15: 0000000000000000
[  169.339097]  </TASK>


--
F e l i x   K u e h l i n g
PMTS Software Development Engineer | Linux Compute Kernel
1 Commerce Valley Dr. East, Markham, ON L3T 7X6 Canada
(O) +1(289)695-1597
    _     _   _   _____   _____
   / \   | \ / | |  _  \  \ _  |
  / A \  | \M/ | | |D) )  /|_| |
 /_/ \_\ |_| |_| |_____/ |__/ \|   facebook.com/AMD | amd.com




[Index of Archives]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux