Hello,
We met a possible recursive locking issue and seeking a solution for resolving this. The log is looking like below:
[ 5102.127454] ============================================
[ 5102.133379] WARNING: possible recursive locking detected
[ 5102.139304] 4.12.0-rc4+ #3 Not tainted
[ 5102.143483] --------------------------------------------
[ 5102.149407] qemu-system-x86/1620 is trying to acquire lock:
[ 5102.155624] (&container->group_lock){++++++}, at: [<ffffffff817768c6>] vfio_unpin_pages+0x96/0xf0
[ 5102.165626]
but task is already holding lock:
[ 5102.172134] (&container->group_lock){++++++}, at: [<ffffffff8177728f>] vfio_fops_unl_ioctl+0x5f/0x280
[ 5102.182522]
other info that might help us debug this:
[ 5102.189806] Possible unsafe locking scenario:
[ 5102.196411] CPU0
[ 5102.199136] ----
[ 5102.201861] lock(&container->group_lock);
[ 5102.206527] lock(&container->group_lock);
[ 5102.211191]
*** DEADLOCK ***
[ 5102.217796] May be due to missing lock nesting notation
[ 5102.225370] 3 locks held by qemu-system-x86/1620:
[ 5102.230618] #0: (&container->group_lock){++++++}, at: [<ffffffff8177728f>] vfio_fops_unl_ioctl+0x5f/0x280
[ 5102.241482] #1: (&(&iommu->notifier)->rwsem){++++..}, at: [<ffffffff810de775>] __blocking_notifier_call_chain+0x35/0x70
[ 5102.253713] #2: (&vgpu->vdev.cache_lock){+.+...}, at: [<ffffffff8157b007>] intel_vgpu_iommu_notifier+0x77/0x120
[ 5102.265163]
stack backtrace:
[ 5102.270022] CPU: 5 PID: 1620 Comm: qemu-system-x86 Not tainted 4.12.0-rc4+ #3
[ 5102.277991] Hardware name: Intel Corporation S1200RP/S1200RP, BIOS S1200RP.86B.03.01.APER.061220151418 06/12/2015
[ 5102.289445] Call Trace:
[ 5102.292175] dump_stack+0x85/0xc7
[ 5102.295871] validate_chain.isra.21+0x9da/0xaf0
[ 5102.300925] __lock_acquire+0x405/0x820
[ 5102.305202] lock_acquire+0xc7/0x220
[ 5102.309191] ? vfio_unpin_pages+0x96/0xf0
[ 5102.313666] down_read+0x2b/0x50
[ 5102.317259] ? vfio_unpin_pages+0x96/0xf0
[ 5102.321732] vfio_unpin_pages+0x96/0xf0
[ 5102.326024] intel_vgpu_iommu_notifier+0xe5/0x120
[ 5102.331283] notifier_call_chain+0x4a/0x70
[ 5102.335851] __blocking_notifier_call_chain+0x4d/0x70
[ 5102.341490] blocking_notifier_call_chain+0x16/0x20
[ 5102.346935] vfio_iommu_type1_ioctl+0x87b/0x920
[ 5102.351994] vfio_fops_unl_ioctl+0x81/0x280
[ 5102.356660] ? __fget+0xf0/0x210
[ 5102.360261] do_vfs_ioctl+0x93/0x6a0
[ 5102.364247] ? __fget+0x111/0x210
[ 5102.367942] SyS_ioctl+0x41/0x70
[ 5102.371542] entry_SYSCALL_64_fastpath+0x1f/0xbe
The call stack is:
vfio_fops_unl_ioctl -> vfio_iommu_type1_ioctl -> vfio_dma_do_unmap -> blocking_notifier_call_chain -> intel_vgpu_iommu_notifier -> vfio_unpin_pages.
The container->group_lock is hold in vfio_fops_unl_ioctl first but then it will be hold again in vfio_unpin_pages. Regarding this, put the vfio_unpin_pages in another thread can resolve this recursive locking. In this way, vfio_unpin_pages will be asynchronies with the vfio_dma_do_unmap. Then it is possible to trigger below kernel panic due to this asynchronies:
[ 4468.975091] ------------[ cut here ]------------
[ 4468.976145] kernel BUG at drivers/vfio/vfio_iommu_type1.c:833!
[ 4468.977193] invalid opcode: 0000 [#1] SMP
[ 4468.978232] Modules linked in: bridge stp llc nls_iso8859_1 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel joydev input_leds crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd hci_uart glue_helper shpchp btbcm cryptd btqca winbond_cir rc_core btintel ipmi_ssif mei_me mei intel_pch_thermal bluetooth acpi_als kfifo_buf industrialio soc_button_array intel_vbtn ipmi_devintf ipmi_msghandler intel_hid ecdh_generic intel_lpss_acpi intel_lpss spidev sparse_keymap acpi_power_meter mac_hid sunrpc parport_pc ppdev lp parport autofs4 kvmgt vfio_mdev mdev vfio_iommu_type1 vfio kvm irqbypass hid_generic usbhid i915 drm_kms_helper igb e1000e syscopyarea sysfillrect sysimgblt dca fb_sys_fops ptp pps_core drm i2c_algo_bit ahci libahci wmi video pinctrl_sunrisepoint
[ 4468.982783] pinctrl_intel i2c_hid hid
[ 4468.983995] CPU: 3 PID: 1549 Comm: qemu-system-x86 Not tainted 4.12.0-rc7+ #1
[ 4468.985132] Hardware name: Intel Corporation Kabylake Greenlow Refresh UP Server Platform/Zumba Beach Server EV, BIOS KBLSE2R1.R00.0006.B08.1702011304 02/
[ 4468.986350] task: ffff8cd9afa28000 task.stack: ffffb136c2f68000
[ 4468.987545] RIP: 0010:vfio_iommu_type1_ioctl+0x894/0x910 [vfio_iommu_type1]
[ 4468.988864] RSP: 0018:ffffb136c2f6bd58 EFLAGS: 00010202
[ 4468.990106] RAX: 0000000000100000 RBX: 00007f80f55b1410 RCX: 000000007ff00000
[ 4468.991290] RDX: ffff8cd9af105d00 RSI: ffff8cd9b4835e40 RDI: 000000000000000b
[ 4468.992536] RBP: ffffb136c2f6be30 R08: 0000000000100000 R09: 0000000080000000
[ 4468.993749] R10: ffffb136c2f6bd30 R11: 000000000000013b R12: 0000000000000000
[ 4468.994991] R13: ffff8cd9ad813b80 R14: ffff8cd9afa28000 R15: ffffb136c2f6bdc8
[ 4468.996131] FS: 00007f80f55b2700(0000) GS:ffff8cd9c7d80000(0000) knlGS:0000000000000000
[ 4468.997305] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4468.998525] CR2: 000001f016dd2218 CR3: 0000000472bfc000 CR4: 00000000003426e0
[ 4468.999668] Call Trace:
[ 4469.000916] ? kvm_set_memory_region+0x38/0x60 [kvm]
[ 4469.002072] vfio_fops_unl_ioctl+0x7b/0x260 [vfio]
[ 4469.003220] do_vfs_ioctl+0xa1/0x5d0
[ 4469.004443] ? SyS_futex+0x7f/0x180
[ 4469.005567] SyS_ioctl+0x79/0x90
[ 4469.006661] entry_SYSCALL_64_fastpath+0x1e/0xa9
Can you help to check this recursive locking issue?
Thanks
Chuanxiao
