On Wed, Apr 29, 2020 at 03:36:38PM +0200, Joerg Roedel wrote: > Hi, > > here is the third version of this patch-set. Older versions can be found > here: > > v1: https://lore.kernel.org/lkml/20200407183742.4344-1-joro@xxxxxxxxxx/ > (Has some more introductory text) > > v2: https://lore.kernel.org/lkml/20200414131542.25608-1-joro@xxxxxxxxxx/ > > Changes v2 -> v3: > > * Rebased v5.7-rc3 > > * Added a missing iommu_group_put() as reported by Lu Baolu. > > * Added a patch to consolidate more initialization work in > __iommu_probe_device(), fixing a bug where no 'struct > device_iommu' was allocated in the hotplug path. > > There is also a git-branch available with these patches applied: > > https://git.kernel.org/pub/scm/linux/kernel/git/joro/linux.git/log/?h=iommu-probe-device-v3 > > Please review. If there are no objections I plan to put these patches > into the IOMMU tree early next week. Looks like this patchset introduced an use-after-free on arm-smmu-v3. Reproduced using mlx5, # echo 1 > /sys/class/net/enp11s0f1np1/device/sriov_numvfs # echo 0 > /sys/class/net/enp11s0f1np1/device/sriov_numvfs The .config, https://github.com/cailca/linux-mm/blob/master/arm64.config Looking at the free stack, iommu_release_device->iommu_group_remove_device was introduced in 07/34 ("iommu: Add probe_device() and release_device() call-backs"). [ 9426.724641][ T3356] pci 0000:0b:01.2: Removing from iommu group 3 [ 9426.731347][ T3356] ================================================================== [ 9426.739263][ T3356] BUG: KASAN: use-after-free in __lock_acquire+0x3458/0x4440 __lock_acquire at kernel/locking/lockdep.c:4250 [ 9426.746477][ T3356] Read of size 8 at addr ffff0089df1a6f68 by task bash/3356 [ 9426.753601][ T3356] [ 9426.755782][ T3356] CPU: 5 PID: 3356 Comm: bash Not tainted 5.8.0-rc3-next-20200630 #2 [ 9426.763687][ T3356] Hardware name: HPE Apollo 70 /C01_APACHE_MB , BIOS L50_5.13_1.11 06/18/2019 [ 9426.774111][ T3356] Call trace: [ 9426.777245][ T3356] dump_backtrace+0x0/0x398 [ 9426.781593][ T3356] show_stack+0x14/0x20 [ 9426.785596][ T3356] dump_stack+0x140/0x1b8 [ 9426.789772][ T3356] print_address_description.isra.12+0x54/0x4a8 [ 9426.795855][ T3356] kasan_report+0x134/0x1b8 [ 9426.800203][ T3356] __asan_report_load8_noabort+0x2c/0x50 [ 9426.805679][ T3356] __lock_acquire+0x3458/0x4440 [ 9426.810373][ T3356] lock_acquire+0x204/0xf10 [ 9426.814722][ T3356] _raw_spin_lock_irqsave+0xf8/0x180 [ 9426.819853][ T3356] arm_smmu_detach_dev+0xd8/0x4a0 arm_smmu_detach_dev at drivers/iommu/arm-smmu-v3.c:2776 [ 9426.824721][ T3356] arm_smmu_release_device+0xb4/0x1c8 arm_smmu_disable_pasid at drivers/iommu/arm-smmu-v3.c:2754 (inlined by) arm_smmu_release_device at drivers/iommu/arm-smmu-v3.c:3000 [ 9426.829937][ T3356] iommu_release_device+0xc0/0x178 iommu_release_device at drivers/iommu/iommu.c:302 [ 9426.834892][ T3356] iommu_bus_notifier+0x118/0x160 [ 9426.839762][ T3356] notifier_call_chain+0xa4/0x128 [ 9426.844630][ T3356] __blocking_notifier_call_chain+0x70/0xa8 [ 9426.850367][ T3356] blocking_notifier_call_chain+0x14/0x20 [ 9426.855929][ T3356] device_del+0x618/0xa00 [ 9426.860105][ T3356] pci_remove_bus_device+0x108/0x2d8 [ 9426.865233][ T3356] pci_stop_and_remove_bus_device+0x1c/0x28 [ 9426.870972][ T3356] pci_iov_remove_virtfn+0x228/0x368 [ 9426.876100][ T3356] sriov_disable+0x8c/0x348 [ 9426.880447][ T3356] pci_disable_sriov+0x5c/0x70 [ 9426.885117][ T3356] mlx5_core_sriov_configure+0xd8/0x260 [mlx5_core] [ 9426.891549][ T3356] sriov_numvfs_store+0x240/0x318 [ 9426.896417][ T3356] dev_attr_store+0x38/0x68 [ 9426.900766][ T3356] sysfs_kf_write+0xdc/0x128 [ 9426.905200][ T3356] kernfs_fop_write+0x23c/0x448 [ 9426.909897][ T3356] __vfs_write+0x54/0xe8 [ 9426.913984][ T3356] vfs_write+0x124/0x3f0 [ 9426.918070][ T3356] ksys_write+0xe8/0x1b8 [ 9426.922157][ T3356] __arm64_sys_write+0x68/0x98 [ 9426.926766][ T3356] do_el0_svc+0x124/0x220 [ 9426.930941][ T3356] el0_sync_handler+0x260/0x408 [ 9426.935634][ T3356] el0_sync+0x140/0x180 [ 9426.939633][ T3356] [ 9426.941810][ T3356] Allocated by task 3356: [ 9426.945985][ T3356] save_stack+0x24/0x50 [ 9426.949986][ T3356] __kasan_kmalloc.isra.13+0xc4/0xe0 [ 9426.955114][ T3356] kasan_kmalloc+0xc/0x18 [ 9426.959288][ T3356] kmem_cache_alloc_trace+0x1ec/0x318 [ 9426.964503][ T3356] arm_smmu_domain_alloc+0x54/0x148 [ 9426.969545][ T3356] iommu_group_alloc_default_domain+0xc0/0x440 [ 9426.975541][ T3356] iommu_probe_device+0x1c0/0x308 [ 9426.980409][ T3356] iort_iommu_configure+0x434/0x518 [ 9426.985452][ T3356] acpi_dma_configure+0xf0/0x128 [ 9426.990235][ T3356] pci_dma_configure+0x114/0x160 [ 9426.995017][ T3356] really_probe+0x124/0x6d8 [ 9426.999364][ T3356] driver_probe_device+0xc4/0x180 [ 9427.004232][ T3356] __device_attach_driver+0x184/0x1e8 [ 9427.009447][ T3356] bus_for_each_drv+0x114/0x1a0 [ 9427.014142][ T3356] __device_attach+0x19c/0x2a8 [ 9427.018749][ T3356] device_attach+0x10/0x18 [ 9427.023009][ T3356] pci_bus_add_device+0x70/0xf8 [ 9427.027704][ T3356] pci_iov_add_virtfn+0x7b4/0xb40 [ 9427.032571][ T3356] sriov_enable+0x5c8/0xc30 [ 9427.036918][ T3356] pci_enable_sriov+0x64/0x80 [ 9427.041485][ T3356] mlx5_core_sriov_configure+0x58/0x260 [mlx5_core] [ 9427.047917][ T3356] sriov_numvfs_store+0x1c0/0x318 [ 9427.052784][ T3356] dev_attr_store+0x38/0x68 [ 9427.057131][ T3356] sysfs_kf_write+0xdc/0x128 [ 9427.061565][ T3356] kernfs_fop_write+0x23c/0x448 [ 9427.066260][ T3356] __vfs_write+0x54/0xe8 [ 9427.070346][ T3356] vfs_write+0x124/0x3f0 [ 9427.074433][ T3356] ksys_write+0xe8/0x1b8 [ 9427.078519][ T3356] __arm64_sys_write+0x68/0x98 [ 9427.083127][ T3356] do_el0_svc+0x124/0x220 [ 9427.087300][ T3356] el0_sync_handler+0x260/0x408 [ 9427.091994][ T3356] el0_sync+0x140/0x180 [ 9427.095992][ T3356] [ 9427.098168][ T3356] Freed by task 3356: [ 9427.101995][ T3356] save_stack+0x24/0x50 [ 9427.105996][ T3356] __kasan_slab_free+0x124/0x198 [ 9427.110777][ T3356] kasan_slab_free+0x10/0x18 [ 9427.115210][ T3356] slab_free_freelist_hook+0x110/0x298 [ 9427.120512][ T3356] kfree+0x128/0x668 [ 9427.124252][ T3356] arm_smmu_domain_free+0xf4/0x1a0 [ 9427.129206][ T3356] iommu_group_release+0xec/0x160 [ 9427.134074][ T3356] kobject_put+0xf4/0x238 [ 9427.138247][ T3356] kobject_del+0x110/0x190 [ 9427.142507][ T3356] kobject_put+0x1e4/0x238 [ 9427.146767][ T3356] iommu_group_remove_device+0x394/0x938 [ 9427.152242][ T3356] iommu_release_device+0x9c/0x178 iommu_release_device at drivers/iommu/iommu.c:300 [ 9427.157196][ T3356] iommu_bus_notifier+0x118/0x160 [ 9427.162065][ T3356] notifier_call_chain+0xa4/0x128 [ 9427.166934][ T3356] __blocking_notifier_call_chain+0x70/0xa8 [ 9427.172670][ T3356] blocking_notifier_call_chain+0x14/0x20 [ 9427.178233][ T3356] device_del+0x618/0xa00 [ 9427.182406][ T3356] pci_remove_bus_device+0x108/0x2d8 [ 9427.187535][ T3356] pci_stop_and_remove_bus_device+0x1c/0x28 [ 9427.193271][ T3356] pci_iov_remove_virtfn+0x228/0x368 [ 9427.198399][ T3356] sriov_disable+0x8c/0x348 [ 9427.202746][ T3356] pci_disable_sriov+0x5c/0x70 [ 9427.207398][ T3356] mlx5_core_sriov_configure+0xd8/0x260 [mlx5_core] [ 9427.213830][ T3356] sriov_numvfs_store+0x240/0x318 [ 9427.218698][ T3356] dev_attr_store+0x38/0x68 [ 9427.223045][ T3356] sysfs_kf_write+0xdc/0x128 [ 9427.227478][ T3356] kernfs_fop_write+0x23c/0x448 [ 9427.232173][ T3356] __vfs_write+0x54/0xe8 [ 9427.236259][ T3356] vfs_write+0x124/0x3f0 [ 9427.240346][ T3356] ksys_write+0xe8/0x1b8 [ 9427.244433][ T3356] __arm64_sys_write+0x68/0x98 [ 9427.249041][ T3356] do_el0_svc+0x124/0x220 [ 9427.253215][ T3356] el0_sync_handler+0x260/0x408 [ 9427.257908][ T3356] el0_sync+0x140/0x180 [ 9427.261907][ T3356] [ 9427.264084][ T3356] The buggy address belongs to the object at ffff0089df1a6e00 [ 9427.264084][ T3356] which belongs to the cache kmalloc-512 of size 512 [ 9427.277980][ T3356] The buggy address is located 360 bytes inside of [ 9427.277980][ T3356] 512-byte region [ffff0089df1a6e00, ffff0089df1a7000) [ 9427.291094][ T3356] The buggy address belongs to the page: [ 9427.296571][ T3356] page:ffffffe02257c680 refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff0089df1a1400 [ 9427.306823][ T3356] flags: 0x7ffff800000200(slab) [ 9427.311520][ T3356] raw: 007ffff800000200 ffffffe02246b8c8 ffffffe02257ff88 ffff000000320680 [ 9427.319949][ T3356] raw: ffff0089df1a1400 00000000002a000e 00000001ffffffff ffff0089df1a5001 [ 9427.328374][ T3356] page dumped because: kasan: bad access detected [ 9427.334630][ T3356] page->mem_cgroup:ffff0089df1a5001 [ 9427.339670][ T3356] [ 9427.341846][ T3356] Memory state around the buggy address: [ 9427.347322][ T3356] ffff0089df1a6e00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 9427.355228][ T3356] ffff0089df1a6e80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 9427.363133][ T3356] >ffff0089df1a6f00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 9427.371038][ T3356] ^ [ 9427.378337][ T3356] ffff0089df1a6f80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb [ 9427.386242][ T3356] ffff0089df1a7000: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 9427.394146][ T3356] ================================================================== [ 9427.402052][ T3356] Disabling lock debugging due to kernel taint > > Thanks, > > Joerg > > Joerg Roedel (33): > iommu: Move default domain allocation to separate function > iommu/amd: Implement iommu_ops->def_domain_type call-back > iommu/vt-d: Wire up iommu_ops->def_domain_type > iommu/amd: Remove dma_mask check from check_device() > iommu/amd: Return -ENODEV in add_device when device is not handled by > IOMMU > iommu: Add probe_device() and release_device() call-backs > iommu: Move default domain allocation to iommu_probe_device() > iommu: Keep a list of allocated groups in __iommu_probe_device() > iommu: Move new probe_device path to separate function > iommu: Split off default domain allocation from group assignment > iommu: Move iommu_group_create_direct_mappings() out of > iommu_group_add_device() > iommu: Export bus_iommu_probe() and make is safe for re-probing > iommu/amd: Remove dev_data->passthrough > iommu/amd: Convert to probe/release_device() call-backs > iommu/vt-d: Convert to probe/release_device() call-backs > iommu/arm-smmu: Convert to probe/release_device() call-backs > iommu/pamu: Convert to probe/release_device() call-backs > iommu/s390: Convert to probe/release_device() call-backs > iommu/virtio: Convert to probe/release_device() call-backs > iommu/msm: Convert to probe/release_device() call-backs > iommu/mediatek: Convert to probe/release_device() call-backs > iommu/mediatek-v1 Convert to probe/release_device() call-backs > iommu/qcom: Convert to probe/release_device() call-backs > iommu/rockchip: Convert to probe/release_device() call-backs > iommu/tegra: Convert to probe/release_device() call-backs > iommu/renesas: Convert to probe/release_device() call-backs > iommu/omap: Remove orphan_dev tracking > iommu/omap: Convert to probe/release_device() call-backs > iommu/exynos: Use first SYSMMU in controllers list for IOMMU core > iommu/exynos: Convert to probe/release_device() call-backs > iommu: Remove add_device()/remove_device() code-paths > iommu: Move more initialization to __iommu_probe_device() > iommu: Unexport iommu_group_get_for_dev() > > Sai Praneeth Prakhya (1): > iommu: Add def_domain_type() callback in iommu_ops > > drivers/iommu/amd_iommu.c | 97 ++++---- > drivers/iommu/amd_iommu_types.h | 1 - > drivers/iommu/arm-smmu-v3.c | 38 +--- > drivers/iommu/arm-smmu.c | 39 ++-- > drivers/iommu/exynos-iommu.c | 24 +- > drivers/iommu/fsl_pamu_domain.c | 22 +- > drivers/iommu/intel-iommu.c | 68 +----- > drivers/iommu/iommu.c | 387 +++++++++++++++++++++++++------- > drivers/iommu/ipmmu-vmsa.c | 60 ++--- > drivers/iommu/msm_iommu.c | 34 +-- > drivers/iommu/mtk_iommu.c | 24 +- > drivers/iommu/mtk_iommu_v1.c | 50 ++--- > drivers/iommu/omap-iommu.c | 99 ++------ > drivers/iommu/qcom_iommu.c | 24 +- > drivers/iommu/rockchip-iommu.c | 26 +-- > drivers/iommu/s390-iommu.c | 22 +- > drivers/iommu/tegra-gart.c | 24 +- > drivers/iommu/tegra-smmu.c | 31 +-- > drivers/iommu/virtio-iommu.c | 41 +--- > include/linux/iommu.h | 21 +- > 20 files changed, 531 insertions(+), 601 deletions(-) > > -- > 2.17.1 > > _______________________________________________ > iommu mailing list > iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx > https://lists.linuxfoundation.org/mailman/listinfo/iommu
