Hi, We are using Linux Kernel 3.18.19 and qemu-2.5 for running KVM VM's with VFIO assigned SRIOV VF interfaces. We are periodically seeing that when we spawn a KVM VM the underlying host freezes for several seconds. At worst this is like 30-40secs & most times it finishes in ~3-secs. With libvirt.xml we do specify cpu-pinning restricting the cpus for the VM. During this stall, all processes on the host appear to be stalled with severe delays in processing (this includes several bad sequences with pacemaker/heartbeat used in high availability at the host level). Running a script that captures iostat or proc/stack also gets stalled during this period. We got a short driver that sets a kthread to SCHED_FIFO & invokes “echo t>/proc/sysrq-trigger” equivalent every second. We see this kthread also gets stalled, but it printed some interesting stack traces with VFIO involved like in the below stack traces. Do you see any insights on what could cause this? Can you suggest ways to debug this issue further? Thanks. Apr 25 14:29:53 node2 kernel: [27357.255386] qemu-system-x86 R running task 0 11964 1 0x00000000 Apr 25 14:29:53 node2 kernel: [27357.255387] 0000000000000200 ffff880335891c00 0000000000200000 ffff88034154ba38 Apr 25 14:29:53 node2 kernel: [27357.255389] ffff88034154b9d8 ffff8803030446e0 ffff8803030446e0 0000000000000200 Apr 25 14:29:53 node2 kernel: [27357.255390] ffff8803030446e0 ffff880c7fff9e08 ffff8803030446e0 0000000300000002 Apr 25 14:29:53 node2 kernel: [27357.255392] Call Trace: Apr 25 14:29:53 node2 kernel: [27357.255395] [<ffffffff811d975d>] ? __mem_cgroup_threshold+0x9d/0x170 Apr 25 14:29:53 node2 kernel: [27357.255398] [<ffffffff811bf227>] ? alloc_pages_current+0x97/0x110 Apr 25 14:29:53 node2 kernel: [27357.255402] [<ffffffff811a1d80>] ? clear_huge_page+0x80/0x140 Apr 25 14:29:53 node2 kernel: [27357.255405] [<ffffffff811d5081>] ? do_huge_pmd_anonymous_page+0x241/0x410 Apr 25 14:29:53 node2 kernel: [27357.255408] [<ffffffff811a012b>] ? handle_mm_fault+0x19b/0x10d0 Apr 25 14:29:53 node2 kernel: [27357.255411] [<ffffffff815dc751>] ? dma_pte_free_level+0x101/0x140 Apr 25 14:29:53 node2 kernel: [27357.255414] [<ffffffff8119b560>] ? __get_user_pages+0x100/0x5f0 Apr 25 14:29:53 node2 kernel: [27357.255416] [<ffffffff8119baa2>] ? get_user_pages+0x52/0x60 Apr 25 14:29:53 node2 kernel: [27357.255419] [<ffffffff81065d60>] ? get_user_pages_fast+0x150/0x190 Apr 25 14:29:53 node2 kernel: [27357.255422] [<ffffffffc0b9832f>] ? vaddr_get_pfn+0x2f/0x100 [vfio_iommu_type1] Apr 25 14:29:53 node2 kernel: [27357.255424] [<ffffffffc0b991a9>] ? vfio_iommu_type1_ioctl+0x8a9/0xa40 [vfio_iommu_type1] Apr 25 14:29:53 node2 kernel: [27357.255427] [<ffffffffc0b8c669>] ? vfio_fops_unl_ioctl+0x79/0x2b0 [vfio] Apr 25 14:29:53 node2 kernel: [27357.255430] [<ffffffff811fb130>] ? do_vfs_ioctl+0x2e0/0x4c0 Apr 25 14:29:53 node2 kernel: [27357.255433] [<ffffffff81205382>] ? __fget+0x72/0xa0 Apr 25 14:29:53 node2 kernel: [27357.255435] [<ffffffff811fb391>] ? SyS_ioctl+0x81/0xa0 Apr 25 14:29:53 node2 kernel: [27357.255437] [<ffffffff817179cd>] ? system_call_fastpath+0x16/0x1b Apr 25 14:29:53 node2 kernel: [27358.339683] qemu-system-x86 D ffff880c4e533540 0 11964 1 0x00000000 Apr 25 14:29:53 node2 kernel: [27358.339685] ffff88034154bc98 0000000000000082 ffff8803030446e0 0000000000013540 Apr 25 14:29:53 node2 kernel: [27358.339686] ffff88034154bfd8 0000000000013540 ffff880c3b8c5b20 ffff8803030446e0 Apr 25 14:29:53 node2 kernel: [27358.339688] 0000000000000003 ffff880c3b780000 ffff88034154bcd8 00000001019c8c82 Apr 25 14:29:53 node2 kernel: [27358.339690] Call Trace: Apr 25 14:29:53 node2 kernel: [27358.339691] [<ffffffff817136c9>] schedule+0x29/0x70 Apr 25 14:29:53 node2 kernel: [27358.339693] [<ffffffff817163cf>] schedule_timeout+0x14f/0x2d0 Apr 25 14:29:53 node2 kernel: [27358.339696] [<ffffffff813cf7ce>] ? pci_bus_write_config_word.part.6+0x5e/0x70 Apr 25 14:29:53 node2 kernel: [27358.339699] [<ffffffff810d8bb0>] ? internal_add_timer+0xb0/0xb0 Apr 25 14:29:53 node2 kernel: [27358.339701] [<ffffffff810da5e7>] msleep+0x37/0x50 Apr 25 14:29:53 node2 kernel: [27358.339703] [<ffffffff813d83c4>] __pci_dev_reset+0x94/0x300 Apr 25 14:29:53 node2 kernel: [27358.339705] [<ffffffff813d8712>] pci_try_reset_function+0x62/0x80 Apr 25 14:29:53 node2 kernel: [27358.339707] [<ffffffffc0b9ef08>] vfio_pci_ioctl+0xf8/0xa50 [vfio_pci] Apr 25 14:29:53 node2 kernel: [27358.339709] [<ffffffff81226516>] ? fsnotify+0x2e6/0x410 Apr 25 14:29:53 node2 kernel: [27358.339711] [<ffffffffc0ba30c3>] ? vfio_pci_config_rw+0x193/0x411 [vfio_pci] Apr 25 14:29:53 node2 kernel: [27358.339714] [<ffffffffc0b8c1a3>] vfio_device_fops_unl_ioctl+0x23/0x30 [vfio] Apr 25 14:29:53 node2 kernel: [27358.339716] [<ffffffff811fb130>] do_vfs_ioctl+0x2e0/0x4c0 Apr 25 14:29:53 node2 kernel: [27358.339718] [<ffffffff81205382>] ? __fget+0x72/0xa0 Apr 25 14:29:53 node2 kernel: [27358.339720] [<ffffffff811fb391>] SyS_ioctl+0x81/0xa0 Apr 25 14:29:53 node2 kernel: [27358.339722] [<ffffffff817179cd>] system_call_fastpath+0x16/0x1b Apr 25 14:29:54 node2 kernel: [27359.427954] qemu-system-x86 R running task 0 11969 1 0x00000000 Apr 25 14:29:54 node2 kernel: [27359.427956] 0000000000000009 0000000000000007 0000000000000000 00000000bd000000 Apr 25 14:29:54 node2 kernel: [27359.427958] ffff88034bc73c38 ffffffff815dd7b6 0000000000000000 0000000000010000 Apr 25 14:29:54 node2 kernel: [27359.427959] 0000000000010000 0000000000010000 0000000000010000 ffff880bd9294e00 Apr 25 14:29:54 node2 kernel: [27359.427961] Call Trace: Apr 25 14:29:54 node2 kernel: [27359.427963] [<ffffffff815dd7b6>] ? iommu_flush_iotlb_psi+0xb6/0xf0 Apr 25 14:29:54 node2 kernel: [27359.427965] [<ffffffff815dfefb>] ? intel_iommu_unmap+0x15b/0x1b0 Apr 25 14:29:54 node2 kernel: [27359.427968] [<ffffffff8117fcc8>] ? put_compound_page+0xb8/0x1e0 Apr 25 14:29:54 node2 kernel: [27359.427971] [<ffffffff8117fe3b>] ? put_page+0x4b/0x50 Apr 25 14:29:54 node2 kernel: [27359.427973] [<ffffffffc0b981ca>] ? put_pfn+0x3a/0x60 [vfio_iommu_type1] Apr 25 14:29:54 node2 kernel: [27359.427975] [<ffffffffc0b985b7>] ? vfio_remove_dma+0x107/0x1a0 [vfio_iommu_type1] Apr 25 14:29:54 node2 kernel: [27359.427977] [<ffffffffc0b98ceb>] ? vfio_iommu_type1_ioctl+0x3eb/0xa40 [vfio_iommu_type1] Apr 25 14:29:54 node2 kernel: [27359.427989] [<ffffffffc067301a>] ? kvm_set_memory_region+0x3a/0x50 [kvm] Apr 25 14:29:54 node2 kernel: [27359.427996] [<ffffffffc067358a>] ? kvm_vm_ioctl+0x55a/0x7b0 [kvm] Apr 25 14:29:54 node2 kernel: [27359.427999] [<ffffffffc0b8c669>] ? vfio_fops_unl_ioctl+0x79/0x2b0 [vfio] Apr 25 14:29:54 node2 kernel: [27359.428001] [<ffffffff811fb130>] ? do_vfs_ioctl+0x2e0/0x4c0 Apr 25 14:29:54 node2 kernel: [27359.428003] [<ffffffff81205382>] ? __fget+0x72/0xa0 Apr 25 14:29:54 node2 kernel: [27359.428005] [<ffffffff811fb391>] ? SyS_ioctl+0x81/0xa0 Apr 25 14:29:54 node2 kernel: [27359.428007] [<ffffffff817179cd>] ? system_call_fastpath+0x16/0x1b Apr 25 14:29:56 node2 kernel: [27360.515078] qemu-system-x86 R running task 0 11969 1 0x00000000 Apr 25 14:29:56 node2 kernel: [27360.515079] ffff88035cb54040 ffff880bf1b60000 ffff88034bc73c88 ffffffffc06a75af Apr 25 14:29:56 node2 kernel: [27360.515081] 756e65470000000d ffff880bf1b60000 0000000000000000 000000000000001c Apr 25 14:29:56 node2 kernel: [27360.515083] ffff88035cb54040 ffffffffc06280fe ffff880bf1b60000 ffff880bf1b60000 Apr 25 14:29:56 node2 kernel: [27360.515084] Call Trace: Apr 25 14:29:56 node2 kernel: [27360.515097] [<ffffffffc06a75af>] ? kvm_emulate_cpuid+0xef/0x100 [kvm] Apr 25 14:29:56 node2 kernel: [27360.515100] [<ffffffff81718900>] ? reboot_interrupt+0x80/0x80 Apr 25 14:29:56 node2 kernel: [27360.515105] [<ffffffffc062765c>] ? vmx_handle_external_intr+0x6c/0x70 [kvm_intel] Apr 25 14:29:56 node2 kernel: [27360.515114] [<ffffffffc0688db8>] ? kvm_arch_vcpu_ioctl_run+0xb58/0x1320 [kvm] Apr 25 14:29:56 node2 kernel: [27360.515119] [<ffffffffc06291ce>] ? __vmx_load_host_state.part.45+0xbe/0x150 [kvm_intel] Apr 25 14:29:56 node2 kernel: [27360.515123] [<ffffffffc062923e>] ? __vmx_load_host_state.part.45+0x12e/0x150 [kvm_intel] Apr 25 14:29:56 node2 kernel: [27360.515132] [<ffffffffc068471e>] ? kvm_arch_vcpu_load+0x4e/0x1f0 [kvm] Apr 25 14:29:56 node2 kernel: [27360.515138] [<ffffffffc0671812>] ? kvm_vcpu_ioctl+0x2a2/0x600 [kvm] Apr 25 14:29:56 node2 kernel: [27360.515140] [<ffffffff810ee999>] ? do_futex+0x109/0xa60 Apr 25 14:29:56 node2 kernel: [27360.515143] [<ffffffffc0b8c675>] ? vfio_fops_unl_ioctl+0x85/0x2b0 [vfio] Apr 25 14:29:56 node2 kernel: [27360.515145] [<ffffffff811fb130>] ? do_vfs_ioctl+0x2e0/0x4c0 Apr 25 14:29:56 node2 kernel: [27360.515147] [<ffffffff81205382>] ? __fget+0x72/0xa0 Apr 25 14:29:56 node2 kernel: [27360.515149] [<ffffffff811fb015>] ? do_vfs_ioctl+0x1c5/0x4c0 Apr 25 14:29:56 node2 kernel: [27360.515151] [<ffffffff811fb391>] ? SyS_ioctl+0x81/0xa0 Apr 25 14:29:56 node2 kernel: [27360.515153] [<ffffffff817179cd>] ? system_call_fastpath+0x16/0x1b --Shyam -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
