W dniu 17.01.2013 07:37, Gleb Natapov pisze:
On Wed, Jan 16, 2013 at 04:42:38PM +0200, Michael S. Tsirkin wrote:
First test:
- kvm.git kernel
- 2 kvm guest running:
- Linux (in idle)
- Freebsd-amd64 (high load, about 7 -- continuous FreeBSD world
and kernel build)
- KVM host hangs after about 5 hours
- nothing special in system logs
- message caught on one of the active SSH session:
kernel:[24742.127690] BUG: soft lockup - CPU#2 stuck for 22s!
[vhost-3686:3700]
Second test:
- kvm.git kernel
- 1 kvm guest running:
- Linux (at the time of hang -- in idle)
- about 10 minutes before KVM host hangs -- load about 6 (kernel build)
- in system logs:
BUG: soft lockup - CPU#0 stuck for 22s! [vhost-1771:1800]
Modules linked in: binfmt_misc ip6table_filter ip6_tables
ebtable_nat ebtables lockd sunrpc nf_conntrack_ipv4 nf_defrag_ipv4
xt_state nf_conntrack xt_CHECKSUM iptable_mangle bridge stp llc
be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3
mdio libcxgbi ib_iser bnep bluetooth rfkill rdma_cm ib_addr iw_cm
ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi ioatdma vhost_net iTCO_wdt iTCO_vendor_support
ses lpc_ich tun macvtap macvlan mfd_core enclosure bnx2 joydev
i7core_edac dca edac_core wmi coretemp dcdbas kvm_intel pcspkr
crc32c_intel kvm serio_raw acpi_power_meter microcode uinput
ipmi_devintf ipmi_si ipmi_msghandler megaraid_sas
CPU 0
Pid: 1800, comm: vhost-1771 Not tainted 3.7.0+ #2 Dell Inc.
PowerEdge R610/086HF8
RIP: 0010:[<ffffffff8152876f>] [<ffffffff8152876f>]
skb_flow_dissect+0xbf/0x3e0
RSP: 0018:ffff88042145dbd8 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8807fa489c00 RCX: f7ab0c277df5b6fd
RDX: ffff880820c59800 RSI: ffff88042145dc58 RDI: ffff8807fa489c00
RBP: ffff88042145dc48 R08: 0000000000000404 R09: 0000000000000412
R10: ffff8807fa489c00 R11: 0000000000000412 R12: ffffffff81522a57
R13: 0000000000000000 R14: ffffffff81521fdc R15: ffff88042145db78
FS: 0000000000000000(0000) GS:ffff88083fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001c0e9f4 CR3: 0000000827973000 CR4: 00000000000027e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vhost-1771 (pid: 1800, threadinfo ffff88042145c000, task
ffff880421151740)
Stack:
ffff88042145dc78 0000000000000412 0000000000000000 0000000000000412
000000000000000c ffff880422570800 ffff88042145dc78 ffffffff81525881
0000000000000000 00000000de057f32 ffff8807fa489c00 0000000000000412
Call Trace:
[<ffffffff81525881>] ? skb_copy_datagram_from_iovec+0x61/0x280
[<ffffffff8152a06a>] __skb_get_rxhash+0x1a/0xd0
[<ffffffffa01109e0>] tun_get_user+0x3e0/0x760 [tun]
[<ffffffffa0110dba>] tun_sendmsg+0x5a/0x80 [tun]
[<ffffffffa013cd2a>] handle_tx+0x28a/0x680 [vhost_net]
[<ffffffffa013d155>] handle_tx_kick+0x15/0x20 [vhost_net]
[<ffffffffa013995d>] vhost_worker+0xed/0x190 [vhost_net]
[<ffffffffa0139870>] ? vhost_work_flush+0x110/0x110 [vhost_net]
[<ffffffff81081750>] kthread+0xc0/0xd0
[<ffffffff81010000>] ? ftrace_define_fields_xen_mc_entry+0x50/0xf0
[<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
[<ffffffff8163feac>] ret_from_fork+0x7c/0xb0
[<ffffffff81081690>] ? kthread_create_on_node+0x120/0x120
Code: 68 41 2b 44 24 6c 29 d8 83 f8 13 0f 8e eb 00 00 00 48 63 d3 49
03 94 24 e8 00 00 00 48 85 d2 74 b0 31 c0 66 f7 42 06 3f ff 75 04
<0f> b6 42 09 48 8b 4a 0c 49 89 0e 0f b6 12 83 e2 0f 8d 1c 93 eb
repeated several times with similar content
--
Artur
Looks like this?
https://git.kernel.org/?p=linux/kernel/git/davem/net.git;a=commit;h=76fe45812a3b134c39170ca32dfd4b7217d33145
Artur,
Can you please apply this fix on top of kvm.git/queue and try again.
Hi, thanks for the advice. I applied the patch created from commit
76fe45812a3b134c39170ca32dfd4b7217d33145 to branch queue previous
evening. Fortunately, it seems that it works now. During the whole night
(about 10 hours) I tested two virtual machines (Linux and FreeBSD-amd64)
under high load and nothing bad happened.
From the first observation it seems to me that the performance of the
VM with FreeBSD-amd64 is much worse than the VM with FreeBSD-i386 --
but, it requires further testing.
--
Artur
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
