Hello, I noticed few times, that when I try to run tcpdump on 2.6.32.x, the guest end up badly, it gets stuck for some time, and either reboots, or gets into pretty good state. I'm using virtio, both host and guests are x86_64, host is 2.6.37 (but I noticed this problem also with older host kernels). here's first backtrace (guest rebooted): Apr 1 04:45:03 vmnci02 [30055.672138] BUG: soft lockup - CPU#0 stuck for 66s! [crond:3053] Apr 1 04:45:03 vmnci02 [30055.672144] Modules linked in: loop bluetooth rfkill ipv6 lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac parport_pc lp parport nvram sg virtio_balloon thermal processor button thermal_sys tpm_tis tpm tpm_bios piix i2c_piix4 i2c_core pcspkr virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] Apr 1 04:45:03 vmnci02 [30055.672220] CPU 0: Apr 1 04:45:03 vmnci02 [30055.672223] Modules linked in: loop bluetooth rfkill ipv6 lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac parport_pc lp parport nvram sg virtio_balloon thermal processor button thermal_sys tpm_tis tpm tpm_bios piix i2c_piix4 i2c_core pcspkr virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] Apr 1 04:45:03 vmnci02 [30055.672279] Pid: 3053, comm: crond Not tainted 2.6.32lb.36.kvmguest #1 Bochs Apr 1 04:45:03 vmnci02 [30055.672282] RIP: 0010:[<ffffffff8133fe10>] [<ffffffff8133fe10>] _spin_unlock_irq+0x10/0x40 Apr 1 04:45:03 vmnci02 [30055.672437] RSP: 0000:ffff88000469fd98 EFLAGS: 00000202 Apr 1 04:45:03 vmnci02 [30055.672441] RAX: 0000000000000000 RBX: ffff88000469fd98 RCX: 00000000c0000100 Apr 1 04:45:03 vmnci02 [30055.672444] RDX: 0000000000003130 RSI: 0000000000000000 RDI: ffff880001813bc0 Apr 1 04:45:03 vmnci02 [30055.672447] RBP: ffffffff8100bf5e R08: ffff88000469e000 R09: 0000000000000000 Apr 1 04:45:03 vmnci02 [30055.672451] R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff8104bc16 Apr 1 04:45:03 vmnci02 [30055.672454] R13: ffff88000469fd28 R14: 0000000000000000 R15: ffff8800046b9750 Apr 1 04:45:03 vmnci02 [30055.672461] FS: 00007eff7edda710(0000) GS:ffff880001800000(0000) knlGS:0000000000000000 Apr 1 04:45:03 vmnci02 [30055.672465] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Apr 1 04:45:03 vmnci02 [30055.672468] CR2: 00007f2ddef71a3d CR3: 00000000044ba000 CR4: 00000000000006f0 Apr 1 04:45:03 vmnci02 [30055.672480] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Apr 1 04:45:03 vmnci02 [30055.672491] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Apr 1 04:45:03 vmnci02 [30055.672494] Call Trace: Apr 1 04:45:03 vmnci02 [30055.672501] [<ffffffff8133fe09>] ? _spin_unlock_irq+0x9/0x40 Apr 1 04:45:03 vmnci02 [30055.672579] [<ffffffff81041c1f>] ? finish_task_switch+0x4f/0xd0 Apr 1 04:45:03 vmnci02 [30055.672658] [<ffffffff8133cdbf>] ? thread_return+0x4c/0x77d Apr 1 04:45:03 vmnci02 [30055.672664] [<ffffffff8133fe65>] ? _spin_unlock_irqrestore+0x25/0x50 Apr 1 04:45:03 vmnci02 [30055.672774] [<ffffffff811c6f14>] ? _raw_spin_lock+0x54/0x160 Apr 1 04:45:03 vmnci02 [30055.672780] [<ffffffff8133ed45>] ? do_nanosleep+0x75/0xb0 Apr 1 04:45:03 vmnci02 [30055.672820] [<ffffffff8107563e>] ? hrtimer_nanosleep+0x9e/0x120 Apr 1 04:45:03 vmnci02 [30055.672825] [<ffffffff810747a0>] ? hrtimer_wakeup+0x0/0x30 Apr 1 04:45:03 vmnci02 [30055.672830] [<ffffffff8107572f>] ? sys_nanosleep+0x6f/0x80 Apr 1 04:45:03 vmnci02 [30055.672837] [<ffffffff8100b402>] ? system_call_fastpath+0x16/0x1b here's second backtrace, guest survived, but in pretty bad condition (notice ext4 journal abort at the end): [161990.733858] BUG: spinlock lockup on CPU#1, scsi_eh_0/1046, ffff88011a01a3d8 [162016.148705] BUG: soft lockup - CPU#6 stuck for 205s! [sshd:9310] [162016.148705] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162016.148705] CPU 6: [162016.148705] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162016.148705] Pid: 9310, comm: sshd Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162016.148705] RIP: 0010:[<ffffffff811c58a6>] [<ffffffff811c58a6>] iowrite16+0x26/0x40 [162016.148705] RSP: 0018:ffff8800aae0b638 EFLAGS: 00000206 [162016.148705] RAX: 0000000000000001 RBX: ffff8800aae0b638 RCX: ffff88011a1bd000 [162016.148705] RDX: 000000000000c050 RSI: 000000000001c050 RDI: 0000000000000001 [162016.148705] RBP: ffffffff8100bf5e R08: 0000000000000003 R09: 0000000000000003 [162016.148705] R10: 0000000000000030 R11: 0000000000000000 R12: ffffffff8151ce10 [162016.148705] R13: 0000000000000246 R14: 0000000000000206 R15: ffffea0002aba900 [162016.148705] FS: 00007f52fb21b730(0000) GS:ffff880028380000(0000) knlGS:0000000000000000 [162016.148705] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [162016.148705] CR2: 00007f2ed8640e80 CR3: 00000000d2c71000 CR4: 00000000000006e0 [162016.148705] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162016.148705] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162016.148705] Call Trace: [162016.148705] [<ffffffffa01b613d>] ? vp_notify+0x2d/0x40 [virtio_pci] [162016.148705] [<ffffffffa01af033>] ? vring_kick+0x33/0x40 [virtio_ring] [162016.148705] [<ffffffffa01be4a0>] ? start_xmit+0x1a0/0x400 [virtio_net] [162016.148705] [<ffffffff812a6f49>] ? dev_hard_start_xmit+0x2c9/0x360 [162016.148705] [<ffffffff812bbc3f>] ? sch_direct_xmit+0x16f/0x1e0 [162016.148705] [<ffffffff812aa30b>] ? dev_queue_xmit+0x24b/0x3f0 [162016.148705] [<ffffffff812d513c>] ? ip_finish_output+0x12c/0x340 [162016.148705] [<ffffffff812d61e0>] ? ip_output+0xb0/0x100 [162016.148705] [<ffffffff812d4bf0>] ? ip_local_out+0x20/0x30 [162016.148705] [<ffffffff812d6441>] ? ip_queue_xmit+0x211/0x3f0 [162016.148705] [<ffffffff811258f0>] ? __pollwait+0x0/0x100 [162016.148705] [<ffffffff812e9545>] ? tcp_transmit_skb+0x3a5/0x780 [162016.148705] [<ffffffff811259f0>] ? pollwake+0x0/0x60 [162016.148705] [<ffffffff812eac32>] ? tcp_write_xmit+0x3b2/0xa00 [162016.148705] [<ffffffff812eb2ab>] ? __tcp_push_pending_frames+0x2b/0xa0 [162016.148705] [<ffffffff812de5e7>] ? tcp_sendmsg+0x7e7/0xb10 [162016.148705] [<ffffffff812949f7>] ? sock_aio_write+0x137/0x150 [162016.148705] [<ffffffff8103d14e>] ? __wake_up+0x4e/0x70 [162016.148705] [<ffffffff81113bf1>] ? do_sync_write+0xf1/0x130 [162016.148705] [<ffffffff81071050>] ? autoremove_wake_function+0x0/0x40 [162016.148705] [<ffffffff81058532>] ? current_fs_time+0x22/0x30 [162016.148705] [<ffffffff81196c41>] ? security_file_permission+0x11/0x20 [162016.148705] [<ffffffff811144d9>] ? vfs_write+0x189/0x1a0 [162016.148705] [<ffffffff81114b20>] ? sys_write+0x50/0x90 [162016.148705] [<ffffffff8100b402>] ? system_call_fastpath+0x16/0x1b [162335.680168] BUG: soft lockup - CPU#0 stuck for 297s! [jbd2/sda-8:1068] [162335.680173] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.680239] CPU 0: [162335.680240] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.680279] Pid: 1068, comm: jbd2/sda-8 Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162335.680283] RIP: 0010:[<ffffffff8133ee50>] [<ffffffff8133ee50>] _spin_unlock_irq+0x10/0x40 [162335.680401] RSP: 0018:ffff880028203e70 EFLAGS: 00000286 [162335.680403] RAX: 0000000000000000 RBX: ffff880028203e70 RCX: ffffffff816458e0 [162335.680404] RDX: 000000000000d1d0 RSI: ffff880028203e40 RDI: ffffffff81644940 [162335.680406] RBP: ffffffff8100bf63 R08: 00000000000022fc R09: ffff880028203be0 [162335.680410] R10: 0000000000000014 R11: 0000000000000000 R12: ffff880028203df0 [162335.680414] R13: ffffffff81644940 R14: ffff880028203ea0 R15: ffffffff81024857 [162335.680418] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 [162335.680423] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [162335.680424] CR2: 00007f2ed5713548 CR3: 00000000d101c000 CR4: 00000000000006f0 [162335.680434] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162335.680441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162335.680443] Call Trace: [162335.680446] <IRQ> [<ffffffff81060c54>] ? run_timer_softirq+0x214/0x320 [162335.680486] [<ffffffff81306fe0>] ? inet_frag_secret_rebuild+0x0/0x100 [162335.680493] [<ffffffff81059cdc>] ? __do_softirq+0xcc/0x230 [162335.680498] [<ffffffff8100c59c>] ? call_softirq+0x1c/0x30 [162335.680504] [<ffffffff8100e6a5>] ? do_softirq+0x65/0xa0 [162335.680507] [<ffffffff81059b97>] ? irq_exit+0x87/0x90 [162335.680523] [<ffffffff8102485c>] ? smp_apic_timer_interrupt+0x6c/0xa0 [162335.680530] [<ffffffff8100bf63>] ? apic_timer_interrupt+0x13/0x20 [162335.680532] <EOI> [<ffffffff8133ee96>] ? _spin_unlock_irqrestore+0x16/0x50 [162335.680551] [<ffffffffa00c2a2a>] ? scsi_dispatch_cmd+0x10a/0x2a0 [scsi_mod] [162335.680562] [<ffffffffa00ca58f>] ? scsi_request_fn+0x16f/0x4f0 [scsi_mod] [162335.680594] [<ffffffff811a3c30>] ? __generic_unplug_device+0x30/0x40 [162335.680599] [<ffffffff811a4619>] ? generic_unplug_device+0x29/0x40 [162335.680615] [<ffffffff8119fddf>] ? blk_unplug+0x2f/0xa0 [162335.680621] [<ffffffff8119fe5d>] ? blk_backing_dev_unplug+0xd/0x10 [162335.680640] [<ffffffff8113ae89>] ? sync_buffer+0x39/0x50 [162335.680657] [<ffffffff8133cc55>] ? __wait_on_bit+0x55/0x80 [162335.680662] [<ffffffff8113ae50>] ? sync_buffer+0x0/0x50 [162335.680667] [<ffffffff8113ae50>] ? sync_buffer+0x0/0x50 [162335.680672] [<ffffffff8133ccf8>] ? out_of_line_wait_on_bit+0x78/0x90 [162335.680680] [<ffffffff81071090>] ? wake_bit_function+0x0/0x30 [162335.680685] [<ffffffff8113adc6>] ? __wait_on_buffer+0x26/0x30 [162335.680768] [<ffffffffa00301a2>] ? jbd2_journal_commit_transaction+0x782/0x18e0 [jbd2] [162335.680774] [<ffffffff8133f0b7>] ? _spin_lock_irqsave+0x37/0x50 [162335.680778] [<ffffffff811c639e>] ? _raw_spin_unlock+0x5e/0xb0 [162335.680819] [<ffffffffa0036c63>] ? kjournald2+0xb3/0x230 [jbd2] [162335.680829] [<ffffffff81071050>] ? autoremove_wake_function+0x0/0x40 [162335.678699] Pid: 1046, comm: scsi_eh_0 Not tainted 2.6.32lb.29.kvmguest #1 [162335.680843] [<ffffffffa0036bb0>] ? kjournald2+0x0/0x230 [jbd2] [162335.680850] [<ffffffff81070f0e>] ? kthread+0x8e/0xa0 [162335.678699] Call Trace: [162335.680983] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162335.678699] [<ffffffff811c6517>] _raw_spin_lock+0x127/0x160 [162335.680996] [<ffffffff8100c49a>] ? child_rip+0xa/0x20 [162335.678699] [<ffffffff8133f0b7>] _spin_lock_irqsave+0x37/0x50 [162335.681003] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162335.681007] [<ffffffff81070e80>] ? kthread+0x0/0xa0 [162335.678699] [<ffffffffa011fa9f>] ata_scsi_error+0x3f/0x870 [libata] [162335.681056] [<ffffffff8100c490>] ? child_rip+0x0/0x20 [162335.678699] [<ffffffffa00c79e0>] ? scsi_error_handler+0x0/0x5b0 [scsi_mod] [162335.678699] [<ffffffffa00c7aa6>] scsi_error_handler+0xc6/0x5b0 [scsi_mod] [162335.678699] [<ffffffffa00c79e0>] ? scsi_error_handler+0x0/0x5b0 [scsi_mod] [162335.678699] [<ffffffff81070f0e>] kthread+0x8e/0xa0 [162335.678699] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162335.678699] [<ffffffff8100c49a>] child_rip+0xa/0x20 [162335.678699] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162335.678699] [<ffffffff81070e80>] ? kthread+0x0/0xa0 [162335.678699] [<ffffffff8100c490>] ? child_rip+0x0/0x20 [162335.678699] sending NMI to all CPUs: [162335.681177] NMI backtrace for cpu 0 [162335.681179] CPU 0: [162335.681181] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.681316] Pid: 1068, comm: jbd2/sda-8 Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162335.681324] RIP: 0010:[<ffffffff8103ff38>] [<ffffffff8103ff38>] find_busiest_group+0x198/0x9e0 [162335.681340] RSP: 0018:ffff880119ce59f0 EFLAGS: 00010006 [162335.681347] RAX: 0000000000000c00 RBX: 0000000000000001 RCX: 0000000000000020 [162335.681357] RDX: ffff880028240000 RSI: 0000000000000000 RDI: 0000000000000001 [162335.681367] RBP: ffff880119ce5b50 R08: 0000000000000000 R09: ffff88002824e430 [162335.681378] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880028253bc0 [162335.681388] R13: 0000000000000c00 R14: 0000000000000c00 R15: 0000000000000000 [162335.681402] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 [162335.681410] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [162335.681418] CR2: 00007f2ed5713548 CR3: 00000000d101c000 CR4: 00000000000006f0 [162335.681430] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162335.681441] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162335.681450] Call Trace: [162335.681455] <#DB[1]> <<EOE>> Pid: 1068, comm: jbd2/sda-8 Not tainted 2.6.32lb.29.kvmguest #1 [162335.681483] Call Trace: [162335.681487] <NMI> [<ffffffff81009d83>] ? show_regs+0x43/0x50 [162335.681500] [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260 [162335.681509] [<ffffffff8100cc5d>] do_nmi+0x8d/0x290 [162335.681517] [<ffffffff8133f820>] nmi+0x20/0x30 [162335.681540] [<ffffffff8103ff38>] ? find_busiest_group+0x198/0x9e0 [162335.681548] <<EOE>> [<ffffffff8133c2ce>] thread_return+0x4cd/0x73f [162335.681563] [<ffffffff8102a449>] ? kvm_clock_get_cycles+0x9/0x10 [162335.681573] [<ffffffff8133c71e>] io_schedule+0x6e/0xc0 [162335.681583] [<ffffffff8113ae8e>] sync_buffer+0x3e/0x50 [162335.681593] [<ffffffff8133cc55>] __wait_on_bit+0x55/0x80 [162335.681602] [<ffffffff8113ae50>] ? sync_buffer+0x0/0x50 [162335.681611] [<ffffffff8113ae50>] ? sync_buffer+0x0/0x50 [162335.681621] [<ffffffff8133ccf8>] out_of_line_wait_on_bit+0x78/0x90 [162335.681630] [<ffffffff81071090>] ? wake_bit_function+0x0/0x30 [162335.681639] [<ffffffff8113adc6>] __wait_on_buffer+0x26/0x30 [162335.681653] [<ffffffffa00301a2>] jbd2_journal_commit_transaction+0x782/0x18e0 [jbd2] [162335.681664] [<ffffffff8133f0b7>] ? _spin_lock_irqsave+0x37/0x50 [162335.681674] [<ffffffff811c639e>] ? _raw_spin_unlock+0x5e/0xb0 [162335.681687] [<ffffffffa0036c63>] kjournald2+0xb3/0x230 [jbd2] [162335.681695] [<ffffffff81071050>] ? autoremove_wake_function+0x0/0x40 [162335.681708] [<ffffffffa0036bb0>] ? kjournald2+0x0/0x230 [jbd2] [162335.681718] [<ffffffff81070f0e>] kthread+0x8e/0xa0 [162335.681727] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162335.681738] [<ffffffff8100c49a>] child_rip+0xa/0x20 [162335.681745] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162335.681754] [<ffffffff81070e80>] ? kthread+0x0/0xa0 [162335.681760] [<ffffffff8100c490>] ? child_rip+0x0/0x20 [162335.678699] NMI backtrace for cpu 1 [162335.681771] CPU 1: [162335.681771] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.681771] Pid: 1046, comm: scsi_eh_0 Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162335.681771] RIP: 0010:[<ffffffff81028144>] [<ffffffff81028144>] flat_send_IPI_mask+0x74/0xa0 [162335.681771] RSP: 0018:ffff88011a279d40 EFLAGS: 00010046 [162335.681771] RAX: 0000000000000c00 RBX: 0000000000000c00 RCX: 0000000000000000 [162335.681771] RDX: ffff88011a279fd8 RSI: 0000000000000002 RDI: 0000000000000300 [162335.681771] RBP: ffff88011a279d60 R08: 0000000000000000 R09: 0000000000000000 [162335.681771] R10: 000000000000000a R11: 0000000000000000 R12: 0000000000000002 [162335.681771] R13: 0000000000000082 R14: 00000000000000ff R15: 0000000000000000 [162335.681771] FS: 0000000000000000(0000) GS:ffff880028240000(0000) knlGS:0000000000000000 [162335.681771] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [162335.681771] CR2: 0000000002143858 CR3: 00000000aacaa000 CR4: 00000000000006e0 [162335.681771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162335.681771] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162335.681771] Call Trace: [162335.681771] <#DB[1]> <<EOE>> Pid: 1046, comm: scsi_eh_0 Not tainted 2.6.32lb.29.kvmguest #1 [162335.681771] Call Trace: [162335.681771] <NMI> [<ffffffff81009d83>] ? show_regs+0x43/0x50 [162335.681771] [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260 [162335.681771] [<ffffffff8100cc5d>] do_nmi+0x8d/0x290 [162335.681771] [<ffffffff8133f820>] nmi+0x20/0x30 [162335.681771] [<ffffffff81028144>] ? flat_send_IPI_mask+0x74/0xa0 [162335.681771] <<EOE>> [<ffffffff810281cd>] flat_send_IPI_all+0x5d/0x60 [162335.681771] [<ffffffff81024ccc>] arch_trigger_all_cpu_backtrace+0x3c/0x70 [162335.681771] [<ffffffff811c651c>] _raw_spin_lock+0x12c/0x160 [162335.681771] [<ffffffff8133f0b7>] _spin_lock_irqsave+0x37/0x50 [162335.681771] [<ffffffffa011fa9f>] ata_scsi_error+0x3f/0x870 [libata] [162335.681771] [<ffffffffa00c79e0>] ? scsi_error_handler+0x0/0x5b0 [scsi_mod] [162335.681771] [<ffffffffa00c7aa6>] scsi_error_handler+0xc6/0x5b0 [scsi_mod] [162335.681771] [<ffffffffa00c79e0>] ? scsi_error_handler+0x0/0x5b0 [scsi_mod] [162335.681771] [<ffffffff81070f0e>] kthread+0x8e/0xa0 [162335.681771] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162335.681771] [<ffffffff8100c49a>] child_rip+0xa/0x20 [162335.681771] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162335.681771] [<ffffffff81070e80>] ? kthread+0x0/0xa0 [162335.681771] [<ffffffff8100c490>] ? child_rip+0x0/0x20 [162335.682115] NMI backtrace for cpu 4 [162335.682131] CPU 4: [162335.682142] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.682443] Pid: 8137, comm: sadc Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162335.682470] RIP: 0010:[<ffffffff810a86ac>] [<ffffffff810a86ac>] kstat_irqs_cpu+0xc/0x30 [162335.682536] RSP: 0018:ffff8800aaea9c60 EFLAGS: 00010282 [162335.682557] RAX: ffffffff81510540 RBX: 0000000000000006 RCX: 0000000000000000 [162335.682582] RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000443 [162335.682607] RBP: ffff8800aaea9c68 R08: 0000000000000000 R09: ffffffff81571230 [162335.682632] R10: 0000000000000022 R11: 0000000000000000 R12: 0000000000000006 [162335.682655] R13: 000000000000e4a0 R14: 0000000000000006 R15: 0000000000000000 [162335.682676] FS: 00007fe70e3846e0(0000) GS:ffff880028300000(0000) knlGS:0000000000000000 [162335.682703] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [162335.682724] CR2: 00007fe70e39f000 CR3: 00000000ce68d000 CR4: 00000000000006e0 [162335.682753] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162335.682781] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162335.682803] Call Trace: [162335.682814] <#DB[1]> <<EOE>> Pid: 8137, comm: sadc Not tainted 2.6.32lb.29.kvmguest #1 [162335.682858] Call Trace: [162335.682868] <NMI> [<ffffffff81009d83>] ? show_regs+0x43/0x50 [162335.682890] [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260 [162335.682914] [<ffffffff8100cc5d>] do_nmi+0x8d/0x290 [162335.682928] [<ffffffff8133f820>] nmi+0x20/0x30 [162335.682947] [<ffffffff810a86ac>] ? kstat_irqs_cpu+0xc/0x30 [162335.682967] <<EOE>> [<ffffffff811717c0>] show_stat+0x190/0x610 [162335.683045] [<ffffffff8107f42f>] ? debug_mutex_add_waiter+0x3f/0x70 [162335.683074] [<ffffffff8133d6b3>] ? __mutex_lock_slowpath+0x263/0x340 [162335.683097] [<ffffffff81130e90>] ? seq_read+0x0/0x3c0 [162335.683119] [<ffffffff81130f4d>] seq_read+0xbd/0x3c0 [162335.683141] [<ffffffff81130e90>] ? seq_read+0x0/0x3c0 [162335.683162] [<ffffffff8116953a>] proc_reg_read+0x7a/0xb0 [162335.683190] [<ffffffff811145b4>] vfs_read+0xc4/0x190 [162335.683209] [<ffffffff81114a90>] sys_read+0x50/0x90 [162335.683231] [<ffffffff8100b402>] system_call_fastpath+0x16/0x1b [162335.683255] NMI backtrace for cpu 6 [162335.683262] CPU 6: [162335.683267] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.683407] Pid: 9310, comm: sshd Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162335.683415] RIP: 0033:[<00007f52fa16cce0>] [<00007f52fa16cce0>] 0x7f52fa16cce0 [162335.683419] RSP: 002b:00007fffcf542598 EFLAGS: 00010246 [162335.683419] RAX: 00007fffcf5425b0 RBX: 000000000000000f RCX: 0000000000000010 [162335.683419] RDX: 00007f52fbb02cc0 RSI: 00007fffcf5425b0 RDI: 00007f52fbb02db4 [162335.683419] RBP: 00007f52fbafce70 R08: 00007f52fbafce70 R09: 00007f52fb4a7310 [162335.683419] R10: 26f5a1b2c4cc4703 R11: 0000000000000246 R12: 00007f52fbaffea0 [162335.683419] R13: 0000000000000000 R14: 00007f52fbb02cc0 R15: 00007f52fbb02dc3 [162335.683419] FS: 00007f52fb21b730(0000) GS:ffff880028380000(0000) knlGS:0000000000000000 [162335.683419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [162335.683419] CR2: 00007f2ed8640e80 CR3: 00000000d2c71000 CR4: 00000000000006e0 [162335.683419] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162335.683419] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162335.683419] Call Trace: [162335.683419] <#DB[1]> <<EOE>> Pid: 9310, comm: sshd Not tainted 2.6.32lb.29.kvmguest #1 [162335.683419] Call Trace: [162335.683419] <NMI> [<ffffffff81009d83>] ? show_regs+0x43/0x50 [162335.683419] [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260 [162335.683419] [<ffffffff8100cc5d>] do_nmi+0x8d/0x290 [162335.683419] [<ffffffff8133f820>] nmi+0x20/0x30 [162335.683419] <<EOE>> [162335.683623] NMI backtrace for cpu 2 [162335.683627] CPU 2: [162335.683630] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.683736] Pid: 0, comm: swapper Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162335.683743] RIP: 0010:[<ffffffff8102a496>] [<ffffffff8102a496>] native_safe_halt+0x6/0x10 [162335.683753] RSP: 0018:ffff88011f8a7ee8 EFLAGS: 00010246 [162335.683759] RAX: ffff88011f8a7fd8 RBX: ffff88011f8a7fd8 RCX: 00000000ffffffff [162335.683763] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001 [162335.683768] RBP: ffff88011f8a7ee8 R08: 0000000000000000 R09: 0000000000000000 [162335.683776] R10: 0000000000000802 R11: 0000000000000000 R12: ffffffff81571238 [162335.683781] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [162335.683787] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000 [162335.683792] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [162335.683802] CR2: 00007f5e4aa82a30 CR3: 000000011a7b5000 CR4: 00000000000006e0 [162335.683811] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162335.683822] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162335.683826] Call Trace: [162335.683831] <#DB[1]> <<EOE>> Pid: 0, comm: swapper Not tainted 2.6.32lb.29.kvmguest #1 [162335.683855] Call Trace: [162335.683858] <NMI> [<ffffffff81009d83>] ? show_regs+0x43/0x50 [162335.683869] [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260 [162335.683876] [<ffffffff8100cc5d>] do_nmi+0x8d/0x290 [162335.683884] [<ffffffff8133f820>] nmi+0x20/0x30 [162335.683892] [<ffffffff8102a496>] ? native_safe_halt+0x6/0x10 [162335.683910] <<EOE>> [<ffffffff810136c8>] default_idle+0x38/0xc0 [162335.683952] [<ffffffff8100a215>] cpu_idle+0x55/0x90 [162335.683959] [<ffffffff81337ea3>] start_secondary+0x183/0x1e0 [162335.682548] NMI backtrace for cpu 3 [162335.682548] CPU 3: [162335.682548] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.682548] Pid: 0, comm: swapper Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162335.682548] RIP: 0010:[<ffffffff8102a496>] [<ffffffff8102a496>] native_safe_halt+0x6/0x10 [162335.682548] RSP: 0018:ffff88011f8e1ee8 EFLAGS: 00010246 [162335.682548] RAX: ffff88011f8e1fd8 RBX: ffff88011f8e1fd8 RCX: 00000000ffffffff [162335.682548] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001 [162335.682548] RBP: ffff88011f8e1ee8 R08: 0000000000000000 R09: 0000000000000000 [162335.682548] R10: 0000000000000800 R11: 0000000000000000 R12: ffffffff81571238 [162335.682548] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [162335.682548] FS: 0000000000000000(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000 [162335.682548] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [162335.682548] CR2: 00007f5e479c0040 CR3: 000000009e845000 CR4: 00000000000006e0 [162335.682548] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162335.682548] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162335.682548] Call Trace: [162335.682548] <#DB[1]> <<EOE>> Pid: 0, comm: swapper Not tainted 2.6.32lb.29.kvmguest #1 [162335.682548] Call Trace: [162335.682548] <NMI> [<ffffffff81009d83>] ? show_regs+0x43/0x50 [162335.682548] [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260 [162335.682548] [<ffffffff8100cc5d>] do_nmi+0x8d/0x290 [162335.682548] [<ffffffff8133f820>] nmi+0x20/0x30 [162335.682548] [<ffffffff8102a496>] ? native_safe_halt+0x6/0x10 [162335.682548] <<EOE>> [<ffffffff810136c8>] default_idle+0x38/0xc0 [162335.682548] [<ffffffff8100a215>] cpu_idle+0x55/0x90 [162335.682548] [<ffffffff81337ea3>] start_secondary+0x183/0x1e0 [162335.684274] NMI backtrace for cpu 7 [162335.684274] CPU 7: [162335.684274] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.684274] Pid: 0, comm: swapper Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162335.684274] RIP: 0010:[<ffffffff81108d35>] [<ffffffff81108d35>] kfree+0x75/0x170 [162335.684274] RSP: 0018:ffff8800283c3c20 EFLAGS: 00010282 [162335.684274] RAX: 0000000000000282 RBX: ffffea0003448800 RCX: ffff88011a1bd000 [162335.684274] RDX: ffffea0000000000 RSI: 0000000000000205 RDI: ffff8800d1225c00 [162335.684274] RBP: ffff8800283c3c50 R08: 0000000000000000 R09: 0000000000000002 [162335.684274] R10: 0000000000000020 R11: 0000000000000000 R12: 0000000000000000 [162335.684274] R13: ffffffff8151d5f0 R14: ffff8800d1225c00 R15: ffffffff8129d668 [162335.684274] FS: 0000000000000000(0000) GS:ffff8800283c0000(0000) knlGS:0000000000000000 [162335.684274] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [162335.684274] CR2: 00007f91d1931408 CR3: 00000000d36b2000 CR4: 00000000000006e0 [162335.684274] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162335.684274] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162335.684274] Call Trace: [162335.684274] <#DB[1]> <<EOE>> Pid: 0, comm: swapper Not tainted 2.6.32lb.29.kvmguest #1 [162335.684274] Call Trace: [162335.684274] <NMI> [<ffffffff81009d83>] ? show_regs+0x43/0x50 [162335.684274] [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260 [162335.684274] [<ffffffff8129d668>] ? skb_release_data+0x68/0xd0 [162335.684274] [<ffffffff8100cc5d>] do_nmi+0x8d/0x290 [162335.684274] [<ffffffff8133f820>] nmi+0x20/0x30 [162335.684274] [<ffffffff8129d668>] ? skb_release_data+0x68/0xd0 [162335.684274] [<ffffffff81108d35>] ? kfree+0x75/0x170 [162335.684274] <<EOE>> <IRQ> [<ffffffff8129d668>] skb_release_data+0x68/0xd0 [162335.684274] [<ffffffff8129d269>] skb_release_all+0x19/0x20 [162335.684274] [<ffffffff8129d2c1>] __kfree_skb+0x11/0xa0 [162335.684274] [<ffffffff8129d369>] consume_skb+0x19/0x40 [162335.684274] [<ffffffff812f83d8>] arp_process+0xe8/0x6d0 [162335.684274] [<ffffffff8102b164>] ? pvclock_clocksource_read+0x44/0xb0 [162335.684274] [<ffffffff812f8a8f>] arp_rcv+0xcf/0x150 [162335.684274] [<ffffffff812a5ff1>] netif_receive_skb+0x3b1/0x620 [162335.684274] [<ffffffffa01bdeb6>] virtnet_poll+0x536/0x690 [virtio_net] [162335.684274] [<ffffffff812a94a2>] net_rx_action+0x162/0x280 [162335.684274] [<ffffffff81059cdc>] __do_softirq+0xcc/0x230 [162335.684274] [<ffffffff8100c59c>] call_softirq+0x1c/0x30 [162335.684274] [<ffffffff8100e6a5>] do_softirq+0x65/0xa0 [162335.684274] [<ffffffff81059b97>] irq_exit+0x87/0x90 [162335.684274] [<ffffffff8100da77>] do_IRQ+0x77/0xf0 [162335.684274] [<ffffffff8100bd93>] ret_from_intr+0x0/0x11 [162335.684274] <EOI> [<ffffffff8102a496>] ? native_safe_halt+0x6/0x10 [162335.684274] [<ffffffff810136c8>] ? default_idle+0x38/0xc0 [162335.684274] [<ffffffff8100a215>] ? cpu_idle+0x55/0x90 [162335.684274] [<ffffffff81337ea3>] ? start_secondary+0x183/0x1e0 [162002.937850] NMI backtrace for cpu 5 [162002.937850] CPU 5: [162002.937850] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162002.937850] Pid: 19, comm: ksoftirqd/5 Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162002.937850] RIP: 0010:[<ffffffffa001cb1d>] [<ffffffffa001cb1d>] uhci_check_ports+0x4d/0x170 [uhci_hcd] [162002.937850] RSP: 0018:ffff880028343df8 EFLAGS: 00010006 [162002.937850] RAX: 0000000000000080 RBX: 000000000000c032 RCX: 0000000000000000 [162002.937850] RDX: 000000000000c032 RSI: 0000000000000101 RDI: ffff88011a211550 [162002.937850] RBP: ffff880028343e38 R08: 0000000000000000 R09: dead000000200200 [162002.937850] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [162002.937850] R13: ffff88011a211550 R14: 000000000000c032 R15: 0000000000000001 [162002.937850] FS: 0000000000000000(0000) GS:ffff880028340000(0000) knlGS:0000000000000000 [162002.937850] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [162002.937850] CR2: 00007f28972c4980 CR3: 00000000aacc1000 CR4: 00000000000006e0 [162002.937850] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162002.937850] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162002.937850] Call Trace: [162002.937850] <#DB[1]> <<EOE>> Pid: 19, comm: ksoftirqd/5 Not tainted 2.6.32lb.29.kvmguest #1 [162002.937850] Call Trace: [162002.937850] <NMI> [<ffffffff81009d83>] ? show_regs+0x43/0x50 [162002.937850] [<ffffffff81024f4a>] nmi_watchdog_tick+0x17a/0x260 [162002.937850] [<ffffffff8100cc5d>] do_nmi+0x8d/0x290 [162002.937850] [<ffffffff8133f820>] nmi+0x20/0x30 [162002.937850] [<ffffffffa001cb1d>] ? uhci_check_ports+0x4d/0x170 [uhci_hcd] [162002.937850] <<EOE>> <IRQ> [<ffffffffa001dd58>] uhci_hub_status_data+0x88/0x1c0 [uhci_hcd] [162002.937850] [<ffffffff811c639e>] ? _raw_spin_unlock+0x5e/0xb0 [162002.937850] [<ffffffff8124c387>] usb_hcd_poll_rh_status+0x57/0x180 [162002.937850] [<ffffffff8124c4b9>] rh_timer_func+0x9/0x10 [162002.937850] [<ffffffff81060b9d>] run_timer_softirq+0x15d/0x320 [162002.937850] [<ffffffff8124c4b0>] ? rh_timer_func+0x0/0x10 [162002.937850] [<ffffffff81059cdc>] __do_softirq+0xcc/0x230 [162002.937850] [<ffffffff8100c59c>] call_softirq+0x1c/0x30 [162002.937850] <EOI> [<ffffffff8100e6a5>] do_softirq+0x65/0xa0 [162002.937850] [<ffffffff8105a819>] ksoftirqd+0x79/0x1a0 [162002.937850] [<ffffffff8105a7a0>] ? ksoftirqd+0x0/0x1a0 [162002.937850] [<ffffffff81070f0e>] kthread+0x8e/0xa0 [162002.937850] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162002.937850] [<ffffffff8100c49a>] child_rip+0xa/0x20 [162002.937850] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162002.937850] [<ffffffff81070e80>] ? kthread+0x0/0xa0 [162002.937850] [<ffffffff8100c490>] ? child_rip+0x0/0x20 [162335.694727] sd 0:0:0:0: [sda] Unhandled error code [162335.694754] sd 0:0:0:0: [sda] [162335.694768] BUG: soft lockup - CPU#5 stuck for 310s! [ksoftirqd/5:19] [162335.694779] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcpResult: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT [162335.694849] bnx2i [162335.694863] sd 0:0:0:0: [sda] CDB: Write(10) cnic uio cxgb3i: 2a cxgb3 mdio 00 00 3e a4 90 00 00 18 00 [162335.695060] libiscsi_tcp [162335.695067] end_request: I/O error, dev sda, sector 4105360 [162335.695143] libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.696012] CPU 5: [162335.696127] Modules linked in: bluetooth rfkill be2iscsi iscsi_tcp bnx2i cnic uio cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi dm_snapshot dm_zero netconsole ipv6 nfs nfs_acl auth_rpcgss lockd sunrpc dm_mirror dm_region_hash dm_log dm_mod video backlight output sbs sbshc fan container battery ac thermal processor button virtio_balloon pcspkr i2c_piix4 thermal_sys i2c_core sg virtio_blk virtio_net virtio_pci virtio_ring virtio 8139cp mii pata_acpi ide_pci_generic ide_core ata_piix ata_generic libata sd_mod scsi_mod crc_t10dif ext4 jbd2 crc16 uhci_hcd ohci_hcd ehci_hcd [last unloaded: scsi_wait_scan] [162335.696127] Pid: 19, comm: ksoftirqd/5 Not tainted 2.6.32lb.29.kvmguest #1 Bochs [162335.696127] RIP: 0010:[<ffffffff8133ee50>] [<ffffffff8133ee50>] _spin_unlock_irq+0x10/0x40 [162335.696127] RSP: 0018:ffff880028343ed8 EFLAGS: 00000206 [162335.696127] RAX: 0000000000000005 RBX: ffff880028343ed8 RCX: ffff88011f900fd0 [162335.696127] RDX: 0000000000005554 RSI: ffff880028343ea8 RDI: ffff88011f900000 [162335.696127] RBP: ffffffff8100bf63 R08: 0000000000001e08 R09: 0000000000000000 [162335.696127] R10: 000000000000012c R11: 0000000000000000 R12: ffff880028343e50 [162335.696127] R13: ffff88011f900000 R14: ffff880028343f08 R15: ffffffff81024857 [162335.696127] FS: 0000000000000000(0000) GS:ffff880028340000(0000) knlGS:0000000000000000 [162335.696127] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [162335.696127] CR2: 00007f28972c4980 CR3: 00000000aacc1000 CR4: 00000000000006e0 [162335.696127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [162335.696127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [162335.696127] Call Trace: [162335.696127] <IRQ> [<ffffffff8133ee49>] ? _spin_unlock_irq+0x9/0x40 [162335.696127] [<ffffffff81060c54>] ? run_timer_softirq+0x214/0x320 [162335.696127] [<ffffffff8101d410>] ? mcheck_timer+0x0/0x120 [162335.696127] [<ffffffff81059cdc>] ? __do_softirq+0xcc/0x230 [162335.696127] [<ffffffff8100c59c>] ? call_softirq+0x1c/0x30 [162335.696127] <EOI> [<ffffffff8100e6a5>] ? do_softirq+0x65/0xa0 [162335.696127] [<ffffffff8105a819>] ? ksoftirqd+0x79/0x1a0 [162335.696127] [<ffffffff8105a7a0>] ? ksoftirqd+0x0/0x1a0 [162335.696127] [<ffffffff81070f0e>] ? kthread+0x8e/0xa0 [162335.696793] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162335.696793] [<ffffffff8100c49a>] ? child_rip+0xa/0x20 [162335.696793] [<ffffffff8158a140>] ? early_idt_handler+0x0/0x71 [162335.696793] [<ffffffff81070e80>] ? kthread+0x0/0xa0 [162335.696793] [<ffffffff8100c490>] ? child_rip+0x0/0x20 [162335.697039] Aborting journal on device sda-8. [162335.697082] EXT4-fs error (device sda): ext4_journal_start_sb: [162335.697146] EXT4-fs error (device sda) in ext4_reserve_inode_write: Journal has aborted [162335.697226] EXT4-fs error (device sda) in ext4_reserve_inode_write: Journal has aborted [162335.702749] Detected aborted journal [162335.702752] EXT4-fs (sda): Remounting filesystem read-only [162335.705754] EXT4-fs error (device sda) in ext4_dirty_inode: Journal has aborted [162335.707725] EXT4-fs (sda): ext4_da_writepages: jbd2_start: 1024 pages, ino 32392; err -30 I'm using virtio network channel, and on one of the guests (the one with aborted ext4) I use it also for one of virtual disks. One more interesting thing, I can't reproduce this immediately after guest boot, but for example second day after boot, I can reproduce this. perhaps this can suggest something? Could somebody please help me to find and possibly fix this bug? If needed, I can provide further debugging information, bisect etc... Thanks a lot in advance! nik -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis@xxxxxxxxxxx ------------------------------------- -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
