On Sun, Sep 09, 2012 at 10:22:02AM +0100, Brian Candler wrote:
> And I've also run this workload on another test box here over a week and not
> been able to reproduce the problem :-(
I did a full update on the test box (kernel now 3.2.0-30-generic). On
subsequent testing the machine does crash in a different way, but I think
this is probably unrelated to my original problem.
After last set of testing I noticed disk /dev/sdd was misbehaving, so I
rebuilt the RAID0 array without it, and a new XFS filesystem on the
remaining disks of course. sdd is not mounted, but I've not yet physically
removed it.
When running a test, the machine locked up in a really bad way - I couldn't
ssh into it, but I did get a console capture (see attached). This is
repeatable, I've done it three times now.
After a forced reset, I got some messages from /var/log/syslog (below). The
errors occur nearly 30 seconds after an error on sdd, and I see
"scsi_remove_target" in the backtrace.
However, a misbehaving disk ought to be a fairly regular occurrence in a
RAID array, and I'm sure it shouldn't bring the system to its knees
(especially a disk which is not being used)
Maybe I'm coming round to the view that a hardware RAID controller, which
can isolate the system from the quirks of bad drives, might be a good idea
after all - even though having a single virtual disk makes it hard to do
SMART queries against individual drives.
This of course assumes the RAID controller manufacturer tests the various
error scenarios better than the Linux kernel drivers do :-(
So I'd still be grateful for a recommendation of 24-port SATA RAID
controller with good Linux support.
Regards,
Brian.
Sep 9 20:37:59 dev-storage1 kernel: [ 838.093290] sd 4:0:2:0: [sdd] Synchronizing SCSI cache
Sep 9 20:37:59 dev-storage1 kernel: [ 838.093350] sd 4:0:2:0: [sdd] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Sep 9 20:38:27 dev-storage1 kernel: [ 865.824696] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u:12:1602]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825049] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825076] CPU 1
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825077] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825095]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825097] Pid: 1602, comm: kworker/u:12 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825099] RIP: 0010:[<ffffffff8165a629>] [<ffffffff8165a629>] _raw_spin_unlock_irqrestore+0x19/0x30
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825104] RSP: 0018:ffff880210d55b28 EFLAGS: 00000282
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825105] RAX: 0000000000000282 RBX: 0000000000000004 RCX: 0000000000000004
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825107] RDX: ffff8802050d8818 RSI: 0000000000000282 RDI: 0000000000000282
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825108] RBP: ffff880210d55b30 R08: ffffffff81cddaa0 R09: 0000000000000100
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825109] R10: 0000000000000004 R11: 0000000000000000 R12: ffff88020ee0a150
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825110] R13: ffff880210d55b30 R14: ffff88020ee0a144 R15: ffff88020ee09d60
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825112] FS: 0000000000000000(0000) GS:ffff88021fc80000(0000) knlGS:0000000000000000
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825113] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825114] CR2: 00007f70a894c000 CR3: 0000000001c05000 CR4: 00000000000406e0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825117] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825119] Process kworker/u:12 (pid: 1602, threadinfo ffff880210d54000, task ffff8802050aae00)
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825120] Stack:
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825215] ffff8802050d8800 ffff880210d55b60 ffffffff8142d1e7 ffff8802050d8400
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825217] ffff8802050d8400 ffff88020dacc010 ffff88020dacc000 ffff880210d55b90
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825219] ffffffff8142f3c9 ffff8802050d8400 ffff880211940000 ffff880211eaa4c8
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825222] Call Trace:
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825347] [<ffffffff8142d1e7>] scsi_target_reap+0x77/0xf0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825349] [<ffffffff8142f3c9>] scsi_remove_target+0xc9/0xe0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825354] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825357] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825360] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825364] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825369] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825373] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825377] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825381] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825385] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825388] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825391] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825394] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825397] [<ffffffff81085814>] worker_thread+0x164/0x370
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825399] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825401] [<ffffffff8108a03c>] kthread+0x8c/0xa0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825403] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825405] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825407] [<ffffffff81664b70>] ? gs_change+0x13/0x13
Sep 9 20:38:27 dev-storage1 kernel: [ 865.825407] Code: 48 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 7e 36 9e ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826836] Call Trace:
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826838] [<ffffffff8142d1e7>] scsi_target_reap+0x77/0xf0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826839] [<ffffffff8142f3c9>] scsi_remove_target+0xc9/0xe0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826842] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826845] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826848] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826850] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826854] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826858] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826861] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826865] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826868] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826870] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826874] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826876] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826878] [<ffffffff81085814>] worker_thread+0x164/0x370
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826880] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826882] [<ffffffff8108a03c>] kthread+0x8c/0xa0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826884] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826885] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
Sep 9 20:38:27 dev-storage1 kernel: [ 865.826887] [<ffffffff81664b70>] ? gs_change+0x13/0x13
Sep 9 20:38:55 dev-storage1 kernel: [ 893.744859] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u:12:1602]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745212] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745239] CPU 1
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745239] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745258]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745260] Pid: 1602, comm: kworker/u:12 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745262] RIP: 0010:[<ffffffff8165a629>] [<ffffffff8165a629>] _raw_spin_unlock_irqrestore+0x19/0x30
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745268] RSP: 0018:ffff880210d55b58 EFLAGS: 00000286
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745269] RAX: 0000000000000286 RBX: ffffffff81cddaa0 RCX: 0000000000000004
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745270] RDX: ffff8802050d8808 RSI: 0000000000000286 RDI: 0000000000000286
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745271] RBP: ffff880210d55b60 R08: ffffffff81cddaa0 R09: 0000000000000100
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745273] R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000004
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745274] R13: 0000000000000000 R14: 0000000000000004 R15: ffff880210d55b30
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745276] FS: 0000000000000000(0000) GS:ffff88021fc80000(0000) knlGS:0000000000000000
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745277] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745278] CR2: 00007f70a894c000 CR3: 0000000001c05000 CR4: 00000000000406e0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745279] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745281] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745282] Process kworker/u:12 (pid: 1602, threadinfo ffff880210d54000, task ffff8802050aae00)
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745283] Stack:
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745378] ffff8802050d8400 ffff880210d55b90 ffffffff8142f3b4 ffff8802050d8400
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745381] ffff880211940000 ffff880211eaa4c8 ffff880211940000 ffff880210d55bb0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745383] ffffffffa00a2e65 ffff8802050d8400 ffff880211eaa480 ffff880210d55bd0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745385] Call Trace:
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745510] [<ffffffff8142f3b4>] scsi_remove_target+0xb4/0xe0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745514] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745518] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745521] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745524] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745530] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745534] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745538] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745541] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745545] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745548] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745551] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745555] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745557] [<ffffffff81085814>] worker_thread+0x164/0x370
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745560] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745562] [<ffffffff8108a03c>] kthread+0x8c/0xa0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745564] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745566] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745568] [<ffffffff81664b70>] ? gs_change+0x13/0x13
Sep 9 20:38:55 dev-storage1 kernel: [ 893.745569] Code: 48 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 7e 36 9e ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00
Sep 9 20:38:55 dev-storage1 kernel: [ 893.746982] Call Trace:
Sep 9 20:38:55 dev-storage1 kernel: [ 893.746984] [<ffffffff8142f3b4>] scsi_remove_target+0xb4/0xe0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.746987] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.746990] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.746993] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.746995] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
Sep 9 20:38:55 dev-storage1 kernel: [ 893.746998] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747002] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747006] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747009] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747013] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747015] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747018] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747020] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747022] [<ffffffff81085814>] worker_thread+0x164/0x370
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747024] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747026] [<ffffffff8108a03c>] kthread+0x8c/0xa0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747028] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747030] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
Sep 9 20:38:55 dev-storage1 kernel: [ 893.747032] [<ffffffff81664b70>] ? gs_change+0x13/0x13
Sep 9 20:38:59 dev-storage1 kernel: [ 897.928887] INFO: rcu_sched detected stall on CPU 1 (t=15000 jiffies)
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929229] sending NMI to all CPUs:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929230] NMI backtrace for cpu 1
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929232] CPU 1
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929233] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929268]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929270] Pid: 1602, comm: kworker/u:12 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929274] RIP: 0010:[<ffffffff8101b316>] [<ffffffff8101b316>] native_read_tsc+0x6/0x20
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929280] RSP: 0018:ffff88021fc83d68 EFLAGS: 00000002
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929281] RAX: 00000000a61a449e RBX: 0000000000002710 RCX: 00000000a61a4476
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929283] RDX: 0000000000000313 RSI: 0000000000000100 RDI: 0000000000335923
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929285] RBP: ffff88021fc83d68 R08: 000000000000000a R09: 0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929286] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000a61a4476
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929288] R13: 0000000000000001 R14: 0000000000335923 R15: 0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929290] FS: 0000000000000000(0000) GS:ffff88021fc80000(0000) knlGS:0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929292] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929294] CR2: 00007f70a894c000 CR3: 0000000001c05000 CR4: 00000000000406e0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929296] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929297] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929299] Process kworker/u:12 (pid: 1602, threadinfo ffff880210d54000, task ffff8802050aae00)
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929301] Stack:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929396] ffff88021fc83d98 ffffffff81317bda 0000000000002710 ffffffff81c31b00
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929399] ffffffff81c31c00 ffff88021fc8e7c0 ffff88021fc83da8 ffffffff81317b01
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929402] ffff88021fc83dc8 ffffffff810327b2 ffffffff81cddaa0 ffffffff81c31b00
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929406] Call Trace:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929523] <IRQ>
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929644] [<ffffffff81317bda>] delay_tsc+0x4a/0x80
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929647] [<ffffffff81317b01>] __const_udelay+0x31/0x40
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929651] [<ffffffff810327b2>] arch_trigger_all_cpu_backtrace+0x72/0xa0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929655] [<ffffffff810e04e7>] check_cpu_stall.isra.36+0x97/0xf0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929657] [<ffffffff810e0578>] __rcu_pending+0x38/0x1b0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929660] [<ffffffff810e09e9>] rcu_check_callbacks+0x79/0x1e0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929664] [<ffffffff81078528>] update_process_times+0x48/0x90
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929667] [<ffffffff8109c054>] tick_sched_timer+0x64/0xc0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929671] [<ffffffff8108e738>] __run_hrtimer+0x78/0x1f0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929673] [<ffffffff8109bff0>] ? tick_nohz_handler+0x100/0x100
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929677] [<ffffffff8106e57d>] ? __do_softirq+0xfd/0x210
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929679] [<ffffffff8108f067>] hrtimer_interrupt+0xf7/0x230
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929683] [<ffffffff81664c6c>] ? call_softirq+0x1c/0x30
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929686] [<ffffffff81665609>] smp_apic_timer_interrupt+0x69/0x99
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929689] [<ffffffff816634de>] apic_timer_interrupt+0x6e/0x80
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929690] <EOI>
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929799] [<ffffffff8165a629>] ? _raw_spin_unlock_irqrestore+0x19/0x30
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929802] [<ffffffff8142f3b4>] scsi_remove_target+0xb4/0xe0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929807] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929811] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929815] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929819] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929825] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929831] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929836] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929841] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929846] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929849] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929853] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929857] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929860] [<ffffffff81085814>] worker_thread+0x164/0x370
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929863] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929866] [<ffffffff8108a03c>] kthread+0x8c/0xa0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929868] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929871] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929873] [<ffffffff81664b70>] ? gs_change+0x13/0x13
Sep 9 20:38:59 dev-storage1 kernel: [ 897.929875] Code: c3 90 90 90 90 55 89 f8 48 89 e5 e6 70 e4 71 5d c3 0f 1f 40 00 55 89 f0 48 89 e5 e6 70 89 f8 e6 71 5d c3 66 90 55 48 89 e5 0f 31 <89> c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 66 66 66 2e 0f 1f 84
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931273] Call Trace:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931274] <IRQ> [<ffffffff81317bda>] delay_tsc+0x4a/0x80
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931279] [<ffffffff81317b01>] __const_udelay+0x31/0x40
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931281] [<ffffffff810327b2>] arch_trigger_all_cpu_backtrace+0x72/0xa0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931284] [<ffffffff810e04e7>] check_cpu_stall.isra.36+0x97/0xf0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931287] [<ffffffff810e0578>] __rcu_pending+0x38/0x1b0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931290] [<ffffffff810e09e9>] rcu_check_callbacks+0x79/0x1e0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931292] [<ffffffff81078528>] update_process_times+0x48/0x90
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931295] [<ffffffff8109c054>] tick_sched_timer+0x64/0xc0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931297] [<ffffffff8108e738>] __run_hrtimer+0x78/0x1f0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931300] [<ffffffff8109bff0>] ? tick_nohz_handler+0x100/0x100
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931302] [<ffffffff8106e57d>] ? __do_softirq+0xfd/0x210
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931305] [<ffffffff8108f067>] hrtimer_interrupt+0xf7/0x230
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931308] [<ffffffff81664c6c>] ? call_softirq+0x1c/0x30
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931310] [<ffffffff81665609>] smp_apic_timer_interrupt+0x69/0x99
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931313] [<ffffffff816634de>] apic_timer_interrupt+0x6e/0x80
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931314] <EOI> [<ffffffff8165a629>] ? _raw_spin_unlock_irqrestore+0x19/0x30
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931318] [<ffffffff8142f3b4>] scsi_remove_target+0xb4/0xe0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931322] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931326] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931330] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931333] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931337] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931342] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931347] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931351] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931356] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931358] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931363] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931366] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931369] [<ffffffff81085814>] worker_thread+0x164/0x370
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931371] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931374] [<ffffffff8108a03c>] kthread+0x8c/0xa0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931376] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931379] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931381] [<ffffffff81664b70>] ? gs_change+0x13/0x13
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931383] NMI backtrace for cpu 3
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931384] CPU 3
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931385] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931414]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931416] Pid: 0, comm: swapper/3 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931419] RIP: 0010:[<ffffffff8136ba5f>] [<ffffffff8136ba5f>] intel_idle+0xbf/0x150
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931423] RSP: 0018:ffff8802121c5e38 EFLAGS: 00000046
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931424] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931426] RDX: 0000000000000000 RSI: ffff8802121c5fd8 RDI: ffffffff81c12e00
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931428] RBP: ffff8802121c5e98 R08: 00000000fffffffc R09: 0000000000000f9e
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931430] R10: 0000000000001f21 R11: 0000000000000000 R12: 0000000000000003
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931431] R13: ffff88021fd9ac70 R14: 0000000000000020 R15: 12b249087f5451ca
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931433] FS: 0000000000000000(0000) GS:ffff88021fd80000(0000) knlGS:0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931435] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931437] CR2: 00007f70a894c000 CR3: 0000000001c05000 CR4: 00000000000406e0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931439] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931440] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931442] Process swapper/3 (pid: 0, threadinfo ffff8802121c4000, task ffff8802121a1700)
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931444] Stack:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931539] 0000000000000000 00000000003ce5da ffff8802121c5e98 ffffffff81506fb7
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931543] 0000000000000000 00000003003ce5da 0000000000000000 ffff88021fd9ac70
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931546] 00000000fffffff0 0000000000000003 0000000000000003 0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931549] Call Trace:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931671] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931673] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931677] [<ffffffff8101222a>] cpu_idle+0xca/0x120
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931680] [<ffffffff81637fed>] start_secondary+0xd9/0xdb
Sep 9 20:38:59 dev-storage1 kernel: [ 897.931682] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> cc 93 d2 ff 4c 29 f8 48 89 c7 e8 b1 11 d0 ff 4c 69 f0 40 42
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933089] Call Trace:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933091] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933093] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933096] [<ffffffff8101222a>] cpu_idle+0xca/0x120
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933098] [<ffffffff81637fed>] start_secondary+0xd9/0xdb
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933100] NMI backtrace for cpu 0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933102] CPU 0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933103] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933160]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933162] Pid: 0, comm: swapper/0 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933167] RIP: 0010:[<ffffffff8136ba5f>] [<ffffffff8136ba5f>] intel_idle+0xbf/0x150
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933173] RSP: 0018:ffffffff81c01e18 EFLAGS: 00000046
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933175] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933178] RDX: 0000000000000000 RSI: ffffffff81c01fd8 RDI: ffffffff81c12e00
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933181] RBP: ffffffff81c01e78 R08: 00000000fffffffd R09: 0000000000000f99
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933184] R10: 0000000000001eb7 R11: 0000000000000000 R12: 0000000000000002
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933186] R13: ffff88021fc1ac70 R14: 0000000000000010 R15: 12b249087f546234
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933189] FS: 0000000000000000(0000) GS:ffff88021fc00000(0000) knlGS:0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933192] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933195] CR2: 00007ffb58064000 CR3: 0000000001c05000 CR4: 00000000000406f0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933198] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933201] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933204] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c0d020)
Sep 9 20:38:59 dev-storage1 kernel: [ 897.933206] Stack:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.940315] 0000000000000000 00000000003cd540 ffffffff81c01e78 ffffffff81506fb7
Sep 9 20:38:59 dev-storage1 kernel: [ 897.940319] 0000000000000000 00000000003cd540 0000000000000000 ffff88021fc1ac70
Sep 9 20:38:59 dev-storage1 kernel: [ 897.940322] 00000000fffffff0 0000000000000002 0000000000000000 000000000008c800
Sep 9 20:38:59 dev-storage1 kernel: [ 897.940325] Call Trace:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.947289] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.947292] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
Sep 9 20:38:59 dev-storage1 kernel: [ 897.947295] [<ffffffff8101222a>] cpu_idle+0xca/0x120
Sep 9 20:38:59 dev-storage1 kernel: [ 897.947299] [<ffffffff81620c0e>] rest_init+0x72/0x74
Sep 9 20:38:59 dev-storage1 kernel: [ 897.947302] [<ffffffff81cfbc03>] start_kernel+0x3b0/0x3bd
Sep 9 20:38:59 dev-storage1 kernel: [ 897.947305] [<ffffffff81cfb388>] x86_64_start_reservations+0x132/0x136
Sep 9 20:38:59 dev-storage1 kernel: [ 897.947308] [<ffffffff81cfb140>] ? early_idt_handlers+0x140/0x140
Sep 9 20:38:59 dev-storage1 kernel: [ 897.947311] [<ffffffff81cfb459>] x86_64_start_kernel+0xcd/0xdc
Sep 9 20:38:59 dev-storage1 kernel: [ 897.947312] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> cc 93 d2 ff 4c 29 f8 48 89 c7 e8 b1 11 d0 ff 4c 69 f0 40 42
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969301] Call Trace:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969304] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969306] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969309] [<ffffffff8101222a>] cpu_idle+0xca/0x120
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969311] [<ffffffff81620c0e>] rest_init+0x72/0x74
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969314] [<ffffffff81cfbc03>] start_kernel+0x3b0/0x3bd
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969316] [<ffffffff81cfb388>] x86_64_start_reservations+0x132/0x136
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969319] [<ffffffff81cfb140>] ? early_idt_handlers+0x140/0x140
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969322] [<ffffffff81cfb459>] x86_64_start_kernel+0xcd/0xdc
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969324] NMI backtrace for cpu 2
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969325] CPU 2
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969326] Modules linked in: xfs
Sep 9 20:38:59 dev-storage1 kernel: [ 897.969327] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=15010 jiffies)
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984392] ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984423]
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984425] Pid: 0, comm: swapper/2 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984428] RIP: 0010:[<ffffffff8136ba5f>] [<ffffffff8136ba5f>] intel_idle+0xbf/0x150
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984432] RSP: 0018:ffff880212189e38 EFLAGS: 00000046
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984434] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984436] RDX: 0000000000000000 RSI: ffff880212189fd8 RDI: ffffffff81c12e00
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984438] RBP: ffff880212189e98 R08: 00000000fffffffc R09: 0000000000000f9b
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984440] R10: 0000000000001ef1 R11: 0000000000000000 R12: 0000000000000003
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984442] R13: ffff88021fd1ac70 R14: 0000000000000020 R15: 12b249087f545cba
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984445] FS: 0000000000000000(0000) GS:ffff88021fd00000(0000) knlGS:0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984447] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984449] CR2: 00007f9905b7e000 CR3: 0000000001c05000 CR4: 00000000000406e0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984451] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984453] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984455] Process swapper/2 (pid: 0, threadinfo ffff880212188000, task ffff880212181700)
Sep 9 20:38:59 dev-storage1 kernel: [ 897.984457] Stack:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.991838] 0000000000000000 00000000003cdb87 ffff880212189e98 ffffffff81506fb7
Sep 9 20:38:59 dev-storage1 kernel: [ 897.991842] 0000000000000000 00000002003cdb87 0000000000000000 ffff88021fd1ac70
Sep 9 20:38:59 dev-storage1 kernel: [ 897.991846] 00000000fffffff0 0000000000000003 0000000000000002 0000000000000000
Sep 9 20:38:59 dev-storage1 kernel: [ 897.991850] Call Trace:
Sep 9 20:38:59 dev-storage1 kernel: [ 897.998926] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
Sep 9 20:38:59 dev-storage1 kernel: [ 897.998928] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
Sep 9 20:38:59 dev-storage1 kernel: [ 897.998930] [<ffffffff8101222a>] cpu_idle+0xca/0x120
Sep 9 20:38:59 dev-storage1 kernel: [ 897.998932] [<ffffffff81637fed>] start_secondary+0xd9/0xdb
Sep 9 20:38:59 dev-storage1 kernel: [ 897.998933] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> cc 93 d2 ff 4c 29 f8 48 89 c7 e8 b1 11 d0 ff 4c 69 f0 40 42
Sep 9 20:38:59 dev-storage1 kernel: [ 898.021107] Call Trace:
Sep 9 20:38:59 dev-storage1 kernel: [ 898.021109] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
Sep 9 20:38:59 dev-storage1 kernel: [ 898.021111] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
Sep 9 20:38:59 dev-storage1 kernel: [ 898.021113] [<ffffffff8101222a>] cpu_idle+0xca/0x120
Sep 9 20:38:59 dev-storage1 kernel: [ 898.021114] [<ffffffff81637fed>] start_secondary+0xd9/0xdb
Sep 9 20:39:27 dev-storage1 kernel: [ 925.653613] BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u:12:1602]
... etc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2012-09-09 at 22.15.55.png
Type: image/png
Size: 58629 bytes
Desc: not available
URL: <http://gluster.org/pipermail/gluster-users/attachments/20120909/40d0cfb0/attachment-0001.png>
