Hi Everyone, I'm using kernel 3.8.0r3 with Ubuntu 12.10. I'm mounting ceph rbd devices and exporting the target over fibre channel with target-utils. I'm noticing under high load situations when the ceph cluster goes into recovery mode it sometimes causes the kernel on my proxy server to crash. If anyone has insight it would be much appreciated :-) Here's the dstat output I see just before it crashes: ----total-cpu-usage---- ---procs--- -dsk/total- ----interrupts--- --io/total- -net/bond0- -----virtual-memory---- usr sys idl wai hiq siq|run blk new| read writ| 69 70 71 | read writ| recv send|majpf minpf alloc free 0 0 100 0 0 1| 0 0 0| 0 6144B| 21 132 11 | 0 12.0 |7186B 1081k| 0 0 392 8 0 1 99 0 0 0| 0 0 0| 0 709k| 85 606 31 | 0 26.0 |4702B 547k| 0 0 4181 10 0 1 99 0 0 0| 0 0 0| 0 7168B| 22 426 18 | 0 14.0 | 72k 17M| 0 0 17 22 0 1 100 0 0 0| 0 0 0| 0 8192B| 35 264 15 | 0 16.0 | 19k 4840k| 0 0 1178 10 0 0 66 34 0 0| 0 5.0 0| 0 685k| 41 223 14 | 0 10.0 | 13k 2185k| 0 0 1181 6 0 0 0 99 0 0| 0 47 32| 0 7168B| 117 34 13 | 0 14.0 |2484B 5899B| 0 0 5412 12 0 0 0 100 0 0| 0 51 5.0| 0 5120B| 17 32 11 | 0 10.0 |3499B 16k| 0 0 429 9 0 0 0 100 0 0| 0 52 1.0| 0 52k| 9 48 13 | 0 8.00 |4368B 36k| 0 0 24 6 0 0 0 100 0 0| 0 52 1.0| 0 8192B| 5 31 14 | 0 16.0 |2201B 4511B| 0 0 19 9 0 0 0 100 0 0| 0 53 1.0| 0 5120B| 7 32 14 | 0 10.0 |3765B 8308B| 0 0 146 7 0 0 0 100 0 0| 0 53 0| 0 27k| 12 49 10 | 0 12.0 |3441B 47k| 0 0 13 6 0 0 0 100 0 0| 0 55 2.0| 0 7168B| 14 34 12 | 0 14.0 |2399B 4000B| 0 0 174 9 0 0 0 100 0 0| 0 55 0| 0 37k| 26 207 70 | 0 14.0 | 15k 1283k| 0 0 796 7 0 0 0 99 0 0| 0 56 0| 0 3072B| 19 26 9 | 0 6.00 |2041B 1951k| 0 0 149 5 0 0 0 100 0 0| 0 57 2.0| 0 47k| 16 41 11 | 0 14.0 |3027B 35k| 0 0 31 9 0 0 0 100 0 0| 0 58 0| 0 5120B| 19 28 11 | 0 10.0 |3286B 6639B| 0 0 17 8 0 0 0 100 0 0| 0 60 2.0| 0 3072B| 33 32 9 | 0 6.00 |2165B 13k| 0 0 150 7 0 0 0 100 0 0| 0 60 0| 0 7168B| 18 34 15 | 0 14.0 |3589B 4821B| 0 0 18 10 0 0 0 100 0 0| 0 61 1.0| 0 85k| 44 53 11 | 0 16.0 |4980B 55k| 0 0 33 14 0 0 0 100 0 0| 0 61 0| 0 3072B|1458 23 10 | 0 6.00 |1739B 3977B| 0 0 12 4 0 0 0 100 0 0| 0 61 1.0| 0 7168B| 138 37 12 | 0 14.0 |2265B 3882B| 0 0 17 8 0 0 0 100 0 0| 0 61 0| 0 5120B| 176 30 11 | 0 10.0 |3694B 36k| 0 0 6 6 0 1 0 99 0 0|1.0 72 10|4096B 645k| 321 659 36 |8.00 10.0 | 49k 11M| 0 0 4281 22 0 0 0 100 0 0| 0 74 2.0| 0 6144B| 240 32 11 | 0 12.0 |2302B 342k| 0 0 36 14 0 3 0 97 0 0| 0 25 5.0| 0 80M| 298 1582 301 | 0 182 | 90k 9742k| 0 0 947 10k 0 16 71 12 0 2|1.0 0 0|7168B 303M| 816 4719 548 |14.0 730 | 486k 77M| 0 0 25k 39k 1 7 91 0 0 1| 0 0 0| 0 135M| 374 4250 815 | 0 518 | 449k 91M| 0 0 16k 17k Here's the kernel panic message I see: [21084.227187] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1161676 [21084.228438] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1159432 [21084.228518] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 1188824 [21084.228522] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1194632 [21084.228525] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1149312 [21084.228527] ABORT_TASK: Found referenced qla2xxx task_tag: 1152172 [21084.235317] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1159520 [21084.236891] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1159564 [21084.238471] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1159608 [21084.240043] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1159652 [21084.241614] ABORT_TASK: Found referenced qla2xxx task_tag: 1157540 [21084.496096] ------------[ cut here ]------------ [21084.497830] Kernel BUG at ffffffff8107a4af [verbose debug info unavailable] [21084.499699] invalid opcode: 0000 [#1] SMP [21084.500064] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21084.500064] CPU 0 [21084.500064] Pid: 3914, comm: kworker/u:4 Not tainted 3.8.0-030800rc4-generic #201301172335 HP ProLiant DL360 G5 [21084.500064] RIP: 0010:[<ffffffff8107a4af>] [<ffffffff8107a4af>] __cancel_work_timer+0x8f/0xa0 [21084.500064] RSP: 0018:ffff880126337ce8 EFLAGS: 00010246 [21084.500064] RAX: 0000000000000200 RBX: ffff880124b45118 RCX: 0000000000000000 [21084.500064] RDX: 0000000000000001 RSI: ffff8801248e1740 RDI: ffff880126337cc0 [21084.500064] RBP: ffff880126337d18 R08: ffff880126336000 R09: 0000000000000001 [21084.500064] R10: 0000000000000020 R11: 0000000000000001 R12: 0000000000000000 [21084.500064] R13: 0000000000000000 R14: ffff880125e2c680 R15: ffff88012805c5a0 [21084.544012] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 1160576 [21084.544036] ABORT_TASK: Found referenced qla2xxx task_tag: 1140512 [21084.500064] FS: 0000000000000000(0000) GS:ffff88012fc00000(0000) knlGS:0000000000000000 [21084.500064] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [21084.500064] CR2: 0000000000a3e000 CR3: 0000000125d1d000 CR4: 00000000000007f0 [21084.500064] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21084.500064] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [21084.500064] Process kworker/u:4 (pid: 3914, threadinfo ffff880126336000, task ffff8801248e1740) [21084.500064] Stack: [21084.500064] 0000000000000286 0000000000000296 ffffffff816eb319 ffff880124b44f90 [21084.500064] ffff880124b45028 000000000011a9a4 ffff880126337d28 ffffffff8107a4f0 [21084.500064] ffff880126337d98 ffffffffa008ddaa 0000000000000082 ffff880127381000 [21084.500064] Call Trace: [21084.500064] [<ffffffff816eb319>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [21084.500064] [<ffffffff8107a4f0>] cancel_work_sync+0x10/0x20 [21084.500064] [<ffffffffa008ddaa>] core_tmr_abort_task+0x17a/0x240 [target_core_mod] [21084.500064] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21084.500064] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21084.632006] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21084.632006] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21084.632006] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21084.632006] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21084.632006] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.632006] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21084.632006] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.632006] Code: 66 90 66 90 48 89 df e8 e0 fe ff ff 48 8b 03 a8 01 74 18 45 85 ed 48 c7 03 00 02 02 00 0f 95 c0 48 83 c4 18 5b 41 5c 41 5d 5d c3 <0f> 0b 48 89 df e8 b7 fe ff ff e9 7a ff ff ff 66 90 66 66 66 66 [21084.632006] RIP [<ffffffff8107a4af>] __cancel_work_timer+0x8f/0xa0 [21084.632006] RSP <ffff880126337ce8> [21084.698021] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 1152172 [21084.707218] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1153580 [21084.723240] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1153272 [21084.734054] ---[ end trace 8e5f870637161349 ]--- [21084.734104] BUG: unable to handle kernel paging request at ffffffffffffffd8 [21084.734111] IP: [<ffffffff8107f350>] kthread_data+0x10/0x20 [21084.734113] PGD 1c0f067 PUD 1c10067 PMD 0 [21084.734115] Oops: 0000 [#2] SMP [21084.734139] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21084.734142] CPU 0 [21084.734142] Pid: 3914, comm: kworker/u:4 Tainted: G D 3.8.0-030800rc4-generic #201301172335 HP ProLiant DL360 G5 [21084.734145] RIP: 0010:[<ffffffff8107f350>] [<ffffffff8107f350>] kthread_data+0x10/0x20 [21084.734146] RSP: 0018:ffff8801263379b8 EFLAGS: 00010092 [21084.734147] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff81e734e0 [21084.734148] RDX: 0000000000000007 RSI: 0000000000000000 RDI: ffff8801248e1740 [21084.734148] RBP: ffff8801263379b8 R08: 0000000000000001 R09: ffff88012628fcb0 [21084.734149] R10: 0000000000000000 R11: ffff88012628fca0 R12: 0000000000000000 [21084.734150] R13: ffff8801248e1b30 R14: 0000000000000001 R15: 0000000000000006 [21084.734151] FS: 0000000000000000(0000) GS:ffff88012fc00000(0000) knlGS:0000000000000000 [21084.734152] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [21084.734153] CR2: ffffffffffffffd8 CR3: 0000000125d1d000 CR4: 00000000000007f0 [21084.734154] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [21084.734155] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [21084.734157] Process kworker/u:4 (pid: 3914, threadinfo ffff880126336000, task ffff8801248e1740) [21084.734157] Stack: [21084.734160] ffff8801263379d8 ffffffff8107a625 ffff8801263379d8 ffff88012fc13f40 [21084.734161] ffff880126337a58 ffffffff816e9dcf ffff880126337a18 ffff8801248e1740 [21084.734163] ffff880126337fd8 ffff880126337fd8 ffff880126337fd8 0000000000013f40 [21084.734163] Call Trace: [21084.734167] [<ffffffff8107a625>] wq_worker_sleeping+0x15/0xc0 [21084.734171] [<ffffffff816e9dcf>] __schedule+0x5df/0x6c0 [21084.734173] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.734177] [<ffffffff8105fdcb>] do_exit+0x2bb/0x480 [21084.734180] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.734183] [<ffffffff81017df8>] die+0x58/0x90 [21084.734185] [<ffffffff816ebe9b>] do_trap+0xcb/0x170 [21084.734190] [<ffffffff8101540a>] do_invalid_op+0x9a/0xd0 [21084.734191] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.734193] [<ffffffff816eb3f5>] ? _raw_spin_lock_irq+0x15/0x20 [21084.734195] [<ffffffff816ea07d>] ? wait_for_common+0x13d/0x180 [21084.734198] [<ffffffff816f529e>] invalid_op+0x1e/0x30 [21084.734200] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.734201] [<ffffffff8107a490>] ? __cancel_work_timer+0x70/0xa0 [21084.734203] [<ffffffff816eb319>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [21084.734205] [<ffffffff8107a4f0>] cancel_work_sync+0x10/0x20 [21084.734224] [<ffffffffa008ddaa>] core_tmr_abort_task+0x17a/0x240 [target_core_mod] [21084.734230] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21084.734232] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21084.734238] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21084.734239] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21084.734241] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21084.734243] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21084.734244] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.734246] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21084.734248] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.734264] Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8b 87 98 03 00 00 55 48 89 e5 <48> 8b 40 d8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 [21084.734266] RIP [<ffffffff8107f350>] kthread_data+0x10/0x20 [21084.734266] RSP <ffff8801263379b8> [21084.734267] CR2: ffffffffffffffd8 [21084.734268] ---[ end trace 8e5f87063716134a ]--- [21084.734269] Fixing recursive fault but reboot is needed! [21085.097457] ------------[ cut here ]------------ [21085.097457] WARNING: at /home/apw/COD/linux/kernel/watchdog.c:246 watchdog_overflow_callback+0x9a/0xc0() [21085.097457] Hardware name: ProLiant DL360 G5 [21085.097457] Watchdog detected hard LOCKUP on cpu 1 [21085.097457] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21085.097457] Pid: 3912, comm: kworker/u:2 Tainted: G D 3.8.0-030800rc4-generic #201301172335 [21085.097457] Call Trace: [21085.097457] <NMI> [<ffffffff810599af>] warn_slowpath_common+0x7f/0xc0 [21085.097457] [<ffffffff81059aa6>] warn_slowpath_fmt+0x46/0x50 [21085.097457] [<ffffffff810ecafa>] watchdog_overflow_callback+0x9a/0xc0 [21085.097457] [<ffffffff8112bb3c>] __perf_event_overflow+0x9c/0x220 [21085.097457] [<ffffffff81025c1a>] ? x86_perf_event_set_period+0xda/0x150 [21085.097457] [<ffffffff8112c334>] perf_event_overflow+0x14/0x20 [21085.097457] [<ffffffff8102b99b>] intel_pmu_handle_irq+0x19b/0x250 [21085.097457] [<ffffffff816ed061>] perf_event_nmi_handler+0x21/0x30 [21085.097457] [<ffffffff816ec7a1>] nmi_handle.isra.0+0x51/0x80 [21085.097457] [<ffffffff816ec949>] default_do_nmi+0xe9/0x240 [21085.097457] [<ffffffff816ecb30>] do_nmi+0x90/0xd0 [21085.097457] [<ffffffff816ebd41>] end_repeat_nmi+0x1e/0x2e [21085.097457] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21085.097457] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21085.097457] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21085.097457] <<EOE>> <IRQ> [<ffffffff816eb10e>] _raw_spin_lock+0xe/0x20 [21085.097457] [<ffffffff8109e938>] do_sched_rt_period_timer+0x128/0x330 [21085.097457] [<ffffffff8101bb69>] ? read_tsc+0x9/0x20 [21085.097457] [<ffffffff8109eb6a>] sched_rt_period_timer+0x2a/0x60 [21085.097457] [<ffffffff810836d8>] __run_hrtimer+0x78/0x1f0 [21085.097457] [<ffffffff8109eb40>] ? do_sched_rt_period_timer+0x330/0x330 [21085.097457] [<ffffffff810af55c>] ? ktime_get_update_offsets+0x4c/0xd0 [21085.097457] [<ffffffff81083f67>] hrtimer_interrupt+0xf7/0x230 [21085.097457] [<ffffffff816f5df9>] smp_apic_timer_interrupt+0x69/0x99 [21085.097457] [<ffffffff816f4d5d>] apic_timer_interrupt+0x6d/0x80 [21085.097457] <EOI> [<ffffffff810855c2>] ? up+0x32/0x50 [21085.097457] [<ffffffff8105b510>] ? vprintk_emit+0x170/0x490 [21085.097457] [<ffffffff816d2c2d>] printk+0x61/0x63 [21085.097457] [<ffffffffa008de23>] core_tmr_abort_task+0x1f3/0x240 [target_core_mod] [21085.097457] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21085.097457] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21085.097457] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21085.097457] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21085.097457] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21085.097457] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21085.097457] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21085.097457] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21085.097457] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21085.097457] ---[ end trace 8e5f87063716134b ]--- [21084.736010] ------------[ cut here ]------------ [21084.736010] WARNING: at /home/apw/COD/linux/kernel/watchdog.c:246 watchdog_overflow_callback+0x9a/0xc0() [21084.736010] Hardware name: ProLiant DL360 G5 [21084.736010] Watchdog detected hard LOCKUP on cpu 0 [21084.736010] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper[21084.736010] ------------[ cut here ]------------ [21084.736010] WARNING: at /home/apw/COD/linux/kernel/softirq.c:160 local_bh_enable_ip+0x7a/0xa0() [21084.736010] Hardware name: ProLiant DL360 G5 [21084.736010] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21084.736010] Pid: 3914, comm: kworker/u:4 Tainted: G D W 3.8.0-030800rc4-generic #201301172335 [21084.736010] Call Trace: [21084.736010] <NMI> [<ffffffff810599af>] warn_slowpath_common+0x7f/0xc0 [21084.736010] [<ffffffff81059a0a>] warn_slowpath_null+0x1a/0x20 [21084.736010] [<ffffffff8106218a>] local_bh_enable_ip+0x7a/0xa0 [21084.736010] [<ffffffff816eb219>] _raw_spin_unlock_bh+0x19/0x20 [21084.736010] [<ffffffffa002e986>] bnx2_reg_rd_ind+0x46/0x60 [bnx2] [21084.736010] [<ffffffffa002e9b4>] bnx2_shmem_rd+0x14/0x20 [bnx2] [21084.736010] [<ffffffffa0035948>] bnx2_phy_int+0x98/0xf0 [bnx2] [21084.736010] [<ffffffffa003c993>] bnx2_poll+0x153/0x1a0 [bnx2] [21084.736010] [<ffffffff815f34e7>] poll_one_napi.isra.20+0x67/0xe0 [21084.736010] [<ffffffff810eedb8>] ? enable_irq+0x58/0x90 [21084.736010] [<ffffffff815f35d8>] poll_napi+0x78/0xa0 [21084.736010] [<ffffffff815f3ba0>] netpoll_poll_dev+0x50/0xf0 [21084.736010] [<ffffffff815f3e2f>] netpoll_send_skb_on_dev+0x1ef/0x390 [21084.736010] [<ffffffffa0505f28>] bond_dev_queue_xmit.part.47+0x4c/0x5a [bonding] [21084.736010] [<ffffffffa04fa245>] bond_dev_queue_xmit+0x35/0x40 [bonding] [21084.736010] [<ffffffffa050036a>] bond_3ad_xmit_xor+0x13a/0x180 [bonding] [21084.736010] [<ffffffff815cfe2b>] ? __alloc_skb+0x8b/0x2a0 [21084.736010] [<ffffffffa04fa713>] __bond_start_xmit+0x173/0x250 [bonding] [21084.736010] [<ffffffffa04fa85b>] bond_start_xmit+0x6b/0x80 [bonding] [21084.736010] [<ffffffff815f3deb>] netpoll_send_skb_on_dev+0x1ab/0x390 [21084.736010] [<ffffffff815f42c6>] netpoll_send_udp+0x256/0x270 [21084.736010] [<ffffffffa00e75f3>] write_msg.part.2+0xb3/0x110 [netconsole] [21084.736010] [<ffffffffa00e7685>] write_msg+0x35/0x40 [netconsole] [21084.736010] [<ffffffff8105acd3>] call_console_drivers.constprop.11+0xa3/0x120 [21084.736010] [<ffffffff8105aedd>] console_cont_flush.constprop.10+0x8d/0xb0 [21084.736010] [<ffffffff8105af28>] console_unlock.part.9+0x28/0x280 [21084.736010] [<ffffffff8105b19a>] console_unlock+0x1a/0x30 [21084.736010] [<ffffffff8105b5f5>] vprintk_emit+0x255/0x490 [21084.736010] [<ffffffff810ecafa>] ? watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff816d2c2d>] printk+0x61/0x63 [21084.736010] [<ffffffff810c2ac2>] print_modules+0x92/0xc0 [21084.736010] [<ffffffff810599aa>] warn_slowpath_common+0x7a/0xc0 [21084.736010] [<ffffffff81059aa6>] warn_slowpath_fmt+0x46/0x50 [21084.736010] [<ffffffff810ecafa>] watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff8112bb3c>] __perf_event_overflow+0x9c/0x220 [21084.736010] [<ffffffff81025c1a>] ? x86_perf_event_set_period+0xda/0x150 [21084.736010] [<ffffffff8112c334>] perf_event_overflow+0x14/0x20 [21084.736010] [<ffffffff8102b99b>] intel_pmu_handle_irq+0x19b/0x250 [21084.736010] [<ffffffff816ed061>] perf_event_nmi_handler+0x21/0x30 [21084.736010] [<ffffffff816ec7a1>] nmi_handle.isra.0+0x51/0x80 [21084.736010] [<ffffffff816ec8b8>] default_do_nmi+0x58/0x240 [21084.736010] [<ffffffff816ecb30>] do_nmi+0x90/0xd0 [21084.736010] [<ffffffff816ebd41>] end_repeat_nmi+0x1e/0x2e [21084.736010] [<ffffffff816ea1c9>] ? schedule+0x29/0x70 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] <<EOE>> [<ffffffff816eb3f5>] _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816e989a>] __schedule+0xaa/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105ff59>] do_exit+0x449/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff816d1e57>] no_context+0x1ab/0x1ba [21084.736010] [<ffffffff816d2037>] __bad_area_nosemaphore+0x1d1/0x1f0 [21084.736010] [<ffffffff816d1692>] ? pmd_offset+0x1a/0x20 [21084.736010] [<ffffffff816d2069>] bad_area_nosemaphore+0x13/0x15 [21084.736010] [<ffffffff816ef4b2>] __do_page_fault+0x382/0x540 [21084.736010] [<ffffffff81097a23>] ? update_curr+0x143/0x1f0 [21084.736010] [<ffffffff81359321>] ? vsnprintf+0x461/0x640 [21084.736010] [<ffffffff816ef69b>] do_page_fault+0x2b/0x50 [21084.736010] [<ffffffff816eb9d8>] page_fault+0x28/0x30 [21084.736010] [<ffffffff8107f350>] ? kthread_data+0x10/0x20 [21084.736010] [<ffffffff8107a625>] wq_worker_sleeping+0x15/0xc0 [21084.736010] [<ffffffff816e9dcf>] __schedule+0x5df/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105fdcb>] do_exit+0x2bb/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff81017df8>] die+0x58/0x90 [21084.736010] [<ffffffff816ebe9b>] do_trap+0xcb/0x170 [21084.736010] [<ffffffff8101540a>] do_invalid_op+0x9a/0xd0 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff816eb3f5>] ? _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816ea07d>] ? wait_for_common+0x13d/0x180 [21084.736010] [<ffffffff816f529e>] invalid_op+0x1e/0x30 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff8107a490>] ? __cancel_work_timer+0x70/0xa0 [21084.736010] [<ffffffff816eb319>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [21084.736010] [<ffffffff8107a4f0>] cancel_work_sync+0x10/0x20 [21084.736010] [<ffffffffa008ddaa>] core_tmr_abort_task+0x17a/0x240 [target_core_mod] [21084.736010] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21084.736010] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21084.736010] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21084.736010] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21084.736010] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21084.736010] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] ---[ end trace 8e5f87063716134c ]--- [21084.736010] ------------[ cut here ]------------ [21084.736010] WARNING: at /home/apw/COD/linux/net/core/skbuff.c:573 skb_release_head_state+0x10f/0x120() [21084.736010] Hardware name: ProLiant DL360 G5 [21084.736010] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21084.736010] Pid: 3914, comm: kworker/u:4 Tainted: G D W 3.8.0-030800rc4-generic #201301172335 [21084.736010] Call Trace: [21084.736010] <NMI> [<ffffffff810599af>] warn_slowpath_common+0x7f/0xc0 [21084.736010] [<ffffffff81059a0a>] warn_slowpath_null+0x1a/0x20 [21084.736010] [<ffffffff815ce3cf>] skb_release_head_state+0x10f/0x120 [21084.736010] [<ffffffff815ce1e6>] __kfree_skb+0x16/0x30 [21084.736010] [<ffffffff815ce75d>] consume_skb+0x3d/0xb0 [21084.736010] [<ffffffffa00385f1>] bnx2_tx_int.constprop.93+0x221/0x3e0 [bnx2] [21084.736010] [<ffffffffa003c738>] bnx2_poll_work+0x48/0x90 [bnx2] [21084.736010] [<ffffffffa003c899>] bnx2_poll+0x59/0x1a0 [bnx2] [21084.736010] [<ffffffff815f34e7>] poll_one_napi.isra.20+0x67/0xe0 [21084.736010] [<ffffffff810eedb8>] ? enable_irq+0x58/0x90 [21084.736010] [<ffffffff815f35d8>] poll_napi+0x78/0xa0 [21084.736010] [<ffffffff815f3ba0>] netpoll_poll_dev+0x50/0xf0 [21084.736010] [<ffffffff815f3e2f>] netpoll_send_skb_on_dev+0x1ef/0x390 [21084.736010] [<ffffffffa0505f28>] bond_dev_queue_xmit.part.47+0x4c/0x5a [bonding] [21084.736010] [<ffffffffa04fa245>] bond_dev_queue_xmit+0x35/0x40 [bonding] [21084.736010] [<ffffffffa050036a>] bond_3ad_xmit_xor+0x13a/0x180 [bonding] [21084.736010] [<ffffffff815cfe2b>] ? __alloc_skb+0x8b/0x2a0 [21084.736010] [<ffffffffa04fa713>] __bond_start_xmit+0x173/0x250 [bonding] [21084.736010] [<ffffffffa04fa85b>] bond_start_xmit+0x6b/0x80 [bonding] [21084.736010] [<ffffffff815f3deb>] netpoll_send_skb_on_dev+0x1ab/0x390 [21084.736010] [<ffffffff815f42c6>] netpoll_send_udp+0x256/0x270 [21084.736010] [<ffffffffa00e75f3>] write_msg.part.2+0xb3/0x110 [netconsole] [21084.736010] [<ffffffffa00e7685>] write_msg+0x35/0x40 [netconsole] [21084.736010] [<ffffffff8105acd3>] call_console_drivers.constprop.11+0xa3/0x120 [21084.736010] [<ffffffff8105aedd>] console_cont_flush.constprop.10+0x8d/0xb0 [21084.736010] [<ffffffff8105af28>] console_unlock.part.9+0x28/0x280 [21084.736010] [<ffffffff8105b19a>] console_unlock+0x1a/0x30 [21084.736010] [<ffffffff8105b5f5>] vprintk_emit+0x255/0x490 [21084.736010] [<ffffffff810ecafa>] ? watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff816d2c2d>] printk+0x61/0x63 [21084.736010] [<ffffffff810c2ac2>] print_modules+0x92/0xc0 [21084.736010] [<ffffffff810599aa>] warn_slowpath_common+0x7a/0xc0 [21084.736010] [<ffffffff81059aa6>] warn_slowpath_fmt+0x46/0x50 [21084.736010] [<ffffffff810ecafa>] watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff8112bb3c>] __perf_event_overflow+0x9c/0x220 [21084.736010] [<ffffffff81025c1a>] ? x86_perf_event_set_period+0xda/0x150 [21084.736010] [<ffffffff8112c334>] perf_event_overflow+0x14/0x20 [21084.736010] [<ffffffff8102b99b>] intel_pmu_handle_irq+0x19b/0x250 [21084.736010] [<ffffffff816ed061>] perf_event_nmi_handler+0x21/0x30 [21084.736010] [<ffffffff816ec7a1>] nmi_handle.isra.0+0x51/0x80 [21084.736010] [<ffffffff816ec8b8>] default_do_nmi+0x58/0x240 [21084.736010] [<ffffffff816ecb30>] do_nmi+0x90/0xd0 [21084.736010] [<ffffffff816ebd41>] end_repeat_nmi+0x1e/0x2e [21084.736010] [<ffffffff816ea1c9>] ? schedule+0x29/0x70 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] <<EOE>> [<ffffffff816eb3f5>] _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816e989a>] __schedule+0xaa/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105ff59>] do_exit+0x449/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff816d1e57>] no_context+0x1ab/0x1ba [21084.736010] [<ffffffff816d2037>] __bad_area_nosemaphore+0x1d1/0x1f0 [21084.736010] [<ffffffff816d1692>] ? pmd_offset+0x1a/0x20 [21084.736010] [<ffffffff816d2069>] bad_area_nosemaphore+0x13/0x15 [21084.736010] [<ffffffff816ef4b2>] __do_page_fault+0x382/0x540 [21084.736010] [<ffffffff81097a23>] ? update_curr+0x143/0x1f0 [21084.736010] [<ffffffff81359321>] ? vsnprintf+0x461/0x640 [21084.736010] [<ffffffff816ef69b>] do_page_fault+0x2b/0x50 [21084.736010] [<ffffffff816eb9d8>] page_fault+0x28/0x30 [21084.736010] [<ffffffff8107f350>] ? kthread_data+0x10/0x20 [21084.736010] [<ffffffff8107a625>] wq_worker_sleeping+0x15/0xc0 [21084.736010] [<ffffffff816e9dcf>] __schedule+0x5df/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105fdcb>] do_exit+0x2bb/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff81017df8>] die+0x58/0x90 [21084.736010] [<ffffffff816ebe9b>] do_trap+0xcb/0x170 [21084.736010] [<ffffffff8101540a>] do_invalid_op+0x9a/0xd0 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff816eb3f5>] ? _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816ea07d>] ? wait_for_common+0x13d/0x180 [21084.736010] [<ffffffff816f529e>] invalid_op+0x1e/0x30 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff8107a490>] ? __cancel_work_timer+0x70/0xa0 [21084.736010] [<ffffffff816eb319>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [21084.736010] [<ffffffff8107a4f0>] cancel_work_sync+0x10/0x20 [21084.736010] [<ffffffffa008ddaa>] core_tmr_abort_task+0x17a/0x240 [target_core_mod] [21084.736010] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21084.736010] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21084.736010] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21084.736010] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21084.736010] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21084.736010] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] ---[ end trace 8e5f87063716134d ]--- [21084.736010] ------------[ cut here ]------------ [21084.736010] WARNING: at /home/apw/COD/linux/net/core/skbuff.c:573 skb_release_head_state+0x10f/0x120() [21084.736010] Hardware name: ProLiant DL360 G5 [21084.736010] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21084.736010] Pid: 3914, comm: kworker/u:4 Tainted: G D W 3.8.0-030800rc4-generic #201301172335 [21084.736010] Call Trace: [21084.736010] <NMI> [<ffffffff810599af>] warn_slowpath_common+0x7f/0xc0 [21084.736010] [<ffffffff81059a0a>] warn_slowpath_null+0x1a/0x20 [21084.736010] [<ffffffff815ce3cf>] skb_release_head_state+0x10f/0x120 [21084.736010] [<ffffffff815ce1e6>] __kfree_skb+0x16/0x30 [21084.736010] [<ffffffff815ce75d>] consume_skb+0x3d/0xb0 [21084.736010] [<ffffffffa00385f1>] bnx2_tx_int.constprop.93+0x221/0x3e0 [bnx2] [21084.736010] [<ffffffffa003c738>] bnx2_poll_work+0x48/0x90 [bnx2] [21084.736010] [<ffffffffa003c899>] bnx2_poll+0x59/0x1a0 [bnx2] [21084.736010] [<ffffffff815f34e7>] poll_one_napi.isra.20+0x67/0xe0 [21084.736010] [<ffffffff810eedb8>] ? enable_irq+0x58/0x90 [21084.736010] [<ffffffff815f35d8>] poll_napi+0x78/0xa0 [21084.736010] [<ffffffff815f3ba0>] netpoll_poll_dev+0x50/0xf0 [21084.736010] [<ffffffff815f3e2f>] netpoll_send_skb_on_dev+0x1ef/0x390 [21084.736010] [<ffffffffa0505f28>] bond_dev_queue_xmit.part.47+0x4c/0x5a [bonding] [21084.736010] [<ffffffffa04fa245>] bond_dev_queue_xmit+0x35/0x40 [bonding] [21084.736010] [<ffffffffa050036a>] bond_3ad_xmit_xor+0x13a/0x180 [bonding] [21084.736010] [<ffffffff815cfe2b>] ? __alloc_skb+0x8b/0x2a0 [21084.736010] [<ffffffffa04fa713>] __bond_start_xmit+0x173/0x250 [bonding] [21084.736010] [<ffffffffa04fa85b>] bond_start_xmit+0x6b/0x80 [bonding] [21084.736010] [<ffffffff815f3deb>] netpoll_send_skb_on_dev+0x1ab/0x390 [21084.736010] [<ffffffff815f42c6>] netpoll_send_udp+0x256/0x270 [21084.736010] [<ffffffffa00e75f3>] write_msg.part.2+0xb3/0x110 [netconsole] [21084.736010] [<ffffffffa00e7685>] write_msg+0x35/0x40 [netconsole] [21084.736010] [<ffffffff8105acd3>] call_console_drivers.constprop.11+0xa3/0x120 [21084.736010] [<ffffffff8105aedd>] console_cont_flush.constprop.10+0x8d/0xb0 [21084.736010] [<ffffffff8105af28>] console_unlock.part.9+0x28/0x280 [21084.736010] [<ffffffff8105b19a>] console_unlock+0x1a/0x30 [21084.736010] [<ffffffff8105b5f5>] vprintk_emit+0x255/0x490 [21084.736010] [<ffffffff810ecafa>] ? watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff816d2c2d>] printk+0x61/0x63 [21084.736010] [<ffffffff810c2ac2>] print_modules+0x92/0xc0 [21084.736010] [<ffffffff810599aa>] warn_slowpath_common+0x7a/0xc0 [21084.736010] [<ffffffff81059aa6>] warn_slowpath_fmt+0x46/0x50 [21084.736010] [<ffffffff810ecafa>] watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff8112bb3c>] __perf_event_overflow+0x9c/0x220 [21084.736010] [<ffffffff81025c1a>] ? x86_perf_event_set_period+0xda/0x150 [21084.736010] [<ffffffff8112c334>] perf_event_overflow+0x14/0x20 [21084.736010] [<ffffffff8102b99b>] intel_pmu_handle_irq+0x19b/0x250 [21084.736010] [<ffffffff816ed061>] perf_event_nmi_handler+0x21/0x30 [21084.736010] [<ffffffff816ec7a1>] nmi_handle.isra.0+0x51/0x80 [21084.736010] [<ffffffff816ec8b8>] default_do_nmi+0x58/0x240 [21084.736010] [<ffffffff816ecb30>] do_nmi+0x90/0xd0 [21084.736010] [<ffffffff816ebd41>] end_repeat_nmi+0x1e/0x2e [21084.736010] [<ffffffff816ea1c9>] ? schedule+0x29/0x70 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] <<EOE>> [<ffffffff816eb3f5>] _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816e989a>] __schedule+0xaa/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105ff59>] do_exit+0x449/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff816d1e57>] no_context+0x1ab/0x1ba [21084.736010] [<ffffffff816d2037>] __bad_area_nosemaphore+0x1d1/0x1f0 [21084.736010] [<ffffffff816d1692>] ? pmd_offset+0x1a/0x20 [21084.736010] [<ffffffff816d2069>] bad_area_nosemaphore+0x13/0x15 [21084.736010] [<ffffffff816ef4b2>] __do_page_fault+0x382/0x540 [21084.736010] [<ffffffff81097a23>] ? update_curr+0x143/0x1f0 [21084.736010] [<ffffffff81359321>] ? vsnprintf+0x461/0x640 [21084.736010] [<ffffffff816ef69b>] do_page_fault+0x2b/0x50 [21084.736010] [<ffffffff816eb9d8>] page_fault+0x28/0x30 [21084.736010] [<ffffffff8107f350>] ? kthread_data+0x10/0x20 [21084.736010] [<ffffffff8107a625>] wq_worker_sleeping+0x15/0xc0 [21084.736010] [<ffffffff816e9dcf>] __schedule+0x5df/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105fdcb>] do_exit+0x2bb/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff81017df8>] die+0x58/0x90 [21084.736010] [<ffffffff816ebe9b>] do_trap+0xcb/0x170 [21084.736010] [<ffffffff8101540a>] do_invalid_op+0x9a/0xd0 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff816eb3f5>] ? _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816ea07d>] ? wait_for_common+0x13d/0x180 [21084.736010] [<ffffffff816f529e>] invalid_op+0x1e/0x30 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff8107a490>] ? __cancel_work_timer+0x70/0xa0 [21084.736010] [<ffffffff816eb319>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [21084.736010] [<ffffffff8107a4f0>] cancel_work_sync+0x10/0x20 [21084.736010] [<ffffffffa008ddaa>] core_tmr_abort_task+0x17a/0x240 [target_core_mod] [21084.736010] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21084.736010] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21084.736010] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21084.736010] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21084.736010] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21084.736010] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] ---[ end trace 8e5f87063716134e ]--- [21084.736010] ------------[ cut here ]------------ [21084.736010] WARNING: at /home/apw/COD/linux/net/core/skbuff.c:573 skb_release_head_state+0x10f/0x120() [21084.736010] Hardware name: ProLiant DL360 G5 [21084.736010] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21084.736010] Pid: 3914, comm: kworker/u:4 Tainted: G D W 3.8.0-030800rc4-generic #201301172335 [21084.736010] Call Trace: [21084.736010] <NMI> [<ffffffff810599af>] warn_slowpath_common+0x7f/0xc0 [21084.736010] [<ffffffff81059a0a>] warn_slowpath_null+0x1a/0x20 [21084.736010] [<ffffffff815ce3cf>] skb_release_head_state+0x10f/0x120 [21084.736010] [<ffffffff815ce1e6>] __kfree_skb+0x16/0x30 [21084.736010] [<ffffffff815ce75d>] consume_skb+0x3d/0xb0 [21084.736010] [<ffffffffa00385f1>] bnx2_tx_int.constprop.93+0x221/0x3e0 [bnx2] [21084.736010] [<ffffffffa003c738>] bnx2_poll_work+0x48/0x90 [bnx2] [21084.736010] [<ffffffffa003c899>] bnx2_poll+0x59/0x1a0 [bnx2] [21084.736010] [<ffffffff815f34e7>] poll_one_napi.isra.20+0x67/0xe0 [21084.736010] [<ffffffff810eedb8>] ? enable_irq+0x58/0x90 [21084.736010] [<ffffffff815f35d8>] poll_napi+0x78/0xa0 [21084.736010] [<ffffffff815f3ba0>] netpoll_poll_dev+0x50/0xf0 [21084.736010] [<ffffffff815f3e2f>] netpoll_send_skb_on_dev+0x1ef/0x390 [21084.736010] [<ffffffffa0505f28>] bond_dev_queue_xmit.part.47+0x4c/0x5a [bonding] [21084.736010] [<ffffffffa04fa245>] bond_dev_queue_xmit+0x35/0x40 [bonding] [21084.736010] [<ffffffffa050036a>] bond_3ad_xmit_xor+0x13a/0x180 [bonding] [21084.736010] [<ffffffff815cfe2b>] ? __alloc_skb+0x8b/0x2a0 [21084.736010] [<ffffffffa04fa713>] __bond_start_xmit+0x173/0x250 [bonding] [21084.736010] [<ffffffffa04fa85b>] bond_start_xmit+0x6b/0x80 [bonding] [21084.736010] [<ffffffff815f3deb>] netpoll_send_skb_on_dev+0x1ab/0x390 [21084.736010] [<ffffffff815f42c6>] netpoll_send_udp+0x256/0x270 [21084.736010] [<ffffffffa00e75f3>] write_msg.part.2+0xb3/0x110 [netconsole] [21084.736010] [<ffffffffa00e7685>] write_msg+0x35/0x40 [netconsole] [21084.736010] [<ffffffff8105acd3>] call_console_drivers.constprop.11+0xa3/0x120 [21084.736010] [<ffffffff8105aedd>] console_cont_flush.constprop.10+0x8d/0xb0 [21084.736010] [<ffffffff8105af28>] console_unlock.part.9+0x28/0x280 [21084.736010] [<ffffffff8105b19a>] console_unlock+0x1a/0x30 [21084.736010] [<ffffffff8105b5f5>] vprintk_emit+0x255/0x490 [21084.736010] [<ffffffff810ecafa>] ? watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff816d2c2d>] printk+0x61/0x63 [21084.736010] [<ffffffff810c2ac2>] print_modules+0x92/0xc0 [21084.736010] [<ffffffff810599aa>] warn_slowpath_common+0x7a/0xc0 [21084.736010] [<ffffffff81059aa6>] warn_slowpath_fmt+0x46/0x50 [21084.736010] [<ffffffff810ecafa>] watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff8112bb3c>] __perf_event_overflow+0x9c/0x220 [21084.736010] [<ffffffff81025c1a>] ? x86_perf_event_set_period+0xda/0x150 [21084.736010] [<ffffffff8112c334>] perf_event_overflow+0x14/0x20 [21084.736010] [<ffffffff8102b99b>] intel_pmu_handle_irq+0x19b/0x250 [21084.736010] [<ffffffff816ed061>] perf_event_nmi_handler+0x21/0x30 [21084.736010] [<ffffffff816ec7a1>] nmi_handle.isra.0+0x51/0x80 [21084.736010] [<ffffffff816ec8b8>] default_do_nmi+0x58/0x240 [21084.736010] [<ffffffff816ecb30>] do_nmi+0x90/0xd0 [21084.736010] [<ffffffff816ebd41>] end_repeat_nmi+0x1e/0x2e [21084.736010] [<ffffffff816ea1c9>] ? schedule+0x29/0x70 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] <<EOE>> [<ffffffff816eb3f5>] _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816e989a>] __schedule+0xaa/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105ff59>] do_exit+0x449/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff816d1e57>] no_context+0x1ab/0x1ba [21084.736010] [<ffffffff816d2037>] __bad_area_nosemaphore+0x1d1/0x1f0 [21084.736010] [<ffffffff816d1692>] ? pmd_offset+0x1a/0x20 [21084.736010] [<ffffffff816d2069>] bad_area_nosemaphore+0x13/0x15 [21084.736010] [<ffffffff816ef4b2>] __do_page_fault+0x382/0x540 [21084.736010] [<ffffffff81097a23>] ? update_curr+0x143/0x1f0 [21084.736010] [<ffffffff81359321>] ? vsnprintf+0x461/0x640 [21084.736010] [<ffffffff816ef69b>] do_page_fault+0x2b/0x50 [21084.736010] [<ffffffff816eb9d8>] page_fault+0x28/0x30 [21084.736010] [<ffffffff8107f350>] ? kthread_data+0x10/0x20 [21084.736010] [<ffffffff8107a625>] wq_worker_sleeping+0x15/0xc0 [21084.736010] [<ffffffff816e9dcf>] __schedule+0x5df/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105fdcb>] do_exit+0x2bb/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff81017df8>] die+0x58/0x90 [21084.736010] [<ffffffff816ebe9b>] do_trap+0xcb/0x170 [21084.736010] [<ffffffff8101540a>] do_invalid_op+0x9a/0xd0 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff816eb3f5>] ? _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816ea07d>] ? wait_for_common+0x13d/0x180 [21084.736010] [<ffffffff816f529e>] invalid_op+0x1e/0x30 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff8107a490>] ? __cancel_work_timer+0x70/0xa0 [21084.736010] [<ffffffff816eb319>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [21084.736010] [<ffffffff8107a4f0>] cancel_work_sync+0x10/0x20 [21084.736010] [<ffffffffa008ddaa>] core_tmr_abort_task+0x17a/0x240 [target_core_mod] [21084.736010] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21084.736010] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21084.736010] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21084.736010] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21084.736010] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21084.736010] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] ---[ end trace 8e5f87063716134f ]--- [21084.736010] ------------[ cut here ]------------ [21084.736010] WARNING: at /home/apw/COD/linux/net/core/skbuff.c:573 skb_release_head_state+0x10f/0x120() [21084.736010] Hardware name: ProLiant DL360 G5 [21084.736010] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21084.736010] Pid: 3914, comm: kworker/u:4 Tainted: G D W 3.8.0-030800rc4-generic #201301172335 [21084.736010] Call Trace: [21084.736010] <NMI> [<ffffffff810599af>] warn_slowpath_common+0x7f/0xc0 [21084.736010] [<ffffffff81059a0a>] warn_slowpath_null+0x1a/0x20 [21084.736010] [<ffffffff815ce3cf>] skb_release_head_state+0x10f/0x120 [21084.736010] [<ffffffff815ce1e6>] __kfree_skb+0x16/0x30 [21084.736010] [<ffffffff815ce75d>] consume_skb+0x3d/0xb0 [21084.736010] [<ffffffffa00385f1>] bnx2_tx_int.constprop.93+0x221/0x3e0 [bnx2] [21084.736010] [<ffffffffa003c738>] bnx2_poll_work+0x48/0x90 [bnx2] [21084.736010] [<ffffffffa003c899>] bnx2_poll+0x59/0x1a0 [bnx2] [21084.736010] [<ffffffff815f34e7>] poll_one_napi.isra.20+0x67/0xe0 [21084.736010] [<ffffffff810eedb8>] ? enable_irq+0x58/0x90 [21084.736010] [<ffffffff815f35d8>] poll_napi+0x78/0xa0 [21084.736010] [<ffffffff815f3ba0>] netpoll_poll_dev+0x50/0xf0 [21084.736010] [<ffffffff815f3e2f>] netpoll_send_skb_on_dev+0x1ef/0x390 [21084.736010] [<ffffffffa0505f28>] bond_dev_queue_xmit.part.47+0x4c/0x5a [bonding] [21084.736010] [<ffffffffa04fa245>] bond_dev_queue_xmit+0x35/0x40 [bonding] [21084.736010] [<ffffffffa050036a>] bond_3ad_xmit_xor+0x13a/0x180 [bonding] [21084.736010] [<ffffffff815cfe2b>] ? __alloc_skb+0x8b/0x2a0 [21084.736010] [<ffffffffa04fa713>] __bond_start_xmit+0x173/0x250 [bonding] [21084.736010] [<ffffffffa04fa85b>] bond_start_xmit+0x6b/0x80 [bonding] [21084.736010] [<ffffffff815f3deb>] netpoll_send_skb_on_dev+0x1ab/0x390 [21084.736010] [<ffffffff815f42c6>] netpoll_send_udp+0x256/0x270 [21084.736010] [<ffffffffa00e75f3>] write_msg.part.2+0xb3/0x110 [netconsole] [21084.736010] [<ffffffffa00e7685>] write_msg+0x35/0x40 [netconsole] [21084.736010] [<ffffffff8105acd3>] call_console_drivers.constprop.11+0xa3/0x120 [21084.736010] [<ffffffff8105aedd>] console_cont_flush.constprop.10+0x8d/0xb0 [21084.736010] [<ffffffff8105af28>] console_unlock.part.9+0x28/0x280 [21084.736010] [<ffffffff8105b19a>] console_unlock+0x1a/0x30 [21084.736010] [<ffffffff8105b5f5>] vprintk_emit+0x255/0x490 [21084.736010] [<ffffffff810ecafa>] ? watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff816d2c2d>] printk+0x61/0x63 [21084.736010] [<ffffffff810c2ac2>] print_modules+0x92/0xc0 [21084.736010] [<ffffffff810599aa>] warn_slowpath_common+0x7a/0xc0 [21084.736010] [<ffffffff81059aa6>] warn_slowpath_fmt+0x46/0x50 [21084.736010] [<ffffffff810ecafa>] watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff8112bb3c>] __perf_event_overflow+0x9c/0x220 [21084.736010] [<ffffffff81025c1a>] ? x86_perf_event_set_period+0xda/0x150 [21084.736010] [<ffffffff8112c334>] perf_event_overflow+0x14/0x20 [21084.736010] [<ffffffff8102b99b>] intel_pmu_handle_irq+0x19b/0x250 [21084.736010] [<ffffffff816ed061>] perf_event_nmi_handler+0x21/0x30 [21084.736010] [<ffffffff816ec7a1>] nmi_handle.isra.0+0x51/0x80 [21084.736010] [<ffffffff816ec8b8>] default_do_nmi+0x58/0x240 [21084.736010] [<ffffffff816ecb30>] do_nmi+0x90/0xd0 [21084.736010] [<ffffffff816ebd41>] end_repeat_nmi+0x1e/0x2e [21084.736010] [<ffffffff816ea1c9>] ? schedule+0x29/0x70 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] <<EOE>> [<ffffffff816eb3f5>] _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816e989a>] __schedule+0xaa/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105ff59>] do_exit+0x449/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff816d1e57>] no_context+0x1ab/0x1ba [21084.736010] [<ffffffff816d2037>] __bad_area_nosemaphore+0x1d1/0x1f0 [21084.736010] [<ffffffff816d1692>] ? pmd_offset+0x1a/0x20 [21084.736010] [<ffffffff816d2069>] bad_area_nosemaphore+0x13/0x15 [21084.736010] [<ffffffff816ef4b2>] __do_page_fault+0x382/0x540 [21084.736010] [<ffffffff81097a23>] ? update_curr+0x143/0x1f0 [21084.736010] [<ffffffff81359321>] ? vsnprintf+0x461/0x640 [21084.736010] [<ffffffff816ef69b>] do_page_fault+0x2b/0x50 [21084.736010] [<ffffffff816eb9d8>] page_fault+0x28/0x30 [21084.736010] [<ffffffff8107f350>] ? kthread_data+0x10/0x20 [21084.736010] [<ffffffff8107a625>] wq_worker_sleeping+0x15/0xc0 [21084.736010] [<ffffffff816e9dcf>] __schedule+0x5df/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105fdcb>] do_exit+0x2bb/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff81017df8>] die+0x58/0x90 [21084.736010] [<ffffffff816ebe9b>] do_trap+0xcb/0x170 [21084.736010] [<ffffffff8101540a>] do_invalid_op+0x9a/0xd0 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff816eb3f5>] ? _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816ea07d>] ? wait_for_common+0x13d/0x180 [21084.736010] [<ffffffff816f529e>] invalid_op+0x1e/0x30 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff8107a490>] ? __cancel_work_timer+0x70/0xa0 [21084.736010] [<ffffffff816eb319>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [21084.736010] [<ffffffff8107a4f0>] cancel_work_sync+0x10/0x20 [21084.736010] [<ffffffffa008ddaa>] core_tmr_abort_task+0x17a/0x240 [target_core_mod] [21084.736010] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21084.736010] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21084.736010] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21084.736010] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21084.736010] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21084.736010] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] ---[ end trace 8e5f870637161350 ]--- [21084.736010] ------------[ cut here ]------------ [21084.736010] WARNING: at /home/apw/COD/linux/net/core/skbuff.c:573 skb_release_head_state+0x10f/0x120() [21084.736010] Hardware name: ProLiant DL360 G5 [21084.736010] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21084.736010] Pid: 3914, comm: kworker/u:4 Tainted: G D W 3.8.0-030800rc4-generic #201301172335 [21084.736010] Call Trace: [21084.736010] <NMI> [<ffffffff810599af>] warn_slowpath_common+0x7f/0xc0 [21084.736010] [<ffffffff81059a0a>] warn_slowpath_null+0x1a/0x20 [21084.736010] [<ffffffff815ce3cf>] skb_release_head_state+0x10f/0x120 [21084.736010] [<ffffffff815ce1e6>] __kfree_skb+0x16/0x30 [21084.736010] [<ffffffff815ce75d>] consume_skb+0x3d/0xb0 [21084.736010] [<ffffffffa00385f1>] bnx2_tx_int.constprop.93+0x221/0x3e0 [bnx2] [21084.736010] [<ffffffffa003c738>] bnx2_poll_work+0x48/0x90 [bnx2] [21084.736010] [<ffffffffa003c899>] bnx2_poll+0x59/0x1a0 [bnx2] [21084.736010] [<ffffffff815f34e7>] poll_one_napi.isra.20+0x67/0xe0 [21084.736010] [<ffffffff810eedb8>] ? enable_irq+0x58/0x90 [21084.736010] [<ffffffff815f35d8>] poll_napi+0x78/0xa0 [21084.736010] [<ffffffff815f3ba0>] netpoll_poll_dev+0x50/0xf0 [21084.736010] [<ffffffff815f3e2f>] netpoll_send_skb_on_dev+0x1ef/0x390 [21084.736010] [<ffffffffa0505f28>] bond_dev_queue_xmit.part.47+0x4c/0x5a [bonding] [21084.736010] [<ffffffffa04fa245>] bond_dev_queue_xmit+0x35/0x40 [bonding] [21084.736010] [<ffffffffa050036a>] bond_3ad_xmit_xor+0x13a/0x180 [bonding] [21084.736010] [<ffffffff815cfe2b>] ? __alloc_skb+0x8b/0x2a0 [21084.736010] [<ffffffffa04fa713>] __bond_start_xmit+0x173/0x250 [bonding] [21084.736010] [<ffffffffa04fa85b>] bond_start_xmit+0x6b/0x80 [bonding] [21084.736010] [<ffffffff815f3deb>] netpoll_send_skb_on_dev+0x1ab/0x390 [21084.736010] [<ffffffff815f42c6>] netpoll_send_udp+0x256/0x270 [21084.736010] [<ffffffffa00e75f3>] write_msg.part.2+0xb3/0x110 [netconsole] [21084.736010] [<ffffffffa00e7685>] write_msg+0x35/0x40 [netconsole] [21084.736010] [<ffffffff8105acd3>] call_console_drivers.constprop.11+0xa3/0x120 [21084.736010] [<ffffffff8105aedd>] console_cont_flush.constprop.10+0x8d/0xb0 [21084.736010] [<ffffffff8105af28>] console_unlock.part.9+0x28/0x280 [21084.736010] [<ffffffff8105b19a>] console_unlock+0x1a/0x30 [21084.736010] [<ffffffff8105b5f5>] vprintk_emit+0x255/0x490 [21084.736010] [<ffffffff810ecafa>] ? watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff816d2c2d>] printk+0x61/0x63 [21084.736010] [<ffffffff810c2ac2>] print_modules+0x92/0xc0 [21084.736010] [<ffffffff810599aa>] warn_slowpath_common+0x7a/0xc0 [21084.736010] [<ffffffff81059aa6>] warn_slowpath_fmt+0x46/0x50 [21084.736010] [<ffffffff810ecafa>] watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff8112bb3c>] __perf_event_overflow+0x9c/0x220 [21084.736010] [<ffffffff81025c1a>] ? x86_perf_event_set_period+0xda/0x150 [21084.736010] [<ffffffff8112c334>] perf_event_overflow+0x14/0x20 [21084.736010] [<ffffffff8102b99b>] intel_pmu_handle_irq+0x19b/0x250 [21084.736010] [<ffffffff816ed061>] perf_event_nmi_handler+0x21/0x30 [21084.736010] [<ffffffff816ec7a1>] nmi_handle.isra.0+0x51/0x80 [21084.736010] [<ffffffff816ec8b8>] default_do_nmi+0x58/0x240 [21084.736010] [<ffffffff816ecb30>] do_nmi+0x90/0xd0 [21084.736010] [<ffffffff816ebd41>] end_repeat_nmi+0x1e/0x2e [21084.736010] [<ffffffff816ea1c9>] ? schedule+0x29/0x70 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] <<EOE>> [<ffffffff816eb3f5>] _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816e989a>] __schedule+0xaa/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105ff59>] do_exit+0x449/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff816d1e57>] no_context+0x1ab/0x1ba [21084.736010] [<ffffffff816d2037>] __bad_area_nosemaphore+0x1d1/0x1f0 [21084.736010] [<ffffffff816d1692>] ? pmd_offset+0x1a/0x20 [21084.736010] [<ffffffff816d2069>] bad_area_nosemaphore+0x13/0x15 [21084.736010] [<ffffffff816ef4b2>] __do_page_fault+0x382/0x540 [21084.736010] [<ffffffff81097a23>] ? update_curr+0x143/0x1f0 [21084.736010] [<ffffffff81359321>] ? vsnprintf+0x461/0x640 [21084.736010] [<ffffffff816ef69b>] do_page_fault+0x2b/0x50 [21084.736010] [<ffffffff816eb9d8>] page_fault+0x28/0x30 [21084.736010] [<ffffffff8107f350>] ? kthread_data+0x10/0x20 [21084.736010] [<ffffffff8107a625>] wq_worker_sleeping+0x15/0xc0 [21084.736010] [<ffffffff816e9dcf>] __schedule+0x5df/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105fdcb>] do_exit+0x2bb/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff81017df8>] die+0x58/0x90 [21084.736010] [<ffffffff816ebe9b>] do_trap+0xcb/0x170 [21084.736010] [<ffffffff8101540a>] do_invalid_op+0x9a/0xd0 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff816eb3f5>] ? _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816ea07d>] ? wait_for_common+0x13d/0x180 [21084.736010] [<ffffffff816f529e>] invalid_op+0x1e/0x30 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff8107a490>] ? __cancel_work_timer+0x70/0xa0 [21084.736010] [<ffffffff816eb319>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [21084.736010] [<ffffffff8107a4f0>] cancel_work_sync+0x10/0x20 [21084.736010] [<ffffffffa008ddaa>] core_tmr_abort_task+0x17a/0x240 [target_core_mod] [21084.736010] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21084.736010] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21084.736010] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21084.736010] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21084.736010] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21084.736010] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] ---[ end trace 8e5f870637161351 ]--- [21084.736010] ------------[ cut here ]------------ [21084.736010] WARNING: at /home/apw/COD/linux/net/core/skbuff.c:573 skb_release_head_state+0x10f/0x120() [21084.736010] Hardware name: ProLiant DL360 G5 [21084.736010] Modules linked in: netconsole target_core_pscsi target_core_file target_core_iblock ib_srpt tcm_qla2xxx tcm_loop tcm_fc iscsi_target_mod target_core_mod qla2xxx ib_cm ib_sa ib_mad ib_core libfc ext2 bonding radeon coretemp kvm ttm gpio_ich drm_kms_helper i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21084.736010] Pid: 3914, comm: kworker/u:4 Tainted: G D W 3.8.0-030800rc4-generic #201301172335 [21084.736010] Call Trace: [21084.736010] <NMI> [<ffffffff810599af>] warn_slowpath_common+0x7f/0xc0 [21084.736010] [<ffffffff81059a0a>] warn_slowpath_null+0x1a/0x20 [21084.736010] [<ffffffff815ce3cf>] skb_release_head_state+0x10f/0x120 [21084.736010] [<ffffffff815ce1e6>] __kfree_skb+0x16/0x30 [21084.736010] [<ffffffff815ce75d>] consume_skb+0x3d/0xb0 [21084.736010] [<ffffffffa00385f1>] bnx2_tx_int.constprop.93+0x221/0x3e0 [bnx2] [21084.736010] [<ffffffffa003c738>] bnx2_poll_work+0x48/0x90 [bnx2] [21084.736010] [<ffffffffa003c899>] bnx2_poll+0x59/0x1a0 [bnx2] [21084.736010] [<ffffffff815f34e7>] poll_one_napi.isra.20+0x67/0xe0 [21084.736010] [<ffffffff810eedb8>] ? enable_irq+0x58/0x90 [21084.736010] [<ffffffff815f35d8>] poll_napi+0x78/0xa0 [21084.736010] [<ffffffff815f3ba0>] netpoll_poll_dev+0x50/0xf0 [21084.736010] [<ffffffff815f3e2f>] netpoll_send_skb_on_dev+0x1ef/0x390 [21084.736010] [<ffffffffa0505f28>] bond_dev_queue_xmit.part.47+0x4c/0x5a [bonding] [21084.736010] [<ffffffffa04fa245>] bond_dev_queue_xmit+0x35/0x40 [bonding] [21084.736010] [<ffffffffa050036a>] bond_3ad_xmit_xor+0x13a/0x180 [bonding] [21084.736010] [<ffffffff815cfe2b>] ? __alloc_skb+0x8b/0x2a0 [21084.736010] [<ffffffffa04fa713>] __bond_start_xmit+0x173/0x250 [bonding] [21084.736010] [<ffffffffa04fa85b>] bond_start_xmit+0x6b/0x80 [bonding] [21084.736010] [<ffffffff815f3deb>] netpoll_send_skb_on_dev+0x1ab/0x390 [21084.736010] [<ffffffff815f42c6>] netpoll_send_udp+0x256/0x270 [21084.736010] [<ffffffffa00e75f3>] write_msg.part.2+0xb3/0x110 [netconsole] [21084.736010] [<ffffffffa00e7685>] write_msg+0x35/0x40 [netconsole] [21084.736010] [<ffffffff8105acd3>] call_console_drivers.constprop.11+0xa3/0x120 [21084.736010] [<ffffffff8105aedd>] console_cont_flush.constprop.10+0x8d/0xb0 [21084.736010] [<ffffffff8105af28>] console_unlock.part.9+0x28/0x280 [21084.736010] [<ffffffff8105b19a>] console_unlock+0x1a/0x30 [21084.736010] [<ffffffff8105b5f5>] vprintk_emit+0x255/0x490 [21084.736010] [<ffffffff810ecafa>] ? watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff816d2c2d>] printk+0x61/0x63 [21084.736010] [<ffffffff810c2ac2>] print_modules+0x92/0xc0 [21084.736010] [<ffffffff810599aa>] warn_slowpath_common+0x7a/0xc0 [21084.736010] [<ffffffff81059aa6>] warn_slowpath_fmt+0x46/0x50 [21084.736010] [<ffffffff810ecafa>] watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff8112bb3c>] __perf_event_overflow+0x9c/0x220 [21084.736010] [<ffffffff81025c1a>] ? x86_perf_event_set_period+0xda/0x150 [21084.736010] [<ffffffff8112c334>] perf_event_overflow+0x14/0x20 [21084.736010] [<ffffffff8102b99b>] intel_pmu_handle_irq+0x19b/0x250 [21084.736010] [<ffffffff816ed061>] perf_event_nmi_handler+0x21/0x30 [21084.736010] [<ffffffff816ec7a1>] nmi_handle.isra.0+0x51/0x80 [21084.736010] [<ffffffff816ec8b8>] default_do_nmi+0x58/0x240 [21084.736010] [<ffffffff816ecb30>] do_nmi+0x90/0xd0 [21084.736010] [<ffffffff816ebd41>] end_repeat_nmi+0x1e/0x2e [21084.736010] [<ffffffff816ea1c9>] ? schedule+0x29/0x70 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] <<EOE>> [<ffffffff816eb3f5>] _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816e989a>] __schedule+0xaa/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105ff59>] do_exit+0x449/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff816d1e57>] no_context+0x1ab/0x1ba [21084.736010] [<ffffffff816d2037>] __bad_area_nosemaphore+0x1d1/0x1f0 [21084.736010] [<ffffffff816d1692>] ? pmd_offset+0x1a/0x20 [21084.736010] [<ffffffff816d2069>] bad_area_nosemaphore+0x13/0x15 [21084.736010] [<ffffffff816ef4b2>] __do_page_fault+0x382/0x540 [21084.736010] [<ffffffff81097a23>] ? update_curr+0x143/0x1f0 [21084.736010] [<ffffffff81359321>] ? vsnprintf+0x461/0x640 [21084.736010] [<ffffffff816ef69b>] do_page_fault+0x2b/0x50 [21084.736010] [<ffffffff816eb9d8>] page_fault+0x28/0x30 [21084.736010] [<ffffffff8107f350>] ? kthread_data+0x10/0x20 [21084.736010] [<ffffffff8107a625>] wq_worker_sleeping+0x15/0xc0 [21084.736010] [<ffffffff816e9dcf>] __schedule+0x5df/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105fdcb>] do_exit+0x2bb/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff81017df8>] die+0x58/0x90 [21084.736010] [<ffffffff816ebe9b>] do_trap+0xcb/0x170 [21084.736010] [<ffffffff8101540a>] do_invalid_op+0x9a/0xd0 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff816eb3f5>] ? _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816ea07d>] ? wait_for_common+0x13d/0x180 [21084.736010] [<ffffffff816f529e>] invalid_op+0x1e/0x30 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff8107a490>] ? __cancel_work_timer+0x70/0xa0 [21084.736010] [<ffffffff816eb319>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [21084.736010] [<ffffffff8107a4f0>] cancel_work_sync+0x10/0x20 [21084.736010] [<ffffffffa008ddaa>] core_tmr_abort_task+0x17a/0x240 [target_core_mod] [21084.736010] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21084.736010] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21084.736010] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21084.736010] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21084.736010] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21084.736010] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] ---[ end trace 8e5f870637161352 ]--- [21084.736010] i5000_edac drm edac_core psmouse i5k_amb ipmi_si lpc_ich microcode ipmi_msghandler configfs i2c_algo_bit shpchp mac_hid serio_raw hpilo hpwdt rbd libceph lp parport hid_generic usbhid hid btrfs zlib_deflate libcrc32c hpsa scsi_transport_fc bnx2 cciss scsi_tgt [last unloaded: qla2xxx] [21084.736010] Pid: 3914, comm: kworker/u:4 Tainted: G D W 3.8.0-030800rc4-generic #201301172335 [21084.736010] Call Trace: [21084.736010] <NMI> [<ffffffff810599af>] warn_slowpath_common+0x7f/0xc0 [21084.736010] [<ffffffff81059aa6>] warn_slowpath_fmt+0x46/0x50 [21084.736010] [<ffffffff810ecafa>] watchdog_overflow_callback+0x9a/0xc0 [21084.736010] [<ffffffff8112bb3c>] __perf_event_overflow+0x9c/0x220 [21084.736010] [<ffffffff81025c1a>] ? x86_perf_event_set_period+0xda/0x150 [21084.736010] [<ffffffff8112c334>] perf_event_overflow+0x14/0x20 [21084.736010] [<ffffffff8102b99b>] intel_pmu_handle_irq+0x19b/0x250 [21084.736010] [<ffffffff816ed061>] perf_event_nmi_handler+0x21/0x30 [21084.736010] [<ffffffff816ec7a1>] nmi_handle.isra.0+0x51/0x80 [21084.736010] [<ffffffff816ec8b8>] default_do_nmi+0x58/0x240 [21084.736010] [<ffffffff816ecb30>] do_nmi+0x90/0xd0 [21084.736010] [<ffffffff816ebd41>] end_repeat_nmi+0x1e/0x2e [21084.736010] [<ffffffff816ea1c9>] ? schedule+0x29/0x70 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] [<ffffffff81045c42>] ? __ticket_spin_lock+0x22/0x30 [21084.736010] <<EOE>> [<ffffffff816eb3f5>] _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816e989a>] __schedule+0xaa/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105ff59>] do_exit+0x449/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff816d1e57>] no_context+0x1ab/0x1ba [21084.736010] [<ffffffff816d2037>] __bad_area_nosemaphore+0x1d1/0x1f0 [21084.736010] [<ffffffff816d1692>] ? pmd_offset+0x1a/0x20 [21084.736010] [<ffffffff816d2069>] bad_area_nosemaphore+0x13/0x15 [21084.736010] [<ffffffff816ef4b2>] __do_page_fault+0x382/0x540 [21084.736010] [<ffffffff81097a23>] ? update_curr+0x143/0x1f0 [21084.736010] [<ffffffff81359321>] ? vsnprintf+0x461/0x640 [21084.736010] [<ffffffff816ef69b>] do_page_fault+0x2b/0x50 [21084.736010] [<ffffffff816eb9d8>] page_fault+0x28/0x30 [21084.736010] [<ffffffff8107f350>] ? kthread_data+0x10/0x20 [21084.736010] [<ffffffff8107a625>] wq_worker_sleeping+0x15/0xc0 [21084.736010] [<ffffffff816e9dcf>] __schedule+0x5df/0x6c0 [21084.736010] [<ffffffff816ea1c9>] schedule+0x29/0x70 [21084.736010] [<ffffffff8105fdcb>] do_exit+0x2bb/0x480 [21084.736010] [<ffffffff816ec620>] oops_end+0xb0/0xf0 [21084.736010] [<ffffffff81017df8>] die+0x58/0x90 [21084.736010] [<ffffffff816ebe9b>] do_trap+0xcb/0x170 [21084.736010] [<ffffffff8101540a>] do_invalid_op+0x9a/0xd0 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff816eb3f5>] ? _raw_spin_lock_irq+0x15/0x20 [21084.736010] [<ffffffff816ea07d>] ? wait_for_common+0x13d/0x180 [21084.736010] [<ffffffff816f529e>] invalid_op+0x1e/0x30 [21084.736010] [<ffffffff8107a4af>] ? __cancel_work_timer+0x8f/0xa0 [21084.736010] [<ffffffff8107a490>] ? __cancel_work_timer+0x70/0xa0 [21084.736010] [<ffffffff816eb319>] ? _raw_spin_unlock_irqrestore+0x19/0x30 [21084.736010] [<ffffffff8107a4f0>] cancel_work_sync+0x10/0x20 [21084.736010] [<ffffffffa008ddaa>] core_tmr_abort_task+0x17a/0x240 [target_core_mod] [21084.736010] [<ffffffffa008f9ff>] target_tmr_work+0xcf/0xf0 [target_core_mod] [21084.736010] [<ffffffff81078cd0>] process_one_work+0x130/0x480 [21084.736010] [<ffffffffa008f930>] ? transport_cmd_check_stop+0x160/0x160 [target_core_mod] [21084.736010] [<ffffffff81079a07>] worker_thread+0x167/0x400 [21084.736010] [<ffffffff810798a0>] ? manage_workers+0x120/0x120 [21084.736010] [<ffffffff8107ef20>] kthread+0xc0/0xd0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] [<ffffffff816f3f6c>] ret_from_fork+0x7c/0xb0 [21084.736010] [<ffffffff8107ee60>] ? flush_kthread_worker+0xb0/0xb0 [21084.736010] ---[ end trace 8e5f870637161353 ]--- Chris Holcombe Unix Administrator Corporation Service Company cholcomb@xxxxxxxxxxx 302-636-8667 ________________________________ NOTICE: This e-mail and any attachments is intended only for use by the addressee(s) named herein and may contain legally privileged, proprietary or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this email, and any attachments thereto, is strictly prohibited. If you receive this email in error please immediately notify me via reply email or at (800) 927-9800 and permanently delete the original copy and any copy of any e-mail, and any printout. -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
