Hi!
I've come across a strange kernel panic issue on LIO on Linux 3.6.11
(b2824f4e0990716407b0c0e7acee75bb6353febf). The issue seems to be linked
with late IOs calling back to an already deleted target and loops here
until the kernel panics.
The test setup was the following:
- Exporting more than one IBLOCK (here two MD RAIDs, but seems to do the
same with any block devices) over iSCSI.
- Running intensive IOs (64 ios depth, 548k ios, 100% write, 100%
random) using IO meter from a single fast host
While the above IOs are running, when "deleting" the iSCSI targets using
rtslib, and by intermittence, the kernel panicked.
The issue is quite intermittent and might happen about 1 out of 20 tests
or less.
We have not yet tried to reproduce the issue on Ramdisk or file
backstores but it seems that this could be linked to the async and slow
nature of the iblock backstores. In fact, the issue usually happens
always after 100ms of tearing off the target, which could indicate that
a BIO comes back and causes the issue.
Attached is the kernel trace we get when running this test.
We are going to try to rollback our kernel/update but we were hoping to
stick on the 3.6.y branch for now.
Have you seen this issue before? Do you know if it has been resolved in
a later version?
Thanks a lot in advance for your help, as usual.
Regards,
Ben - MPSTOR.
[ 461.623324] Unable to locate Target IQN: iqn.2003-01.org.linux-iscsi.f4x6dx04x9ex3cx30.i686:sn.14d959eec7cb in Storage Node
[ 461.634607] iSCSI Login negotiation failed.
[ 461.638823] iSCSI Login negotiation failed.
[ 461.738367] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 461.745534] IP: [<4104c1f9>] __wake_up_common+0x19/0x70
[ 461.750782] *pdpt = 0000000051ba6001 *pde = 0000000000000000
[ 461.756559] Oops: 0000 [#1]
[ 461.759472] Modules linked in: target_core_pscsi target_core_file target_core_iblock tcm_loop iscsi_target_mod tcm_qla2xxx tcm_fc target_core_mod fcoe libfcoe libfc myri10ge bnx2x bnx2 qla2xxx
[ 461.776925] Pid: 1330, comm: kworker/0:2 Tainted: G W 3.6.11-00027-g4114d18 #122 To be filled by O.E.M. To be filled by O.E.M./P8B-C Series
[ 461.790350] EIP: 0060:[<4104c1f9>] EFLAGS: 00010007 CPU: 0
[ 461.795827] EIP is at __wake_up_common+0x19/0x70
[ 461.800451] EAX: 9d4dbe00 EBX: fffffff4 ECX: 00000001 EDX: 00000003
[ 461.806745] ESI: 00000286 EDI: 9d634c00 EBP: 9d4dbe00 ESP: b15d9ef4
[ 461.813014] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 461.818419] CR0: 8005003b CR2: 00000000 CR3: 51ba5000 CR4: 000407f0
[ 461.824696] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 461.831002] DR6: ffff0ff0 DR7: 00000400
[ 461.834857] Process kworker/0:2 (pid: 1330, ti=b15d8000 task=f31eea40 task.ti=b15d8000)
[ 461.842856] Stack:
[ 461.844870] 41a02b60 00000001 00000003 00000000 00000286 9d634c00 9d4dbe00 4104e9e2
[ 461.852755] 00000000 00000000 9cd1b538 9cd1b530 f993cd15 00000000 1d000286 9d634d60
[ 461.860624] 9d634e44 b1fe0c40 00000000 f9944f1c f93eba92 9d634e44 41a01a64 41040706
[ 461.868510] Call Trace:
[ 461.870952] [<4104e9e2>] ? __wake_up+0x22/0x30
[ 461.875494] [<f993cd15>] ? iscsit_add_cmd_to_response_queue+0x85/0xb0 [iscsi_target_mod]
[ 461.883689] [<f9944f1c>] ? lio_queue_status+0x1c/0x20 [iscsi_target_mod]
[ 461.890488] [<f93eba92>] ? target_complete_ok_work+0x112/0x150 [target_core_mod]
[ 461.897964] [<41040706>] ? process_one_work+0xe6/0x330
[ 461.903225] [<f93eb980>] ? transport_init_session+0x60/0x60 [target_core_mod]
[ 461.910466] [<410416a9>] ? worker_thread+0xd9/0x310
[ 461.915456] [<410415d0>] ? gcwq_mayday_timeout+0x80/0x80
[ 461.920852] [<410415d0>] ? gcwq_mayday_timeout+0x80/0x80
[ 461.926265] [<41045358>] ? kthread+0x78/0x80
[ 461.930615] [<410452e0>] ? kthread_freezable_should_stop+0x20/0x20
[ 461.936929] [<417210b6>] ? kernel_thread_helper+0x6/0xd
[ 461.942235] Code: 26 00 f3 c3 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 89 c5 57 56 53 83 ec 0c 89 54 24 08 89 4c 24 04 8b 18 83 eb 0c 39 6d 00 <8b> 43 0c 74 3e 8d 70 f4 eb 05 89 f3 8d 72 f4 8b 44 24 24 8b 4c
[ 461.962210] EIP: [<4104c1f9>] __wake_up_common+0x19/0x70 SS:ESP 0068:b15d9ef4
[ 461.969395] CR2: 0000000000000000
[ 461.972715] ---[ end trace 761c4e9eccbbed8d ]---
[ 461.977352] BUG: unable to handle kernel paging request at fffffffc
[ 461.983689] IP: [<41044fb6>] kthread_data+0x6/0x10
[ 461.988495] *pdpt = 0000000001aab001 *pde = 0000000001aaf067 *pte = 0000000000000000
[ 461.996407] Oops: 0000 [#2]
[ 461.999322] Modules linked in: target_core_pscsi target_core_file target_core_iblock tcm_loop iscsi_target_mod tcm_qla2xxx tcm_fc target_core_mod fcoe libfcoe libfc myri10ge bnx2x bnx2 qla2xxx
[ 462.016739] Pid: 1330, comm: kworker/0:2 Tainted: G D W 3.6.11-00027-g4114d18 #122 To be filled by O.E.M. To be filled by O.E.M./P8B-C Series
[ 462.030234] EIP: 0060:[<41044fb6>] EFLAGS: 00010046 CPU: 0
[ 462.035734] EIP is at kthread_data+0x6/0x10
[ 462.039937] EAX: 00000000 EBX: 00000001 ECX: 41a02b60 EDX: 00000000
[ 462.046198] ESI: f31eeb98 EDI: 00000000 EBP: f31eea40 ESP: b15d9cec
[ 462.052476] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 462.057870] CR0: 8005003b CR2: fffffffc CR3: 51ba5000 CR4: 000407f0
[ 462.064132] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 462.070392] DR6: ffff0ff0 DR7: 00000400
[ 462.074213] Process kworker/0:2 (pid: 1330, ti=b15d8000 task=f31eea40 task.ti=b15d8000)
[ 462.082238] Stack:
[ 462.084261] 4104146a 00000001 f31eeb98 f31eea40 4171fdb4 f3002380 000c000a 00000000
[ 462.092180] 00000000 b15d9d2f f31eea40 00003c05 f4e64100 8040003f f338a4c0 410b6111
[ 462.100032] 4102db51 00000001 0001f1a5 f462b7e0 b173f640 f31eea38 410b6111 4102eb48
[ 462.107920] Call Trace:
[ 462.110360] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 462.115592] [<4171fdb4>] ? __schedule+0x244/0x400
[ 462.120391] [<410b6111>] ? kmem_cache_free+0xc1/0xf0
[ 462.125491] [<4102db51>] ? release_task+0x131/0x2e0
[ 462.130471] [<410b6111>] ? kmem_cache_free+0xc1/0xf0
[ 462.135538] [<4102eb48>] ? do_exit+0x3a8/0x6d0
[ 462.140086] [<4102db9a>] ? release_task+0x17a/0x2e0
[ 462.145058] [<4102eb48>] ? do_exit+0x3a8/0x6d0
[ 462.149608] [<4102eb7e>] ? do_exit+0x3de/0x6d0
[ 462.154130] [<41004ac0>] ? oops_end+0x80/0x80
[ 462.158602] [<410223be>] ? no_context+0x16e/0x1f0
[ 462.163426] [<4104d69c>] ? ttwu_do_wakeup+0x2c/0x70
[ 462.168398] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 462.174164] [<4104e680>] ? try_to_wake_up+0x80/0x80
[ 462.179138] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 462.184566] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 462.189538] [<4105150e>] ? update_curr+0x7e/0x110
[ 462.194363] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 462.199407] [<41720948>] ? error_code+0x58/0x60
[ 462.204049] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 462.209099] [<4104c1f9>] ? __wake_up_common+0x19/0x70
[ 462.214244] [<4104e9e2>] ? __wake_up+0x22/0x30
[ 462.218802] [<f993cd15>] ? iscsit_add_cmd_to_response_queue+0x85/0xb0 [iscsi_target_mod]
[ 462.227010] [<f9944f1c>] ? lio_queue_status+0x1c/0x20 [iscsi_target_mod]
[ 462.233815] [<f93eba92>] ? target_complete_ok_work+0x112/0x150 [target_core_mod]
[ 462.241360] [<41040706>] ? process_one_work+0xe6/0x330
[ 462.246578] [<f93eb980>] ? transport_init_session+0x60/0x60 [target_core_mod]
[ 462.253812] [<410416a9>] ? worker_thread+0xd9/0x310
[ 462.258802] [<410415d0>] ? gcwq_mayday_timeout+0x80/0x80
[ 462.264231] [<410415d0>] ? gcwq_mayday_timeout+0x80/0x80
[ 462.269636] [<41045358>] ? kthread+0x78/0x80
[ 462.274020] [<410452e0>] ? kthread_freezable_should_stop+0x20/0x20
[ 462.280270] [<417210b6>] ? kernel_thread_helper+0x6/0xd
[ 462.285605] Code: 42 1b 00 31 d2 85 c0 74 97 eb cc 90 90 90 90 90 90 90 90 90 90 a1 34 92 9f 41 8b 80 2c 01 00 00 8b 40 f8 c3 90 8b 80 2c 01 00 00 <8b> 40 fc c3 8d b6 00 00 00 00 31 c0 c3 8d b6 00 00 00 00 8d bc
[ 462.305537] EIP: [<41044fb6>] kthread_data+0x6/0x10 SS:ESP 0068:b15d9cec
[ 462.312298] CR2: 00000000fffffffc
[ 462.315611] ---[ end trace 761c4e9eccbbed8e ]---
[ 462.320237] Fixing recursive fault but reboot is needed!
[ 462.325563] BUG: unable to handle kernel paging request at fffffffc
[ 462.331892] IP: [<41044fb6>] kthread_data+0x6/0x10
[ 462.336709] *pdpt = 0000000001aab001 *pde = 0000000001aaf067 *pte = 0000000000000000
[ 462.344596] Oops: 0000 [#3]
[ 462.347484] Modules linked in: target_core_pscsi target_core_file target_core_iblock tcm_loop iscsi_target_mod tcm_qla2xxx tcm_fc target_core_mod fcoe libfcoe libfc myri10ge bnx2x bnx2 qla2xxx
[ 462.364916] Pid: 1330, comm: kworker/0:2 Tainted: G D W 3.6.11-00027-g4114d18 #122 To be filled by O.E.M. To be filled by O.E.M./P8B-C Series
[ 462.378337] EIP: 0060:[<41044fb6>] EFLAGS: 00010046 CPU: 0
[ 462.383826] EIP is at kthread_data+0x6/0x10
[ 462.388004] EAX: 00000000 EBX: f31eea40 ECX: 41a02b60 EDX: 00000000
[ 462.394280] ESI: f31eeb98 EDI: 00000000 EBP: f31eea40 ESP: b15d9ae4
[ 462.400577] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 462.406007] CR0: 8005003b CR2: fffffffc CR3: 51ba5000 CR4: 000407f0
[ 462.412283] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 462.418580] DR6: ffff0ff0 DR7: 00000400
[ 462.422428] Process kworker/0:2 (pid: 1330, ti=b15d8000 task=f31eea40 task.ti=b15d8000)
[ 462.430442] Stack:
[ 462.432457] 4104146a f31eea40 f31eeb98 f31eea40 4171fdb4 0000002c 0000002c 00000001
[ 462.440324] 00000006 00000046 f31eea40 00000000 00000000 00000000 41ab4f02 0000002c
[ 462.448210] 41ab4f02 00000046 00000000 00000024 00000004 00000006 f31eea40 00000009
[ 462.456114] Call Trace:
[ 462.458554] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 462.463804] [<4171fdb4>] ? __schedule+0x244/0x400
[ 462.468585] [<4102bd6b>] ? printk+0x3b/0x40
[ 462.472873] [<4102ee33>] ? do_exit+0x693/0x6d0
[ 462.477395] [<4102bd6b>] ? printk+0x3b/0x40
[ 462.481660] [<41029d47>] ? print_oops_end_marker+0x27/0x30
[ 462.487219] [<41004ac0>] ? oops_end+0x80/0x80
[ 462.491657] [<410223be>] ? no_context+0x16e/0x1f0
[ 462.496454] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 462.502197] [<4104d666>] ? check_preempt_curr+0x76/0x80
[ 462.507514] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 462.512935] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 462.517906] [<41049514>] ? hrtimer_forward+0x164/0x1b0
[ 462.523122] [<411f42e7>] ? rb_insert_color+0xb7/0xe0
[ 462.528181] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 462.533222] [<41720948>] ? error_code+0x58/0x60
[ 462.537873] [<41050000>] ? wake_up_new_task+0x80/0x80
[ 462.543002] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 462.548058] [<41044fb6>] ? kthread_data+0x6/0x10
[ 462.552763] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 462.558021] [<4171fdb4>] ? __schedule+0x244/0x400
[ 462.562820] [<410b6111>] ? kmem_cache_free+0xc1/0xf0
[ 462.567879] [<4102db51>] ? release_task+0x131/0x2e0
[ 462.572841] [<410b6111>] ? kmem_cache_free+0xc1/0xf0
[ 462.577876] [<4102eb48>] ? do_exit+0x3a8/0x6d0
[ 462.582439] [<4102db9a>] ? release_task+0x17a/0x2e0
[ 462.587404] [<4102eb48>] ? do_exit+0x3a8/0x6d0
[ 462.591935] [<4102eb7e>] ? do_exit+0x3de/0x6d0
[ 462.596474] [<41004ac0>] ? oops_end+0x80/0x80
[ 462.600926] [<410223be>] ? no_context+0x16e/0x1f0
[ 462.605718] [<4104d69c>] ? ttwu_do_wakeup+0x2c/0x70
[ 462.610681] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 462.616448] [<4104e680>] ? try_to_wake_up+0x80/0x80
[ 462.621430] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 462.626832] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 462.631813] [<4105150e>] ? update_curr+0x7e/0x110
[ 462.636605] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 462.641654] [<41720948>] ? error_code+0x58/0x60
[ 462.646290] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 462.651355] [<4104c1f9>] ? __wake_up_common+0x19/0x70
[ 462.656520] [<4104e9e2>] ? __wake_up+0x22/0x30
[ 462.661068] [<f993cd15>] ? iscsit_add_cmd_to_response_queue+0x85/0xb0 [iscsi_target_mod]
[ 462.669274] [<f9944f1c>] ? lio_queue_status+0x1c/0x20 [iscsi_target_mod]
[ 462.676089] [<f93eba92>] ? target_complete_ok_work+0x112/0x150 [target_core_mod]
[ 462.683610] [<41040706>] ? process_one_work+0xe6/0x330
[ 462.688860] [<f93eb980>] ? transport_init_session+0x60/0x60 [target_core_mod]
[ 462.696129] [<410416a9>] ? worker_thread+0xd9/0x310
[ 462.701094] [<410415d0>] ? gcwq_mayday_timeout+0x80/0x80
[ 462.706508] [<410415d0>] ? gcwq_mayday_timeout+0x80/0x80
[ 462.711910] [<41045358>] ? kthread+0x78/0x80
[ 462.716276] [<410452e0>] ? kthread_freezable_should_stop+0x20/0x20
[ 462.722573] [<417210b6>] ? kernel_thread_helper+0x6/0xd
[ 462.727890] Code: 42 1b 00 31 d2 85 c0 74 97 eb cc 90 90 90 90 90 90 90 90 90 90 a1 34 92 9f 41 8b 80 2c 01 00 00 8b 40 f8 c3 90 8b 80 2c 01 00 00 <8b> 40 fc c3 8d b6 00 00 00 00 31 c0 c3 8d b6 00 00 00 00 8d bc
[ 462.747778] EIP: [<41044fb6>] kthread_data+0x6/0x10 SS:ESP 0068:b15d9ae4
[ 462.754523] CR2: 00000000fffffffc
[ 462.757843] ---[ end trace 761c4e9eccbbed8f ]---
[ 462.762459] Fixing recursive fault but reboot is needed!
[ 462.767777] BUG: unable to handle kernel paging request at fffffffc
[ 462.774097] IP: [<41044fb6>] kthread_data+0x6/0x10
[ 462.778932] *pdpt = 0000000001aab001 *pde = 0000000001aaf067 *pte = 0000000000000000
[ 462.786846] Oops: 0000 [#4]
[ 462.789757] Modules linked in: target_core_pscsi target_core_file target_core_iblock tcm_loop iscsi_target_mod tcm_qla2xxx tcm_fc target_core_mod fcoe libfcoe libfc myri10ge bnx2x bnx2 qla2xxx
[ 462.807155] Pid: 1330, comm: kworker/0:2 Tainted: G D W 3.6.11-00027-g4114d18 #122 To be filled by O.E.M. To be filled by O.E.M./P8B-C Series
[ 462.820568] EIP: 0060:[<41044fb6>] EFLAGS: 00010046 CPU: 0
[ 462.826085] EIP is at kthread_data+0x6/0x10
[ 462.830288] EAX: 00000000 EBX: f31eea40 ECX: 41a02b60 EDX: 00000000
[ 462.836573] ESI: f31eeb98 EDI: 00000000 EBP: f31eea40 ESP: b15d98dc
[ 462.842877] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 462.848263] CR0: 8005003b CR2: fffffffc CR3: 51ba5000 CR4: 000407f0
[ 462.854550] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 462.860837] DR6: ffff0ff0 DR7: 00000400
[ 462.864685] Process kworker/0:2 (pid: 1330, ti=b15d8000 task=f31eea40 task.ti=b15d8000)
[ 462.872708] Stack:
[ 462.874723] 4104146a f31eea40 f31eeb98 f31eea40 4171fdb4 0000002c 0000002c 00000001
[ 462.882592] 00000006 00000046 f31eea40 00000000 00000000 00000000 41ab4f02 0000002c
[ 462.890478] 41ab4f02 00000046 00000000 00000024 00000004 00000006 f31eea40 00000009
[ 462.898338] Call Trace:
[ 462.900803] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 462.906060] [<4171fdb4>] ? __schedule+0x244/0x400
[ 462.910853] [<4102bd6b>] ? printk+0x3b/0x40
[ 462.915131] [<4102ee33>] ? do_exit+0x693/0x6d0
[ 462.919670] [<4102bd6b>] ? printk+0x3b/0x40
[ 462.923943] [<41029d47>] ? print_oops_end_marker+0x27/0x30
[ 462.929511] [<41004ac0>] ? oops_end+0x80/0x80
[ 462.933974] [<410223be>] ? no_context+0x16e/0x1f0
[ 462.938773] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 462.944506] [<411f6ebc>] ? format_decode+0x30c/0x350
[ 462.949564] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 462.954960] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 462.959922] [<411f74b0>] ? sprintf+0x20/0x30
[ 462.964289] [<411f7fb6>] ? __delay+0x6/0x10
[ 462.968555] [<41250b69>] ? wait_for_xmitr+0x29/0x80
[ 462.973551] [<412528d1>] ? serial8250_console_putchar+0x21/0x30
[ 462.979577] [<411f7fb6>] ? __delay+0x6/0x10
[ 462.983859] [<41250b69>] ? wait_for_xmitr+0x29/0x80
[ 462.988846] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 462.993905] [<41720948>] ? error_code+0x58/0x60
[ 462.998541] [<41050000>] ? wake_up_new_task+0x80/0x80
[ 463.003687] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.008760] [<41044fb6>] ? kthread_data+0x6/0x10
[ 463.013475] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 463.018730] [<4171fdb4>] ? __schedule+0x244/0x400
[ 463.023514] [<4102bd6b>] ? printk+0x3b/0x40
[ 463.027794] [<4102ee33>] ? do_exit+0x693/0x6d0
[ 463.032323] [<4102bd6b>] ? printk+0x3b/0x40
[ 463.036577] [<41029d47>] ? print_oops_end_marker+0x27/0x30
[ 463.042197] [<41004ac0>] ? oops_end+0x80/0x80
[ 463.046652] [<410223be>] ? no_context+0x16e/0x1f0
[ 463.051442] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 463.057166] [<4104d666>] ? check_preempt_curr+0x76/0x80
[ 463.062484] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 463.067889] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 463.072886] [<41049514>] ? hrtimer_forward+0x164/0x1b0
[ 463.078137] [<411f42e7>] ? rb_insert_color+0xb7/0xe0
[ 463.083195] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.088242] [<41720948>] ? error_code+0x58/0x60
[ 463.092888] [<41050000>] ? wake_up_new_task+0x80/0x80
[ 463.098076] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.103133] [<41044fb6>] ? kthread_data+0x6/0x10
[ 463.107830] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 463.113059] [<4171fdb4>] ? __schedule+0x244/0x400
[ 463.117843] [<410b6111>] ? kmem_cache_free+0xc1/0xf0
[ 463.122884] [<4102db51>] ? release_task+0x131/0x2e0
[ 463.127864] [<410b6111>] ? kmem_cache_free+0xc1/0xf0
[ 463.132932] [<4102eb48>] ? do_exit+0x3a8/0x6d0
[ 463.137497] [<4102db9a>] ? release_task+0x17a/0x2e0
[ 463.142469] [<4102eb48>] ? do_exit+0x3a8/0x6d0
[ 463.147017] [<4102eb7e>] ? do_exit+0x3de/0x6d0
[ 463.151557] [<41004ac0>] ? oops_end+0x80/0x80
[ 463.156027] [<410223be>] ? no_context+0x16e/0x1f0
[ 463.160818] [<4104d69c>] ? ttwu_do_wakeup+0x2c/0x70
[ 463.165773] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 463.171513] [<4104e680>] ? try_to_wake_up+0x80/0x80
[ 463.176503] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 463.181891] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 463.186871] [<4105150e>] ? update_curr+0x7e/0x110
[ 463.191652] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.196711] [<41720948>] ? error_code+0x58/0x60
[ 463.201347] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.206414] [<4104c1f9>] ? __wake_up_common+0x19/0x70
[ 463.211577] [<4104e9e2>] ? __wake_up+0x22/0x30
[ 463.216116] [<f993cd15>] ? iscsit_add_cmd_to_response_queue+0x85/0xb0 [iscsi_target_mod]
[ 463.224288] [<f9944f1c>] ? lio_queue_status+0x1c/0x20 [iscsi_target_mod]
[ 463.231084] [<f93eba92>] ? target_complete_ok_work+0x112/0x150 [target_core_mod]
[ 463.238579] [<41040706>] ? process_one_work+0xe6/0x330
[ 463.243847] [<f93eb980>] ? transport_init_session+0x60/0x60 [target_core_mod]
[ 463.251068] [<410416a9>] ? worker_thread+0xd9/0x310
[ 463.256056] [<410415d0>] ? gcwq_mayday_timeout+0x80/0x80
[ 463.261442] [<410415d0>] ? gcwq_mayday_timeout+0x80/0x80
[ 463.266848] [<41045358>] ? kthread+0x78/0x80
[ 463.271213] [<410452e0>] ? kthread_freezable_should_stop+0x20/0x20
[ 463.277482] [<417210b6>] ? kernel_thread_helper+0x6/0xd
[ 463.282798] Code: 42 1b 00 31 d2 85 c0 74 97 eb cc 90 90 90 90 90 90 90 90 90 90 a1 34 92 9f 41 8b 80 2c 01 00 00 8b 40 f8 c3 90 8b 80 2c 01 00 00 <8b> 40 fc c3 8d b6 00 00 00 00 31 c0 c3 8d b6 00 00 00 00 8d bc
[ 463.302697] EIP: [<41044fb6>] kthread_data+0x6/0x10 SS:ESP 0068:b15d98dc
[ 463.309476] CR2: 00000000fffffffc
[ 463.312821] ---[ end trace 761c4e9eccbbed90 ]---
[ 463.317441] Fixing recursive fault but reboot is needed!
[ 463.322749] BUG: unable to handle kernel paging request at fffffffc
[ 463.329069] IP: [<41044fb6>] kthread_data+0x6/0x10
[ 463.333885] *pdpt = 0000000001aab001 *pde = 0000000001aaf067 *pte = 0000000000000000
[ 463.341763] Oops: 0000 [#5]
[ 463.344652] Modules linked in: target_core_pscsi target_core_file target_core_iblock tcm_loop iscsi_target_mod tcm_qla2xxx tcm_fc target_core_mod fcoe libfcoe libfc myri10ge bnx2x bnx2 qla2xxx
[ 463.362085] Pid: 1330, comm: kworker/0:2 Tainted: G D W 3.6.11-00027-g4114d18 #122 To be filled by O.E.M. To be filled by O.E.M./P8B-C Series
[ 463.375529] EIP: 0060:[<41044fb6>] EFLAGS: 00010046 CPU: 0
[ 463.381012] EIP is at kthread_data+0x6/0x10
[ 463.385189] EAX: 00000000 EBX: f31eea40 ECX: 41a02b60 EDX: 00000000
[ 463.391484] ESI: f31eeb98 EDI: 00000000 EBP: f31eea40 ESP: b15d96d4
[ 463.397760] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 463.403164] CR0: 8005003b CR2: fffffffc CR3: 51ba5000 CR4: 000407f0
[ 463.409418] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 463.415678] DR6: ffff0ff0 DR7: 00000400
[ 463.419517] Process kworker/0:2 (pid: 1330, ti=b15d8000 task=f31eea40 task.ti=b15d8000)
[ 463.427515] Stack:
[ 463.429521] 4104146a f31eea40 f31eeb98 f31eea40 4171fdb4 0000002c 0000002c 00000001
[ 463.437415] 00000006 00000046 f31eea40 00000000 00000000 00000000 41ab4f02 0000002c
[ 463.445302] 41ab4f02 00000046 00000000 00000024 00000004 00000006 f31eea40 00000009
[ 463.453178] Call Trace:
[ 463.455620] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 463.460877] [<4171fdb4>] ? __schedule+0x244/0x400
[ 463.465683] [<4102bd6b>] ? printk+0x3b/0x40
[ 463.469974] [<4102ee33>] ? do_exit+0x693/0x6d0
[ 463.474537] [<4102bd6b>] ? printk+0x3b/0x40
[ 463.478860] [<41029d47>] ? print_oops_end_marker+0x27/0x30
[ 463.484430] [<41004ac0>] ? oops_end+0x80/0x80
[ 463.488876] [<410223be>] ? no_context+0x16e/0x1f0
[ 463.493674] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 463.499426] [<411f6ebc>] ? format_decode+0x30c/0x350
[ 463.504510] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 463.509930] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 463.514911] [<411f74b0>] ? sprintf+0x20/0x30
[ 463.519260] [<411f7fb6>] ? __delay+0x6/0x10
[ 463.523533] [<41250b69>] ? wait_for_xmitr+0x29/0x80
[ 463.528504] [<412528d1>] ? serial8250_console_putchar+0x21/0x30
[ 463.534523] [<411f7fb6>] ? __delay+0x6/0x10
[ 463.538827] [<41250b69>] ? wait_for_xmitr+0x29/0x80
[ 463.543816] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.548867] [<41720948>] ? error_code+0x58/0x60
[ 463.553485] [<41050000>] ? wake_up_new_task+0x80/0x80
[ 463.558604] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.563652] [<41044fb6>] ? kthread_data+0x6/0x10
[ 463.568384] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 463.573623] [<4171fdb4>] ? __schedule+0x244/0x400
[ 463.578397] [<4102bd6b>] ? printk+0x3b/0x40
[ 463.582667] [<4102ee33>] ? do_exit+0x693/0x6d0
[ 463.587226] [<4102bd6b>] ? printk+0x3b/0x40
[ 463.591515] [<41029d47>] ? print_oops_end_marker+0x27/0x30
[ 463.597100] [<41004ac0>] ? oops_end+0x80/0x80
[ 463.601554] [<410223be>] ? no_context+0x16e/0x1f0
[ 463.606369] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 463.612111] [<411f6ebc>] ? format_decode+0x30c/0x350
[ 463.617169] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 463.622592] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 463.627555] [<411f74b0>] ? sprintf+0x20/0x30
[ 463.631912] [<411f7fb6>] ? __delay+0x6/0x10
[ 463.636183] [<41250b69>] ? wait_for_xmitr+0x29/0x80
[ 463.641184] [<412528d1>] ? serial8250_console_putchar+0x21/0x30
[ 463.647200] [<411f7fb6>] ? __delay+0x6/0x10
[ 463.651490] [<41250b69>] ? wait_for_xmitr+0x29/0x80
[ 463.656488] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.661528] [<41720948>] ? error_code+0x58/0x60
[ 463.666173] [<41050000>] ? wake_up_new_task+0x80/0x80
[ 463.671318] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.676384] [<41044fb6>] ? kthread_data+0x6/0x10
[ 463.681081] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 463.686353] [<4171fdb4>] ? __schedule+0x244/0x400
[ 463.691144] [<4102bd6b>] ? printk+0x3b/0x40
[ 463.695424] [<4102ee33>] ? do_exit+0x693/0x6d0
[ 463.699974] [<4102bd6b>] ? printk+0x3b/0x40
[ 463.704262] [<41029d47>] ? print_oops_end_marker+0x27/0x30
[ 463.709865] [<41004ac0>] ? oops_end+0x80/0x80
[ 463.714345] [<410223be>] ? no_context+0x16e/0x1f0
[ 463.719134] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 463.724895] [<4104d666>] ? check_preempt_curr+0x76/0x80
[ 463.730193] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 463.735608] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 463.740588] [<41049514>] ? hrtimer_forward+0x164/0x1b0
[ 463.745830] [<411f42e7>] ? rb_insert_color+0xb7/0xe0
[ 463.750896] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.755969] [<41720948>] ? error_code+0x58/0x60
[ 463.760587] [<41050000>] ? wake_up_new_task+0x80/0x80
[ 463.765749] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.770800] [<41044fb6>] ? kthread_data+0x6/0x10
[ 463.775522] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 463.780754] [<4171fdb4>] ? __schedule+0x244/0x400
[ 463.785551] [<410b6111>] ? kmem_cache_free+0xc1/0xf0
[ 463.790611] [<4102db51>] ? release_task+0x131/0x2e0
[ 463.795592] [<410b6111>] ? kmem_cache_free+0xc1/0xf0
[ 463.800633] [<4102eb48>] ? do_exit+0x3a8/0x6d0
[ 463.805154] [<4102db9a>] ? release_task+0x17a/0x2e0
[ 463.810117] [<4102eb48>] ? do_exit+0x3a8/0x6d0
[ 463.814683] [<4102eb7e>] ? do_exit+0x3de/0x6d0
[ 463.819214] [<41004ac0>] ? oops_end+0x80/0x80
[ 463.823660] [<410223be>] ? no_context+0x16e/0x1f0
[ 463.828458] [<4104d69c>] ? ttwu_do_wakeup+0x2c/0x70
[ 463.833423] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 463.839181] [<4104e680>] ? try_to_wake_up+0x80/0x80
[ 463.844144] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 463.849567] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 463.854536] [<4105150e>] ? update_curr+0x7e/0x110
[ 463.859319] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.864395] [<41720948>] ? error_code+0x58/0x60
[ 463.869056] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 463.874106] [<4104c1f9>] ? __wake_up_common+0x19/0x70
[ 463.879252] [<4104e9e2>] ? __wake_up+0x22/0x30
[ 463.883799] [<f993cd15>] ? iscsit_add_cmd_to_response_queue+0x85/0xb0 [iscsi_target_mod]
[ 463.892024] [<f9944f1c>] ? lio_queue_status+0x1c/0x20 [iscsi_target_mod]
[ 463.898831] [<f93eba92>] ? target_complete_ok_work+0x112/0x150 [target_core_mod]
[ 463.906317] [<41040706>] ? process_one_work+0xe6/0x330
[ 463.911583] [<f93eb980>] ? transport_init_session+0x60/0x60 [target_core_mod]
[ 463.918819] [<410416a9>] ? worker_thread+0xd9/0x310
[ 463.923800] [<410415d0>] ? gcwq_mayday_timeout+0x80/0x80
[ 463.929206] [<410415d0>] ? gcwq_mayday_timeout+0x80/0x80
[ 463.934608] [<41045358>] ? kthread+0x78/0x80
[ 463.938993] [<410452e0>] ? kthread_freezable_should_stop+0x20/0x20
[ 463.945261] [<417210b6>] ? kernel_thread_helper+0x6/0xd
[ 463.950594] Code: 42 1b 00 31 d2 85 c0 74 97 eb cc 90 90 90 90 90 90 90 90 90 90 a1 34 92 9f 41 8b 80 2c 01 00 00 8b 40 f8 c3 90 8b 80 2c 01 00 00 <8b> 40 fc c3 8d b6 00 00 00 00 31 c0 c3 8d b6 00 00 00 00 8d bc
[ 463.970491] EIP: [<41044fb6>] kthread_data+0x6/0x10 SS:ESP 0068:b15d96d4
[ 463.977221] CR2: 00000000fffffffc
[ 463.980574] ---[ end trace 761c4e9eccbbed91 ]---
[ 463.985202] Fixing recursive fault but reboot is needed!
[ 463.990501] BUG: unable to handle kernel paging request at fffffffc
[ 463.996821] IP: [<41044fb6>] kthread_data+0x6/0x10
[ 464.001655] *pdpt = 0000000001aab001 *pde = 0000000001aaf067 *pte = 0000000000000000
[ 464.009543] Oops: 0000 [#6]
[ 464.012456] Modules linked in: target_core_pscsi target_core_file target_core_iblock tcm_loop iscsi_target_mod tcm_qla2xxx tcm_fc target_core_mod fcoe libfcoe libfc myri10ge bnx2x bnx2 qla2xxx
[ 464.029922] Pid: 1330, comm: kworker/0:2 Tainted: G D W 3.6.11-00027-g4114d18 #122 To be filled by O.E.M. To be filled by O.E.M./P8B-C Series
[ 464.043351] EIP: 0060:[<41044fb6>] EFLAGS: 00010046 CPU: 0
[ 464.048836] EIP is at kthread_data+0x6/0x10
[ 464.053036] EAX: 00000000 EBX: f31eea40 ECX: 41a02b60 EDX: 00000000
[ 464.059296] ESI: f31eeb98 EDI: 00000000 EBP: f31eea40 ESP: b15d94cc
[ 464.065567] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 464.070970] CR0: 8005003b CR2: fffffffc CR3: 51ba5000 CR4: 000407f0
[ 464.077240] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 464.083508] DR6: ffff0ff0 DR7: 00000400
[ 464.087348] Process kworker/0:2 (pid: 1330, ti=b15d8000 task=f31eea40 task.ti=b15d8000)
[ 464.095362] Stack:
[ 464.097378] 4104146a f31eea40 f31eeb98 f31eea40 4171fdb4 0000002c 0000002c 00000001
[ 464.105246] 00000006 00000046 f31eea40 00000000 00000000 00000000 41ab4f02 0000002c
[ 464.113123] 41ab4f02 00000046 00000000 00000024 00000004 00000006 f31eea40 00000009
[ 464.121011] Call Trace:
[ 464.123485] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 464.128707] [<4171fdb4>] ? __schedule+0x244/0x400
[ 464.133515] [<4102bd6b>] ? printk+0x3b/0x40
[ 464.137796] [<4102ee33>] ? do_exit+0x693/0x6d0
[ 464.142352] [<4102bd6b>] ? printk+0x3b/0x40
[ 464.146613] [<41029d47>] ? print_oops_end_marker+0x27/0x30
[ 464.152208] [<41004ac0>] ? oops_end+0x80/0x80
[ 464.156646] [<410223be>] ? no_context+0x16e/0x1f0
[ 464.161443] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 464.167196] [<411f6ebc>] ? format_decode+0x30c/0x350
[ 464.172278] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 464.177684] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 464.182664] [<411f74b0>] ? sprintf+0x20/0x30
[ 464.187040] [<411f7fb6>] ? __delay+0x6/0x10
[ 464.191310] [<41250b69>] ? wait_for_xmitr+0x29/0x80
[ 464.196284] [<412528d1>] ? serial8250_console_putchar+0x21/0x30
[ 464.202301] [<411f7fb6>] ? __delay+0x6/0x10
[ 464.206591] [<41250b69>] ? wait_for_xmitr+0x29/0x80
[ 464.211572] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 464.216662] [<41720948>] ? error_code+0x58/0x60
[ 464.221299] [<41050000>] ? wake_up_new_task+0x80/0x80
[ 464.226460] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 464.231536] [<41044fb6>] ? kthread_data+0x6/0x10
[ 464.236239] [<4104146a>] ? wq_worker_sleeping+0xa/0x60
[ 464.241488] [<4171fdb4>] ? __schedule+0x244/0x400
[ 464.246306] [<4102bd6b>] ? printk+0x3b/0x40
[ 464.250569] [<4102ee33>] ? do_exit+0x693/0x6d0
[ 464.255118] [<4102bd6b>] ? printk+0x3b/0x40
[ 464.259388] [<41029d47>] ? print_oops_end_marker+0x27/0x30
[ 464.264984] [<41004ac0>] ? oops_end+0x80/0x80
[ 464.269445] [<410223be>] ? no_context+0x16e/0x1f0
[ 464.274244] [<410225fa>] ? __bad_area_nosemaphore+0x5a/0x130
[ 464.279993] [<411f6ebc>] ? format_decode+0x30c/0x350
[ 464.285087] [<4102276f>] ? bad_area_nosemaphore+0xf/0x20
[ 464.290490] [<41022a12>] ? do_page_fault+0xb2/0x430
[ 464.295482] [<411f74b0>] ? sprintf+0x20/0x30
[ 464.299838] [<411f7fb6>] ? __delay+0x6/0x10
[ 464.304102] [<41250b69>] ? wait_for_xmitr+0x29/0x80
[ 464.309056] [<412528d1>] ? serial8250_console_putchar+0x21/0x30
[ 464.315074] [<411f7fb6>] ? __delay+0x6/0x10
[ 464.319346] [<41250b69>] ? wait_for_xmitr+0x29/0x80
[ 464.324310] [<41022960>] ? vmalloc_fault+0x1e0/0x1e0
[ 464.329375] [<41720948>] ? error_code+0x58/0x60
[ 464.3340
