Good day,
Running into a simultaneous crash on two Pacemaker nodes with Ubuntu
14.04.1 and kernel 3.19.4. Here is a snippet of what happens just
before the crash. LIO is providing LUNs to ESXi 5.5
Thank you,
Alex Gorbachev
Apr 19 03:29:51 roc-4r-scd212 kernel: [74474.919463] libceph: osd28
10.80.3.30:6800 socket closed (con state OPEN)
Apr 19 03:29:55 roc-4r-scd212 kernel: [74478.776997]
TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000000b
Apr 19 03:29:55 roc-4r-scd212 kernel: [74478.777071] ABORT_TASK: Found
referenced iSCSI task_tag: 199691
Apr 19 03:29:55 roc-4r-scd212 kernel: [74478.777074] ABORT_TASK:
ref_tag: 199691 already complete, skipping
Apr 19 03:29:55 roc-4r-scd212 kernel: [74478.777076] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 199691
Apr 19 03:29:55 roc-4r-scd212 kernel: [74478.777078] ABORT_TASK: Found
referenced iSCSI task_tag: 199692
Apr 19 03:29:55 roc-4r-scd212 kernel: [74478.777080] ABORT_TASK:
ref_tag: 199692 already complete, skipping
Apr 19 03:29:55 roc-4r-scd212 kernel: [74478.777082] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 199692
Apr 19 03:29:55 roc-4r-scd212 kernel: [74478.777085] ABORT_TASK: Found
referenced iSCSI task_tag: 199690
Apr 19 03:29:55 roc-4r-scd212 kernel: [74478.777087] ABORT_TASK:
ref_tag: 199690 already complete, skipping
Apr 19 03:29:55 roc-4r-scd212 kernel: [74478.777089] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 199690
Apr 19 03:30:00 roc-4r-scd212 kernel: [74483.295268] ABORT_TASK: Found
referenced iSCSI task_tag: 257197
Apr 19 03:30:00 roc-4r-scd212 kernel: [74483.295274] ABORT_TASK:
Sending TMR_FUNCTION_COMPLETE for ref_tag: 257197
Apr 19 03:30:00 roc-4r-scd212 kernel: [74483.295279] Unable to locate
ITT: 0x0003ecad on CID: 0
Apr 19 03:30:00 roc-4r-scd212 kernel: [74483.295279] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 257197
Apr 19 03:30:00 roc-4r-scd212 kernel: [74483.295308] Unable to locate
RefTaskTag: 0x0003ecad on CID: 0.
Apr 19 03:30:02 roc-4r-scd212 kernel: [74486.177458] ABORT_TASK: Found
referenced iSCSI task_tag: 9072
Apr 19 03:30:02 roc-4r-scd212 kernel: [74486.177460] Unexpected ret:
-32 send data 48
Apr 19 03:30:02 roc-4r-scd212 kernel: [74486.177488] ABORT_TASK:
Sending TMR_FUNCTION_COMPLETE for ref_tag: 9072
Apr 19 03:30:10 roc-4r-scd212 kernel: [74493.812492] Unexpected ret:
-32 send data 48
Apr 19 03:30:12 roc-4r-scd212 kernel: [74495.813309]
TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000015
Apr 19 03:30:12 roc-4r-scd212 kernel: [74495.987564] libceph: osd25
10.80.3.30:6809 socket closed (con state OPEN)
Apr 19 03:30:15 roc-4r-scd212 kernel: [74498.326272] Unexpected ret:
-32 send data 48
Apr 19 03:30:17 roc-4r-scd212 kernel: [74500.983349] iSCSI Login
timeout on Network Portal 10.70.2.211:3260
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.167913] NMI watchdog:
BUG: soft lockup - CPU#9 stuck for 22s! [iscsi_trx:13963]
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.167959] Modules linked
in: target_core_user uio rbd libceph libcrc32c iscsi_target_mod
target_core_file target_core_pscsi target_core_iblock target_core_mod
configfs xt_multiport iptable_filter ip_tables x_tables
enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE)
ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp
kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac joydev
edac_core mei_me mei lpc_ich ioatdma ipmi_si ipmi_msghandler
8250_fintek wmi mac_hid 8021q garp mrp stp llc bonding lp parport nfsd
auth_rpcgss nfs_acl nfs lockd grace sunrpc fscache mlx4_en vxlan
ip6_udp_tunnel udp_tunnel hid_generic igb usbhid mpt2sas i2c_algo_bit
ahci hid dca mlx4_core raid_class ptp libahci scsi_transport_sas
pps_core
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168026] CPU: 9 PID: 13963
Comm: iscsi_trx Tainted: G C OE 3.19.4-031904-generic
#201504131440
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168028] Hardware name:
Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a
12/05/2013
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168030] task:
ffff8808591a1d70 ti: ffff880859620000 task.ti: ffff880859620000
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168032] RIP:
0010:[<ffffffff8107ae70>] [<ffffffff8107ae70>]
__local_bh_enable_ip+0x0/0x90
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168041] RSP:
0018:ffff880859623d70 EFLAGS: 00000216
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168043] RAX:
ffff88104ae4de80 RBX: ffff8810582a5000 RCX: ffff88104ae4de80
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168044] RDX:
ffff88104ae4de80 RSI: 0000000000000200 RDI: ffffffffc07cb25b
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168046] RBP:
ffff880859623d78 R08: ffff88104ae4df50 R09: 0000000000000101
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168048] R10:
0000000000000001 R11: dead000000200200 R12: ffffffffc07c0f68
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168049] R13:
ffff880859623d08 R14: ffffffff817d2e30 R15: ffff880859623cd8
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168052] FS:
0000000000000000(0000) GS:ffff88107fc60000(0000)
knlGS:0000000000000000
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168054] CS: 0010 DS:
0000 ES: 0000 CR0: 0000000080050033
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168055] CR2:
00007f8e74ed7e60 CR3: 0000000001c15000 CR4: 00000000001407e0
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168057] Stack:
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168059] ffffffff817d2e30
ffff880859623dd8 ffffffffc07cb25b 0000000000000000
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168062] ffff880842942000
ffff8810582a53e0 ffff8810582a5440 ffff8808591a1d70
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168065] ffff8810582a5000
ffff8810582a53f4 ffff880859623e4c ffff8808591a1d70
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168069] Call Trace:
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168076]
[<ffffffff817d2e30>] ? _raw_spin_unlock_bh+0x20/0x50
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168094]
[<ffffffffc07cb25b>] iscsit_close_connection+0x3ab/0x660
[iscsi_target_mod]
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168102]
[<ffffffffc07b69b3>] iscsit_take_action_for_connection_exit+0x83/0x110
[iscsi_target_mod]
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168111]
[<ffffffffc07ca33e>] iscsi_target_rx_thread+0x22e/0x320
[iscsi_target_mod]
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168119]
[<ffffffffc07ca110>] ? iscsi_target_tx_thread+0x220/0x220
[iscsi_target_mod]
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168124]
[<ffffffff81095e29>] kthread+0xc9/0xe0
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168128]
[<ffffffff81095d60>] ? flush_kthread_worker+0x90/0x90
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168132]
[<ffffffff817d3718>] ret_from_fork+0x58/0x90
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168135]
[<ffffffff81095d60>] ? flush_kthread_worker+0x90/0x90
Apr 19 03:30:29 roc-4r-scd212 kernel: [74513.168136] Code: 1f 44 00 00
65 8b 05 70 64 f9 7e 85 c0 75 14 48 89 df 57 9d 0f 1f 44 00 00 48 83
c4 08 5b 5d c3 0f 1f 00 e8 b3 a6 75 00 eb e5 90 <0f> 1f 44 00 00 55 48
89 e5 53 89 f3 48 83 ec 08 65 8b 05 19 0a
Apr 19 03:30:35 roc-4r-scd212 kernel: [74519.038238] ABORT_TASK: Found
referenced iSCSI task_tag: 9823
Apr 19 03:30:35 roc-4r-scd212 kernel: [74519.038241] ABORT_TASK:
ref_tag: 9823 already complete, skipping
Apr 19 03:30:35 roc-4r-scd212 kernel: [74519.038243] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 9823
Apr 19 03:30:35 roc-4r-scd212 kernel: [74519.038249] ABORT_TASK: Found
referenced iSCSI task_tag: 9821
Apr 19 03:30:35 roc-4r-scd212 kernel: [74519.038256] ABORT_TASK:
Sending TMR_FUNCTION_COMPLETE for ref_tag: 9821
Apr 19 03:30:35 roc-4r-scd212 kernel: [74519.038258] ABORT_TASK: Found
referenced iSCSI task_tag: 9822
Apr 19 03:30:35 roc-4r-scd212 kernel: [74519.038259] ABORT_TASK:
ref_tag: 9822 already complete, skipping
Apr 19 03:30:35 roc-4r-scd212 kernel: [74519.038261] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 9822
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812704] ABORT_TASK: Found
referenced iSCSI task_tag: 195485
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812708] ABORT_TASK:
ref_tag: 195485 already complete, skipping
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812709] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 195485
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812711] ABORT_TASK: Found
referenced iSCSI task_tag: 195483
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812713] ABORT_TASK:
ref_tag: 195483 already complete, skipping
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812714] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 195483
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812717] ABORT_TASK: Found
referenced iSCSI task_tag: 195484
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812720] ABORT_TASK:
ref_tag: 195484 already complete, skipping
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812721] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 195484
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812729] ABORT_TASK: Found
referenced iSCSI task_tag: 195483
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812731] ABORT_TASK:
ref_tag: 195483 already complete, skipping
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812732] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 195483
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812733] ABORT_TASK: Found
referenced iSCSI task_tag: 195482
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812736] ABORT_TASK: Found
referenced iSCSI task_tag: 195485
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812737] ABORT_TASK:
ref_tag: 195485 already complete, skipping
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812739] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 195485
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812740] ABORT_TASK:
Sending TMR_FUNCTION_COMPLETE for ref_tag: 195482
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812752] ABORT_TASK: Found
referenced iSCSI task_tag: 195483
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812754] ABORT_TASK:
ref_tag: 195483 already complete, skipping
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812755] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 195483
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812756] ABORT_TASK: Found
referenced iSCSI task_tag: 195484
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812758] ABORT_TASK:
ref_tag: 195484 already complete, skipping
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812759] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 195484
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812764] ABORT_TASK: Found
referenced iSCSI task_tag: 195485
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812765] ABORT_TASK:
ref_tag: 195485 already complete, skipping
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812767] ABORT_TASK:
Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 195485
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812770] Unable to locate
ITT: 0x0002fb9a on CID: 0
Apr 19 03:30:52 roc-4r-scd212 kernel: [74535.812771] Unable to locate
RefTaskTag: 0x0002fb9a on CID: 0.
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178677] NMI watchdog:
BUG: soft lockup - CPU#9 stuck for 22s! [iscsi_trx:13963]
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178726] Modules linked
in: target_core_user uio rbd libceph libcrc32c iscsi_target_mod
target_core_file target_core_pscsi target_core_iblock target_core_mod
configfs xt_multiport iptable_filter ip_tables x_tables
enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE)
ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp
kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac joydev
edac_core mei_me mei lpc_ich ioatdma ipmi_si ipmi_msghandler
8250_fintek wmi mac_hid 8021q garp mrp stp llc bonding lp parport nfsd
auth_rpcgss nfs_acl nfs lockd grace sunrpc fscache mlx4_en vxlan
ip6_udp_tunnel udp_tunnel hid_generic igb usbhid mpt2sas i2c_algo_bit
ahci hid dca mlx4_core raid_class ptp libahci scsi_transport_sas
pps_core
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178788] CPU: 9 PID: 13963
Comm: iscsi_trx Tainted: G C OEL 3.19.4-031904-generic
#201504131440
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178790] Hardware name:
Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a
12/05/2013
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178792] task:
ffff8808591a1d70 ti: ffff880859620000 task.ti: ffff880859620000
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178794] RIP:
0010:[<ffffffff817d3291>] [<ffffffff817d3291>]
_raw_spin_lock_bh+0x11/0x70
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178804] RSP:
0018:ffff880859623cd8 EFLAGS: 00000206
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178806] RAX:
0000000000000001 RBX: 0000000000000001 RCX: 00000000ef0708fc
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178807] RDX:
000000000000ee0c RSI: ffff8810582a5000 RDI: ffff8810582a53e8
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178809] RBP:
ffff880859623d08 R08: ffff88104ae4df50 R09: 0000000000000101
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178811] R10:
0000000000000001 R11: dead000000200200 R12: 0000000000000001
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178812] R13:
ffff880859623cc8 R14: ffff88104ae4de80 R15: ffff880859623cc8
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178815] FS:
0000000000000000(0000) GS:ffff88107fc60000(0000)
knlGS:0000000000000000
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178817] CS: 0010 DS:
0000 ES: 0000 CR0: 0000000080050033
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178818] CR2:
00007f8e74ed7e60 CR3: 0000000001c15000 CR4: 00000000001407e0
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178820] Stack:
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178822] ffff880859623d08
ffffffffc07c0d85 ffff88104ae4dc80 ffff8810582a5000
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178825] 0000000000000001
ffff88104ae4de90 ffff880859623d38 ffffffffc07c2180
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178828] ffffffffc07c2130
ffff88104ae4dc80 0000000000000001 0000000000000001
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178831] Call Trace:
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178848]
[<ffffffffc07c0d85>] ?
iscsit_remove_cmd_from_immediate_queue+0x25/0x120 [iscsi_target_mod]
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178858]
[<ffffffffc07c2180>] __iscsit_free_cmd+0x50/0xa0 [iscsi_target_mod]
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178865]
[<ffffffffc07c2130>] ? iscsit_free_queue_reqs_for_conn+0x150/0x150
[iscsi_target_mod]
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178872]
[<ffffffffc07c228e>] iscsit_free_cmd+0xbe/0x160 [iscsi_target_mod]
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178880]
[<ffffffffc07b5593>] ? iscsit_increment_maxcmdsn+0x13/0x90
[iscsi_target_mod]
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178889]
[<ffffffffc07cb274>] iscsit_close_connection+0x3c4/0x660
[iscsi_target_mod]
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178896]
[<ffffffffc07b69b3>] iscsit_take_action_for_connection_exit+0x83/0x110
[iscsi_target_mod]
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178904]
[<ffffffffc07ca33e>] iscsi_target_rx_thread+0x22e/0x320
[iscsi_target_mod]
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178911]
[<ffffffffc07ca110>] ? iscsi_target_tx_thread+0x220/0x220
[iscsi_target_mod]
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178918]
[<ffffffff81095e29>] kthread+0xc9/0xe0
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178921]
[<ffffffff81095d60>] ? flush_kthread_worker+0x90/0x90
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178925]
[<ffffffff817d3718>] ret_from_fork+0x58/0x90
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178928]
[<ffffffff81095d60>] ? flush_kthread_worker+0x90/0x90
Apr 19 03:30:57 roc-4r-scd212 kernel: [74541.178930] Code: 83 e0 fe 8d
70 02 0f b7 f6 0f 1f 80 00 00 00 00 eb ce 0f 1f 84 00 00 00 00 00 0f
1f 44 00 00 55 65 81 05 0f 86 83 7e 00 02 00 00 <48> 89 e5 b8 00 00 02
00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 75
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248624] INFO: rcu_sched
self-detected stall on CPU { 9} (t=15000 jiffies g=846173 c=846172
q=0)
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248684] Task dump for CPU 9:
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248686] iscsi_trx R
running task 0 13963 2 0x0000000c
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248689] ffffffff81c54cc0
ffff88107fc63d18 ffffffff810a3444 0000000000000009
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248692] ffffffff81c54cc0
ffff88107fc63d38 ffffffff810a60bf 0000000000000087
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248695] ffffffff81c54dc0
ffff88107fc63d68 ffffffff810d68fb ffffffff81c54cc0
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248699] Call Trace:
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248700] <IRQ>
[<ffffffff810a3444>] sched_show_task+0xb4/0x120
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248710]
[<ffffffff810a60bf>] dump_cpu_task+0x3f/0x50
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248717]
[<ffffffff810d68fb>] rcu_dump_cpu_stacks+0x8b/0xc0
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248721]
[<ffffffff810d84d7>] print_cpu_stall+0xd7/0x140
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248725]
[<ffffffff810d8ef5>] check_cpu_stall.isra.55+0xa5/0xb0
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248728]
[<ffffffff810d8f38>] __rcu_pending+0x38/0x170
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248732]
[<ffffffff810daa8d>] rcu_check_callbacks+0x9d/0x190
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248735]
[<ffffffff810dfdf8>] update_process_times+0x38/0x70
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248742]
[<ffffffff810efab3>] tick_sched_handle.isra.12+0x33/0x70
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248745]
[<ffffffff810efbeb>] tick_sched_timer+0x4b/0x80
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248749]
[<ffffffff810e08b6>] __run_hrtimer+0x76/0x230
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248752]
[<ffffffff810efba0>] ? tick_nohz_handler+0xb0/0xb0
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248756]
[<ffffffff810e10bf>] hrtimer_interrupt+0xff/0x260
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248763]
[<ffffffff8104c319>] local_apic_timer_interrupt+0x39/0x60
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248768]
[<ffffffff817d6805>] smp_apic_timer_interrupt+0x45/0x60
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248772]
[<ffffffff817d487d>] apic_timer_interrupt+0x6d/0x80
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248773] <EOI>
[<ffffffffc0765d41>] ? transport_wait_for_tasks+0xc1/0x150
[target_core_mod]
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248798]
[<ffffffffc0765d3b>] ? transport_wait_for_tasks+0xbb/0x150
[target_core_mod]
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248806]
[<ffffffffc07c0f68>] ?
iscsit_remove_cmd_from_response_queue+0xe8/0x120 [iscsi_target_mod]
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248817]
[<ffffffffc0767cf5>] transport_generic_free_cmd+0xc5/0xe0
[target_core_mod]
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248825]
[<ffffffffc07cb25b>] ? iscsit_close_connection+0x3ab/0x660
[iscsi_target_mod]
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248833]
[<ffffffffc07c2266>] iscsit_free_cmd+0x96/0x160 [iscsi_target_mod]
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248840]
[<ffffffffc07cb274>] iscsit_close_connection+0x3c4/0x660
[iscsi_target_mod]
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248847]
[<ffffffffc07b69b3>] iscsit_take_action_for_connection_exit+0x83/0x110
[iscsi_target_mod]
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248855]
[<ffffffffc07ca33e>] iscsi_target_rx_thread+0x22e/0x320
[iscsi_target_mod]
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248862]
[<ffffffffc07ca110>] ? iscsi_target_tx_thread+0x220/0x220
[iscsi_target_mod]
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248865]
[<ffffffff81095e29>] kthread+0xc9/0xe0
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248869]
[<ffffffff81095d60>] ? flush_kthread_worker+0x90/0x90
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248872]
[<ffffffff817d3718>] ret_from_fork+0x58/0x90
Apr 19 03:31:02 roc-4r-scd212 kernel: [74546.248875]
[<ffffffff81095d60>] ? flush_kthread_worker+0x90/0x90
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.190977] NMI watchdog:
BUG: soft lockup - CPU#9 stuck for 22s! [iscsi_trx:13963]
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191023] Modules linked
in: target_core_user uio rbd libceph libcrc32c iscsi_target_mod
target_core_file target_core_pscsi target_core_iblock target_core_mod
configfs xt_multiport iptable_filter ip_tables x_tables
enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE)
ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp
kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac joydev
edac_core mei_me mei lpc_ich ioatdma ipmi_si ipmi_msghandler
8250_fintek wmi mac_hid 8021q garp mrp stp llc bonding lp parport nfsd
auth_rpcgss nfs_acl nfs lockd grace sunrpc fscache mlx4_en vxlan
ip6_udp_tunnel udp_tunnel hid_generic igb usbhid mpt2sas i2c_algo_bit
ahci hid dca mlx4_core raid_class ptp libahci scsi_transport_sas
pps_core
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191069] CPU: 9 PID: 13963
Comm: iscsi_trx Tainted: G C OEL 3.19.4-031904-generic
#201504131440
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191071] Hardware name:
Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a
12/05/2013
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191073] task:
ffff8808591a1d70 ti: ffff880859620000 task.ti: ffff880859620000
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191075] RIP:
0010:[<ffffffff817d2e22>] [<ffffffff817d2e22>]
_raw_spin_unlock_bh+0x12/0x50
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191079] RSP:
0018:ffff880859623cd8 EFLAGS: 00000286
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191081] RAX:
0000000000000000 RBX: ffffffffffffff10 RCX: ffff88104ae4de80
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191083] RDX:
000000000000bcec RSI: ffff8810582a5000 RDI: ffff8810582a53f0
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191084] RBP:
ffff880859623cd8 R08: ffff88104ae4df50 R09: 0000000000000101
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191086] R10:
0000000000000001 R11: dead000000200200 R12: ffff88104ae4de90
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191088] R13:
00000000e526f32a R14: 00000000e526f329 R15: 00000000e526f32a
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191090] FS:
0000000000000000(0000) GS:ffff88107fc60000(0000)
knlGS:0000000000000000
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191092] CS: 0010 DS:
0000 ES: 0000 CR0: 0000000080050033
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191093] CR2:
00007f8e74ed7e60 CR3: 0000000001c15000 CR4: 00000000001407e0
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191095] Stack:
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191096] ffff880859623d08
ffffffffc07c0f68 ffff88104ae4dc80 ffff8810582a5000
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191099] 0000000000000001
ffff88104ae4de90 ffff880859623d38 ffffffffc07c218b
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191102] 00000000d210d210
ffff88104ae4dc80 0000000000000001 0000000000000001
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191105] Call Trace:
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191114]
[<ffffffffc07c0f68>] iscsit_remove_cmd_from_response_queue+0xe8/0x120
[iscsi_target_mod]
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191122]
[<ffffffffc07c218b>] __iscsit_free_cmd+0x5b/0xa0 [iscsi_target_mod]
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191129]
[<ffffffffc07c225b>] iscsit_free_cmd+0x8b/0x160 [iscsi_target_mod]
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191137]
[<ffffffffc07cb274>] iscsit_close_connection+0x3c4/0x660
[iscsi_target_mod]
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191145]
[<ffffffffc07b69b3>] iscsit_take_action_for_connection_exit+0x83/0x110
[iscsi_target_mod]
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191153]
[<ffffffffc07ca33e>] iscsi_target_rx_thread+0x22e/0x320
[iscsi_target_mod]
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191160]
[<ffffffffc07ca110>] ? iscsi_target_tx_thread+0x220/0x220
[iscsi_target_mod]
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191163]
[<ffffffff81095e29>] kthread+0xc9/0xe0
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191167]
[<ffffffff81095d60>] ? flush_kthread_worker+0x90/0x90
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191170]
[<ffffffff817d3718>] ret_from_fork+0x58/0x90
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191173]
[<ffffffff81095d60>] ? flush_kthread_worker+0x90/0x90
Apr 19 03:31:29 roc-4r-scd212 kernel: [74573.191174] Code: 08 5d c3 66
0f 1f 44 00 00 e8 9b c7 8e ff 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f
44 00 00 55 48 89 e5 0f 1f 44 00 00 66 83 07 02 <48> 8b 7d 08 be 00 02
00 00 e8 40 80 8a ff 5d c3 66 0f 1f 44 00
Apr 19 03:31:57 roc-4r-scd212 kernel: [74601.201742] NMI watchdog:
BUG: soft lockup - CPU#9 stuck for 22s! [iscsi_trx:13963]
Apr 19 03:31:57 roc-4r-scd212 kernel: [74601.201789] Modules linked
in: target_core_user uio rbd libceph libcrc32c iscsi_target_mod
target_core_file target_core_pscsi target_core_iblock target_core_mod
configfs xt_multiport iptable_filter ip_tables x_tables
enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE)
ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp
kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel
aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac joydev
edac_core mei_me mei lpc_ich ioatdma ipmi_si ipmi_msghandler
8250_fintek wmi mac_hid 8021q garp mrp stp llc bonding lp parport nfsd
auth_rpcgss nfs_acl nfs lockd grace sunrpc fscache mlx4_en vxlan
ip6_udp_tunnel udp_tunnel hid_generic igb usbhid mpt2sas i2c_algo_bit
ahci hid dca mlx4_core raid_class ptp libahci scsi_transport_sas
pps_core
Apr 19 03:31:57 roc-4r-scd212 kernel: [74601.201835] CPU: 9 PID: 13963
Comm: iscsi_trx Tainted: G C OEL 3.19.4-031904-generic
#201504131440
Apr 19 03:31:57 roc-4r-scd212 kernel: [74601.201837] Hardware name:
Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a
12/05/2013
Apr 19 03:31:57 roc-4r-scd212 kernel: [74601.201839] task:
ffff8808591a1d70 ti: ffff880859620000 task.ti: ffff880859620000
Apr 19 03:31:57 roc-4r-scd212 kernel: [74601.201840] RIP:
0010:[<ffffffff8107aeb8>] [<ffffffff8107aeb8>]
__local_bh_enable_ip+0x48/0x90
Apr 19 03:31:57 roc-4r-scd212 kernel: [74601.201846] RSP:
0018:ffff880859623cb8 EFLAGS: 00000246
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
