On 06/03/2015 11:53 PM, Nicholas A. Bellinger wrote:
>
> Out of curiosity, are RCU stalls still occurring later in the logs..?
> If they are, then I'm still leaning toward a bug outside of iscsi-target
> code.
>
> Just because the NMI watching shows a hung task earlier than the RCU
> stall itself, doesn't mean the first hung task is the root cause.
I suspect you are right, because we are not getting any warnings on the
ceph side about slow IOs and we set that timeout to around 10 or 15
seconds
>
> So more preceding and subsequent context of the log would be helpful..
Alex sent me the log offlist. Still seeing rcu warnings.
We see ABORT_TASK, "Detected NON_EXISTENT_LUN Access",
"Unexpected ret: -32 send data 48" and
"Unexpected ret: -32 send data 148" messages every couple minutes for 5
hours then we see:
Jun 3 05:01:18 roc-4r-scd212 kernel: [69956.793246] ABORT_TASK: Found referenced iSCSI task_tag: 9185786
Jun 3 05:01:18 roc-4r-scd212 kernel: [69956.793252] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 9185786
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283037] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000003e
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283142] ABORT_TASK: Found referenced iSCSI task_tag: 767331
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283145] ABORT_TASK: ref_tag: 767331 already complete, skipping
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283146] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 767331
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283149] ABORT_TASK: Found referenced iSCSI task_tag: 767332
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283151] ABORT_TASK: ref_tag: 767332 already complete, skipping
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283152] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 767332
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283154] ABORT_TASK: Found referenced iSCSI task_tag: 767330
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283157] Unexpected ret: -32 send data 48
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283183] ABORT_TASK: ref_tag: 767330 already complete, skipping
Jun 3 05:01:31 roc-4r-scd212 kernel: [69969.283185] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 767330
Jun 3 05:01:38 roc-4r-scd212 kernel: [69976.231851] libceph: osd22 10.80.3.25:6815 socket closed (con state OPEN)
Jun 3 05:01:42 roc-4r-scd212 kernel: [69980.101342] libceph: osd15 10.80.3.25:6827 socket closed (con state OPEN)
Jun 3 05:01:46 roc-4r-scd212 kernel: [69984.290037] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000002b
Jun 3 05:01:46 roc-4r-scd212 kernel: [69984.290189] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000002c
.. the non existent lun message go for a while up to LUN ff.
Jun 3 05:01:58 roc-4r-scd212 kernel: [69996.513146] Exiting Time2Retain handler because session_reinstatement=1
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.815905] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000021
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.816061] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000022
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.816186] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000023
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.816300] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000024
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.816417] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000025
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.816532] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000026
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.816644] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000027
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.816758] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000028
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.816870] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000029
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.816982] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000002a
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.817109] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000002b
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.817222] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000002c
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.817334] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000002d
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.817446] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000002e
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.817557] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x0000002f
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.817669] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000030
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.817781] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000031
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.817892] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000032
Jun 3 05:02:39 roc-4r-scd212 kernel: [70037.818003] TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000033
Then kablewy we eventually get stuck:
Jun 3 05:02:53 roc-4r-scd212 kernel: [70051.125653] ABORT_TASK: Found referenced iSCSI task_tag: 27972
Jun 3 05:02:53 roc-4r-scd212 kernel: [70051.125657] Unexpected ret: -32 send data 148
Jun 3 05:02:53 roc-4r-scd212 kernel: [70051.125690] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 27972
Jun 3 05:02:54 roc-4r-scd212 kernel: [70052.762739] Unexpected ret: -32 send data 48
Jun 3 05:03:04 roc-4r-scd212 kernel: [70062.430477] Unexpected ret: -32 send data 48
Jun 3 05:03:08 roc-4r-scd212 kernel: [70066.015830] iSCSI Login timeout on Network Portal 10.70.2.211:3260
Jun 3 05:03:08 roc-4r-scd212 kernel: [70066.015834] Unexpected ret: -32 send data 48
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255353] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [iscsi_trx:7052]
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255401] Modules linked in: rbd libceph libcrc32c iscsi_target_mod target_core_file target_core_pscsi target_core_iblock target_core_mod configfs xt_multiport iptable_filter ip_tables x_tables enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE) ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core joydev mei_me lpc_ich mei ioatdma 8021q garp mrp ipmi_si stp 8250_fintek ipmi_msghandler llc bonding shpchp wmi lp mac_hid parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic igb usbhid ahci libahci hid mpt2sas i2c_algo_bit mlx4_core dca ptp raid_class pps_core scsi_transport_sas
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255462] CPU: 3 PID: 7052 Comm: iscsi_trx Tainted: G C OE 4.1.0-040100rc2-generic #201505032335
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255464] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255467] task: ffff880859198000 ti: ffff880079aa4000 task.ti: ffff880079aa4000
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255469] RIP: 0010:[<ffffffff811e6a96>] [<ffffffff811e6a96>] kfree+0x6/0x170
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255477] RSP: 0018:ffff880079aa7ce8 EFLAGS: 00000246
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255480] RAX: 000000000008090d RBX: 0000000000000004 RCX: ffff881049214e80
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255481] RDX: 000000000000dc2a RSI: 0000000000000282 RDI: 0000000000000000
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255483] RBP: ffff880079aa7d28 R08: ffff881049214f50 R09: 0000000000000101
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255485] R10: 0000000000000004 R11: dead000000200200 R12: ffff881049214f50
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255487] R13: ffff880079aa7ce8 R14: 0000000000000004 R15: dead000000200200
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255490] FS: 0000000000000000(0000) GS:ffff88085fac0000(0000) knlGS:0000000000000000
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255492] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255494] CR2: 0000000000b30fe4 CR3: 0000000079b9f000 CR4: 00000000001407e0
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255495] Stack:
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255497] ffff880079aa7d28 ffffffffc0455da1 ffff880079aa7d28 ffff881049214e90
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255501] 0000000000000001 0000000000000001 ffff881049214e90 ffff881049214e90
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255504] ffff880079aa7d58 ffffffffc0455f84 0000000000000292 ffff881049214c80
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255507] Call Trace:
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255526] [<ffffffffc0455da1>] ? transport_put_cmd+0x121/0x290 [target_core_mod]
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255536] [<ffffffffc0455f84>] transport_generic_free_cmd+0x74/0xe0 [target_core_mod]
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255547] [<ffffffffc04ae716>] iscsit_free_cmd+0x96/0x160 [iscsi_target_mod]
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255555] [<ffffffffc04b78dc>] iscsit_close_connection+0x47c/0x770 [iscsi_target_mod]
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255562] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255570] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255575] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255582] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255586] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255590] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255593] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255596] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:03:17 roc-4r-scd212 kernel: [70075.255597] Code: 00 00 00 00 00 4c 89 ef e8 88 fe ff ff 48 83 c4 08 31 c0 5b 41 5c 41 5d 5d c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 <48> 89 e5 48 83 ec 30 48 89 5d d8 4c 89 75 f0 48 89 fb 4c 89 65
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266100] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [iscsi_trx:7052]
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266147] Modules linked in: rbd libceph libcrc32c iscsi_target_mod target_core_file target_core_pscsi target_core_iblock target_core_mod configfs xt_multiport iptable_filter ip_tables x_tables enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE) ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core joydev mei_me lpc_ich mei ioatdma 8021q garp mrp ipmi_si stp 8250_fintek ipmi_msghandler llc bonding shpchp wmi lp mac_hid parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic igb usbhid ahci libahci hid mpt2sas i2c_algo_bit mlx4_core dca ptp raid_class pps_core scsi_transport_sas
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266204] CPU: 3 PID: 7052 Comm: iscsi_trx Tainted: G C OEL 4.1.0-040100rc2-generic #201505032335
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266206] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266208] task: ffff880859198000 ti: ffff880079aa4000 task.ti: ffff880079aa4000
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266210] RIP: 0010:[<ffffffffc04ad897>] [<ffffffffc04ad897>] iscsit_free_r2ts_from_list+0x67/0x80 [iscsi_target_mod]
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266224] RSP: 0018:ffff880079aa7d08 EFLAGS: 00000246
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266226] RAX: ffff881049214d88 RBX: 0000000000000010 RCX: ffff881049214e80
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266228] RDX: 000000000000996a RSI: 0000000000000200 RDI: ffff881049214d68
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266230] RBP: ffff880079aa7d28 R08: ffff881049214f50 R09: 0000000000000101
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266231] R10: 0000000000000004 R11: dead000000200200 R12: ffffffffffffff10
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266233] R13: ffff88104e0e4000 R14: ffff881049214e90 R15: 00000000f032005e
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266235] FS: 0000000000000000(0000) GS:ffff88085fac0000(0000) knlGS:0000000000000000
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266237] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266239] CR2: 0000000000b30fe4 CR3: 0000000079b9f000 CR4: 00000000001407e0
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266241] Stack:
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266242] ffff881049214c80 ffff88104a8fc000 0000000000000001 ffff881049214e90
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266245] ffff880079aa7d58 ffffffffc04ae66d ffff881049214e80 ffff881049214c80
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266248] 0000000000000001 0000000000000001 ffff880079aa7d98 ffffffffc04ae70b
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266252] Call Trace:
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266261] [<ffffffffc04ae66d>] __iscsit_free_cmd+0x8d/0xa0 [iscsi_target_mod]
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266268] [<ffffffffc04ae70b>] iscsit_free_cmd+0x8b/0x160 [iscsi_target_mod]
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266276] [<ffffffffc04b78dc>] iscsit_close_connection+0x47c/0x770 [iscsi_target_mod]
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266284] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266292] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266296] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266304] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266308] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266312] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266315] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266318] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266319] Code: 20 49 39 c5 75 0c eb 23 0f 1f 40 00 48 89 df 48 89 d3 4c 89 e6 e8 5a ff ff ff 48 8b 43 20 48 8d 50 e0 48 8d 43 20 49 39 c5 75 e1 <4c> 89 f7 e8 71 5a 35 c1 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00
Jun 3 05:03:45 roc-4r-scd212 kernel: [70103.266319] Code: 20 49 39 c5 75 0c eb 23 0f 1f 40 00 48 89 df 48 89 d3 4c 89 e6 e8 5a ff ff ff 48 8b 43 20 48 8d 50 e0 48 8d 43 20 49 39 c5 75 e1 <4c> 89 f7 e8 71 5a 35 c1 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 84 00
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145122] INFO: rcu_sched self-detected stall on CPU { 3} (t=15000 jiffies g=1021511 c=1021510 q=0)
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145182] Task dump for CPU 3:
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145184] iscsi_trx R running task 0 7052 2 0x0000000c
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145188] ffffffff81e51940 ffff88085fac3d18 ffffffff810ab341 0000000000000003
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145191] ffffffff81e51940 ffff88085fac3d38 ffffffff810ae2bf 0000000000000087
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145194] ffffffff81e51a40 ffff88085fac3d68 ffffffff810e046b ffffffff81e51940
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145197] Call Trace:
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145199] <IRQ> [<ffffffff810ab341>] sched_show_task+0xb1/0x120
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145207] [<ffffffff810ae2bf>] dump_cpu_task+0x3f/0x50
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145211] [<ffffffff810e046b>] rcu_dump_cpu_stacks+0x8b/0xc0
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145214] [<ffffffff810e22d2>] print_cpu_stall+0xe2/0x140
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145217] [<ffffffff810e2b15>] check_cpu_stall.isra.56+0xa5/0xb0
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145219] [<ffffffff810e2b58>] __rcu_pending+0x38/0x180
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145222] [<ffffffff810e498d>] rcu_check_callbacks+0x9d/0x190
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145226] [<ffffffff810ea2a8>] update_process_times+0x38/0x70
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145231] [<ffffffff810fa6c3>] tick_sched_handle.isra.12+0x33/0x70
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145233] [<ffffffff810fa7fb>] tick_sched_timer+0x4b/0x80
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145236] [<ffffffff810eada6>] __run_hrtimer+0x76/0x230
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145239] [<ffffffff810fa7b0>] ? tick_nohz_handler+0xb0/0xb0
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145242] [<ffffffff810eb573>] hrtimer_interrupt+0xf3/0x220
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145246] [<ffffffff81052179>] local_apic_timer_interrupt+0x39/0x60
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145251] [<ffffffff81806995>] smp_apic_timer_interrupt+0x45/0x60
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145254] [<ffffffff81804a3e>] apic_timer_interrupt+0x6e/0x80
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145255] <EOI> [<ffffffff8180331e>] ? _raw_spin_unlock_bh+0xe/0x50
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145268] [<ffffffffc04ad89f>] iscsit_free_r2ts_from_list+0x6f/0x80 [iscsi_target_mod]
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145276] [<ffffffffc04ae66d>] __iscsit_free_cmd+0x8d/0xa0 [iscsi_target_mod]
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145282] [<ffffffffc04ae691>] ? iscsit_free_cmd+0x11/0x160 [iscsi_target_mod]
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145289] [<ffffffffc04ae70b>] iscsit_free_cmd+0x8b/0x160 [iscsi_target_mod]
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145297] [<ffffffffc04b78dc>] iscsit_close_connection+0x47c/0x770 [iscsi_target_mod]
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145303] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145311] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145314] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145321] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145325] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145328] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145331] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:03:53 roc-4r-scd212 kernel: [70111.145334] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278382] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [iscsi_trx:7052]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278428] Modules linked in: rbd libceph libcrc32c iscsi_target_mod target_core_file target_core_pscsi target_core_iblock target_core_mod configfs xt_multiport iptable_filter ip_tables x_tables enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE) ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core joydev mei_me lpc_ich mei ioatdma 8021q garp mrp ipmi_si stp 8250_fintek ipmi_msghandler llc bonding shpchp wmi lp mac_hid parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic igb usbhid ahci libahci hid mpt2sas i2c_algo_bit mlx4_core dca ptp raid_class pps_core scsi_transport_sas
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278486] CPU: 3 PID: 7052 Comm: iscsi_trx Tainted: G C OEL 4.1.0-040100rc2-generic #201505032335
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278488] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278490] task: ffff880859198000 ti: ffff880079aa4000 task.ti: ffff880079aa4000
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278492] RIP: 0010:[<ffffffffc045511b>] [<ffffffffc045511b>] transport_release_cmd+0x1b/0x60 [target_core_mod]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278513] RSP: 0018:ffff880079aa7cd8 EFLAGS: 00000282
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278515] RAX: 000000000008090d RBX: 00000000e681e3b0 RCX: ffff881049214e80
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278517] RDX: 0000000000007626 RSI: 0000000000000282 RDI: ffff881049214e90
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278518] RBP: ffff880079aa7ce8 R08: ffff881049214f50 R09: 0000000000000101
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278520] R10: 0000000000000004 R11: dead000000200200 R12: ffff881049214f50
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278522] R13: 0000000000000101 R14: 0000000000000004 R15: dead000000200200
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278525] FS: 0000000000000000(0000) GS:ffff88085fac0000(0000) knlGS:0000000000000000
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278529] CR2: 0000000000b30fe4 CR3: 0000000079b9f000 CR4: 00000000001407e0
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278531] Stack:
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278532] 0000000000000000 ffff881049214e90 ffff880079aa7d28 ffffffffc0455d06
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278536] ffff880079aa7d28 ffff881049214e90 0000000000000001 0000000000000001
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278539] ffff881049214e90 ffff881049214e90 ffff880079aa7d58 ffffffffc0455f84
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278542] Call Trace:
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278554] [<ffffffffc0455d06>] transport_put_cmd+0x86/0x290 [target_core_mod]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278563] [<ffffffffc0455f84>] transport_generic_free_cmd+0x74/0xe0 [target_core_mod]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278574] [<ffffffffc04ae691>] ? iscsit_free_cmd+0x11/0x160 [iscsi_target_mod]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278581] [<ffffffffc04ae716>] iscsit_free_cmd+0x96/0x160 [iscsi_target_mod]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278588] [<ffffffffc04a1780>] ? iscsit_increment_maxcmdsn+0x10/0x90 [iscsi_target_mod]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278596] [<ffffffffc04b78dc>] iscsit_close_connection+0x47c/0x770 [iscsi_target_mod]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278603] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278611] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278615] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278622] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278627] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278630] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278633] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278637] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:04:17 roc-4r-scd212 kernel: [70135.278638] Code: e8 0b e0 ff ff 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 83 bf c8 00 00 00 00 74 43 <f6> 47 20 10 75 2f 48 8b bb e8 00 00 00 48 8d 83 f0 00 00 00 48
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289129] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [iscsi_trx:7052]
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289177] Modules linked in: rbd libceph libcrc32c iscsi_target_mod target_core_file target_core_pscsi target_core_iblock target_core_mod configfs xt_multiport iptable_filter ip_tables x_tables enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE) ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core joydev mei_me lpc_ich mei ioatdma 8021q garp mrp ipmi_si stp 8250_fintek ipmi_msghandler llc bonding shpchp wmi lp mac_hid parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic igb usbhid ahci libahci hid mpt2sas i2c_algo_bit mlx4_core dca ptp raid_class pps_core scsi_transport_sas
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289235] CPU: 3 PID: 7052 Comm: iscsi_trx Tainted: G C OEL 4.1.0-040100rc2-generic #201505032335
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289237] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289239] task: ffff880859198000 ti: ffff880079aa4000 task.ti: ffff880079aa4000
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289241] RIP: 0010:[<ffffffff8180379d>] [<ffffffff8180379d>] _raw_spin_lock_bh+0x1d/0x70
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289250] RSP: 0018:ffff880079aa7cf8 EFLAGS: 00000282
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289251] RAX: 00000000cea8cea8 RBX: 0000000000000004 RCX: 00000000de0118ae
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289253] RDX: 000000000000ceaa RSI: 0000000000000200 RDI: ffff881049214d70
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289255] RBP: ffff880079aa7cf8 R08: ffff881049214f50 R09: 0000000000000101
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289257] R10: 0000000000000004 R11: dead000000200200 R12: ffff881049214e90
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289258] R13: ffff880079aa7ce8 R14: 00000000de025583 R15: 00000000de025584
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289260] FS: 0000000000000000(0000) GS:ffff88085fac0000(0000) knlGS:0000000000000000
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289262] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289264] CR2: 0000000000b30fe4 CR3: 0000000079b9f000 CR4: 00000000001407e0
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289266] Stack:
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289267] ffff880079aa7d28 ffffffffc04ad85a ffff881049214c80 ffff88104a8fc000
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289271] 0000000000000001 ffff881049214e90 ffff880079aa7d58 ffffffffc04ae66d
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289274] 00000000fab6fab6 ffff881049214c80 0000000000000001 0000000000000001
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289277] Call Trace:
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289291] [<ffffffffc04ad85a>] iscsit_free_r2ts_from_list+0x2a/0x80 [iscsi_target_mod]
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289299] [<ffffffffc04ae66d>] __iscsit_free_cmd+0x8d/0xa0 [iscsi_target_mod]
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289306] [<ffffffffc04ae73e>] iscsit_free_cmd+0xbe/0x160 [iscsi_target_mod]
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289314] [<ffffffffc04b78dc>] iscsit_close_connection+0x47c/0x770 [iscsi_target_mod]
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289322] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289330] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289333] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289340] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289346] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289349] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289352] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289355] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:04:45 roc-4r-scd212 kernel: [70163.289356] Code: 00 00 00 00 eb ce 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 65 81 05 0f 81 80 7e 00 02 00 00 48 89 e5 b8 00 00 02 00 f0 0f c1 07 <89> c2 c1 ea 10 66 39 c2 75 02 5d c3 0f b7 ca b8 00 80 00 00 44
Jun 3 05:05:01 roc-4r-scd212 CRON[10883]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.299876] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [iscsi_trx:7052]
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.299924] Modules linked in: rbd libceph libcrc32c iscsi_target_mod target_core_file target_core_pscsi target_core_iblock target_core_mod configfs xt_multiport iptable_filter ip_tables x_tables enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE) ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core joydev mei_me lpc_ich mei ioatdma 8021q garp mrp ipmi_si stp 8250_fintek ipmi_msghandler llc bonding shpchp wmi lp mac_hid parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic igb usbhid ahci libahci hid mpt2sas i2c_algo_bit mlx4_core dca ptp raid_class pps_core scsi_transport_sas
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.299982] CPU: 3 PID: 7052 Comm: iscsi_trx Tainted: G C OEL 4.1.0-040100rc2-generic #201505032335
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.299984] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.299986] task: ffff880859198000 ti: ffff880079aa4000 task.ti: ffff880079aa4000
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.299988] RIP: 0010:[<ffffffffc0453d6c>] [<ffffffffc0453d6c>] transport_wait_for_tasks+0x1c/0x150 [target_core_mod]
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300006] RSP: 0018:ffff880079aa7cf8 EFLAGS: 00000282
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300008] RAX: 000000000008090d RBX: 0000000000000282 RCX: ffff881049214e80
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300010] RDX: 000000000000ac32 RSI: 0000000000000001 RDI: ffff881049214e90
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300012] RBP: ffff880079aa7d28 R08: ffff881049214f50 R09: 0000000000000101
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300013] R10: 0000000000000004 R11: dead000000200200 R12: ffff881049214e80
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300015] R13: 000000000008090d R14: ffff881049214f50 R15: 0000000000000101
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300017] FS: 0000000000000000(0000) GS:ffff88085fac0000(0000) knlGS:0000000000000000
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300021] CR2: 0000000000b30fe4 CR3: 0000000079b9f000 CR4: 00000000001407e0
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300022] Stack:
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300024] ffff880079aa7d28 ffffffffc04ad418 ffff881049214c80 ffff881049214e90
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300027] 0000000000000001 ffff881049214e90 ffff880079aa7d58 ffffffffc0455fd5
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300030] ffffffffc04ae711 ffff881049214c80 0000000000000001 0000000000000001
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300033] Call Trace:
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300047] [<ffffffffc04ad418>] ? iscsit_remove_cmd_from_response_queue+0xe8/0x120 [iscsi_target_mod]
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300061] [<ffffffffc0455fd5>] transport_generic_free_cmd+0xc5/0xe0 [target_core_mod]
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300072] [<ffffffffc04ae711>] ? iscsit_free_cmd+0x91/0x160 [iscsi_target_mod]
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300080] [<ffffffffc04ae716>] iscsit_free_cmd+0x96/0x160 [iscsi_target_mod]
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300090] [<ffffffffc04b78dc>] iscsit_close_connection+0x47c/0x770 [iscsi_target_mod]
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300100] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300109] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300114] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300123] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300128] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300132] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300135] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300138] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:05:13 roc-4r-scd212 kernel: [70191.300140] Code: c3 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 30 4c 89 65 f0 4c 8d a7 20 01 00 00 48 89 5d e8 <48> 89 fb 4c 89 6d f8 4c 89 e7 e8 05 f9 3a c1 49 89 c5 8b 43 20
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310622] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [iscsi_trx:7052]
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310669] Modules linked in: rbd libceph libcrc32c iscsi_target_mod target_core_file target_core_pscsi target_core_iblock target_core_mod configfs xt_multiport iptable_filter ip_tables x_tables enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE) ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core joydev mei_me lpc_ich mei ioatdma 8021q garp mrp ipmi_si stp 8250_fintek ipmi_msghandler llc bonding shpchp wmi lp mac_hid parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic igb usbhid ahci libahci hid mpt2sas i2c_algo_bit mlx4_core dca ptp raid_class pps_core scsi_transport_sas
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310724] CPU: 3 PID: 7052 Comm: iscsi_trx Tainted: G C OEL 4.1.0-040100rc2-generic #201505032335
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310725] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310727] task: ffff880859198000 ti: ffff880079aa4000 task.ti: ffff880079aa4000
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310729] RIP: 0010:[<ffffffffc0455fd5>] [<ffffffffc0455fd5>] transport_generic_free_cmd+0xc5/0xe0 [target_core_mod]
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310747] RSP: 0018:ffff880079aa7d38 EFLAGS: 00000246
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310749] RAX: 0000000000000000 RBX: 00000000cd06fae4 RCX: ffff881049214e80
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310750] RDX: 000000000000a214 RSI: 0000000000000282 RDI: 0000000000000282
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310752] RBP: ffff880079aa7d58 R08: ffff881049214f50 R09: 0000000000000101
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310754] R10: 0000000000000004 R11: dead000000200200 R12: ffff881049214f50
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310755] R13: 0000000000000101 R14: 0000000000000004 R15: dead000000200200
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310757] FS: 0000000000000000(0000) GS:ffff88085fac0000(0000) knlGS:0000000000000000
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310761] CR2: 0000000000b30fe4 CR3: 0000000079b9f000 CR4: 00000000001407e0
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310763] Stack:
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310764] ffff881049214e80 ffff881049214c80 0000000000000001 0000000000000001
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310767] ffff880079aa7d98 ffffffffc04ae716 ffff880079aa7d88 ffff88104a8fc000
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310771] ffff8810499ea800 ffff881049214c80 0000000000000008 ffff881049214c80
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310774] Call Trace:
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310787] [<ffffffffc04ae716>] iscsit_free_cmd+0x96/0x160 [iscsi_target_mod]
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310797] [<ffffffffc04b78dc>] iscsit_close_connection+0x47c/0x770 [iscsi_target_mod]
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310806] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310815] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310820] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310829] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310834] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310838] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310842] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310845] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:05:41 roc-4r-scd212 kernel: [70219.310847] Code: 49 89 c5 e8 2e d7 ff ff 4c 89 ee 4c 89 e7 e8 73 d7 3a c1 48 83 7b 78 00 75 b0 eb b6 66 2e 0f 1f 84 00 00 00 00 00 e8 7b dd ff ff <80> bb a8 01 00 00 00 74 8f eb b8 e8 6b dd ff ff e9 56 ff ff ff
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.717899] INFO: task iscsi_np:23171 blocked for more than 120 seconds.
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.717933] Tainted: G C OEL 4.1.0-040100rc2-generic #201505032335
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.717977] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718023] iscsi_np D ffff88105946bb88 0 23171 2 0x00000004
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718028] ffff88105946bb88 ffff88105946bb48 ffff880858a821c0 ffff88085bc164a0
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718032] ffff881059516eb0 ffffffff00000000 ffff88105946c000 7fffffffffffffff
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718035] ffff88104a8fc100 ffff881059516eb0 0000000000000001 ffff88105946bba8
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718038] Call Trace:
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718047] [<ffffffff817ffb47>] schedule+0x37/0x90
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718052] [<ffffffff818027f5>] schedule_timeout+0x1b5/0x210
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718057] [<ffffffff8108a5d0>] ? __send_signal+0x1d0/0x410
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718062] [<ffffffff81800ef8>] wait_for_completion+0xa8/0x170
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718065] [<ffffffff8108b6a6>] ? do_send_sig_info+0x66/0x90
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718070] [<ffffffff810ac050>] ? try_to_wake_up+0x2a0/0x2a0
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718081] [<ffffffffc04a22c3>] iscsit_cause_connection_reinstatement.part.0+0x73/0xd0 [iscsi_target_mod]
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718088] [<ffffffffc04a2377>] iscsit_cause_connection_reinstatement+0x57/0x80 [iscsi_target_mod]
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718098] [<ffffffffc04b6d3b>] iscsit_stop_session+0x11b/0x1d0 [iscsi_target_mod]
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718106] [<ffffffffc04a64cd>] iscsi_check_for_session_reinstatement+0x1cd/0x260 [iscsi_target_mod]
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718113] [<ffffffffc04a8fcb>] iscsi_target_handle_csg_one+0xdb/0x180 [iscsi_target_mod]
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718120] [<ffffffffc04a915c>] iscsi_target_do_login+0xec/0x160 [iscsi_target_mod]
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718127] [<ffffffffc04a9c67>] iscsi_target_start_negotiation+0x27/0xb0 [iscsi_target_mod]
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718133] [<ffffffffc04a7d88>] __iscsi_target_login_thread+0x298/0x670 [iscsi_target_mod]
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718140] [<ffffffffc04a8160>] ? __iscsi_target_login_thread+0x670/0x670 [iscsi_target_mod]
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718146] [<ffffffffc04a8188>] iscsi_target_login_thread+0x28/0x40 [iscsi_target_mod]
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718151] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718154] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718158] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:05:49 roc-4r-scd212 kernel: [70227.718161] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321369] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [iscsi_trx:7052]
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321416] Modules linked in: rbd libceph libcrc32c iscsi_target_mod target_core_file target_core_pscsi target_core_iblock target_core_mod configfs xt_multiport iptable_filter ip_tables x_tables enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE) ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core joydev mei_me lpc_ich mei ioatdma 8021q garp mrp ipmi_si stp 8250_fintek ipmi_msghandler llc bonding shpchp wmi lp mac_hid parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic igb usbhid ahci libahci hid mpt2sas i2c_algo_bit mlx4_core dca ptp raid_class pps_core scsi_transport_sas
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321470] CPU: 3 PID: 7052 Comm: iscsi_trx Tainted: G C OEL 4.1.0-040100rc2-generic #201505032335
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321472] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321474] task: ffff880859198000 ti: ffff880079aa4000 task.ti: ffff880079aa4000
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321476] RIP: 0010:[<ffffffff81803322>] [<ffffffff81803322>] _raw_spin_unlock_bh+0x12/0x50
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321484] RSP: 0018:ffff880079aa7d98 EFLAGS: 00000282
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321486] RAX: ffff881049214e80 RBX: ffff88104a8fc000 RCX: ffff881049214e80
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321488] RDX: ffff881049214e80 RSI: ffff881049214e90 RDI: ffff88104a8fc3e0
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321490] RBP: ffff880079aa7d98 R08: ffff881049214f50 R09: 0000000000000101
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321492] R10: 0000000000000004 R11: dead000000200200 R12: ffffffffc04ad418
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321494] R13: ffff880079aa7d28 R14: ffffffff81803330 R15: ffff880079aa7cf8
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321496] FS: 0000000000000000(0000) GS:ffff88085fac0000(0000) knlGS:0000000000000000
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321500] CR2: 0000000000b30fe4 CR3: 0000000079b9f000 CR4: 00000000001407e0
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321501] Stack:
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321503] ffff880079aa7df8 ffffffffc04b78c3 0000000000000000 ffff8810499ea800
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321506] ffff88104a8fc3e0 ffff88104a8fc440 ffff880859198000 ffff88104a8fc000
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321509] ffff88104a8fc3f4 ffff880079aa7e5c ffff880859198000 0000000000000000
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321512] Call Trace:
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321526] [<ffffffffc04b78c3>] iscsit_close_connection+0x463/0x770 [iscsi_target_mod]
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321535] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321544] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321548] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321556] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321562] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321566] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321569] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321572] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321574] Code: 08 5d c3 66 0f 1f 44 00 00 e8 eb 56 8c ff 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 0f 1f 44 00 00 66 83 07 02 <48> 8b 7d 08 be 00 02 00 00 e8 c0 df 87 ff 5d c3 66 0f 1f 44 00
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321369] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [iscsi_trx:7052]
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321416] Modules linked in: rbd libceph libcrc32c iscsi_target_mod target_core_file target_core_pscsi target_core_iblock target_core_mod configfs xt_multiport iptable_filter ip_tables x_tables enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE) ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core joydev mei_me lpc_ich mei ioatdma 8021q garp mrp ipmi_si stp 8250_fintek ipmi_msghandler llc bonding shpchp wmi lp mac_hid parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic igb usbhid ahci libahci hid mpt2sas i2c_algo_bit mlx4_core dca ptp raid_class pps_core scsi_transport_sas
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321470] CPU: 3 PID: 7052 Comm: iscsi_trx Tainted: G C OEL 4.1.0-040100rc2-generic #201505032335
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321472] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321474] task: ffff880859198000 ti: ffff880079aa4000 task.ti: ffff880079aa4000
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321476] RIP: 0010:[<ffffffff81803322>] [<ffffffff81803322>] _raw_spin_unlock_bh+0x12/0x50
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321484] RSP: 0018:ffff880079aa7d98 EFLAGS: 00000282
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321486] RAX: ffff881049214e80 RBX: ffff88104a8fc000 RCX: ffff881049214e80
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321488] RDX: ffff881049214e80 RSI: ffff881049214e90 RDI: ffff88104a8fc3e0
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321490] RBP: ffff880079aa7d98 R08: ffff881049214f50 R09: 0000000000000101
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321492] R10: 0000000000000004 R11: dead000000200200 R12: ffffffffc04ad418
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321494] R13: ffff880079aa7d28 R14: ffffffff81803330 R15: ffff880079aa7cf8
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321496] FS: 0000000000000000(0000) GS:ffff88085fac0000(0000) knlGS:0000000000000000
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321498] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321500] CR2: 0000000000b30fe4 CR3: 0000000079b9f000 CR4: 00000000001407e0
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321501] Stack:
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321503] ffff880079aa7df8 ffffffffc04b78c3 0000000000000000 ffff8810499ea800
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321506] ffff88104a8fc3e0 ffff88104a8fc440 ffff880859198000 ffff88104a8fc000
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321509] ffff88104a8fc3f4 ffff880079aa7e5c ffff880859198000 0000000000000000
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321512] Call Trace:
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321526] [<ffffffffc04b78c3>] iscsit_close_connection+0x463/0x770 [iscsi_target_mod]
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321535] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321544] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321548] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321556] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321562] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321566] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321569] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321572] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:06:09 roc-4r-scd212 kernel: [70247.321574] Code: 08 5d c3 66 0f 1f 44 00 00 e8 eb 56 8c ff 5d c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 0f 1f 44 00 00 66 83 07 02 <48> 8b 7d 08 be 00 02 00 00 e8 c0 df 87 ff 5d c3 66 0f 1f 44 00
Jun 3 05:06:32 roc-4r-scd212 kernel: [70270.004358] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332115] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [iscsi_trx:7052]
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332162] Modules linked in: rbd libceph libcrc32c iscsi_target_mod target_core_file target_core_pscsi target_core_iblock target_core_mod configfs xt_multiport iptable_filter ip_tables x_tables enhanceio_rand(OE) enhanceio_lru(OE) enhanceio_fifo(OE) enhanceio(OE) ipmi_devintf ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core joydev mei_me lpc_ich mei ioatdma 8021q garp mrp ipmi_si stp 8250_fintek ipmi_msghandler llc bonding shpchp wmi lp mac_hid parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic igb usbhid ahci libahci hid mpt2sas i2c_algo_bit mlx4_core dca ptp raid_class pps_core scsi_transport_sas
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332215] CPU: 3 PID: 7052 Comm: iscsi_trx Tainted: G C OEL 4.1.0-040100rc2-generic #201505032335
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332217] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332219] task: ffff880859198000 ti: ffff880079aa4000 task.ti: ffff880079aa4000
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332221] RIP: 0010:[<ffffffffc04ae5f8>] [<ffffffffc04ae5f8>] __iscsit_free_cmd+0x18/0xa0 [iscsi_target_mod]
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332234] RSP: 0018:ffff880079aa7d38 EFLAGS: 00000202
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332236] RAX: 0000000000000001 RBX: ffff881049214e80 RCX: ffff881049214e80
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332238] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff881049214c80
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332239] RBP: ffff880079aa7d58 R08: ffff881049214f50 R09: 0000000000000101
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332241] R10: 0000000000000004 R11: dead000000200200 R12: ffff881049214f50
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332243] R13: 0000000000000101 R14: 0000000000000004 R15: dead000000200200
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332245] FS: 0000000000000000(0000) GS:ffff88085fac0000(0000) knlGS:0000000000000000
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332247] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332248] CR2: 0000000000b30fe4 CR3: 0000000079b9f000 CR4: 00000000001407e0
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332250] Stack:
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332251] ffffffffc04b78c3 ffff881049214c80 ffff881049214c80 0000000000000001
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332254] ffff880079aa7d98 ffffffffc04ae70b ffff880079aa7d88 ffff88104a8fc000
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332257] ffff8810499ea800 ffff881049214c80 0000000000000008 ffff881049214c80
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332260] Call Trace:
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332271] [<ffffffffc04b78c3>] ? iscsit_close_connection+0x463/0x770 [iscsi_target_mod]
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332280] [<ffffffffc04ae70b>] iscsit_free_cmd+0x8b/0x160 [iscsi_target_mod]
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332288] [<ffffffffc04b78dc>] iscsit_close_connection+0x47c/0x770 [iscsi_target_mod]
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332297] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332306] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332311] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332319] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332325] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332328] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332332] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332335] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:06:37 roc-4r-scd212 kernel: [70275.332337] Code: e8 3e 4d 35 c1 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 40 84 f6 48 89 5d e8 4c 89 6d f8 <48> 89 fb 4c 89 65 f0 41 89 d5 4c 8b a7 e8 01 00 00 74 10 8b 87
un 3 05:06:53 roc-4r-scd212 kernel: [70291.226213] INFO: rcu_sched self-detected stall on CPU { 3} (t=60003 jiffies g=1021511 c=1021510 q=0)
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226274] Task dump for CPU 3:
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226276] iscsi_trx R running task 0 7052 2 0x0000000c
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226280] ffffffff81e51940 ffff88085fac3d18 ffffffff810ab341 0000000000000003
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226283] ffffffff81e51940 ffff88085fac3d38 ffffffff810ae2bf 0000000000000087
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226286] ffffffff81e51a40 ffff88085fac3d68 ffffffff810e046b ffffffff81e51940
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226289] Call Trace:
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226291] <IRQ> [<ffffffff810ab341>] sched_show_task+0xb1/0x120
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226301] [<ffffffff810ae2bf>] dump_cpu_task+0x3f/0x50
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226305] [<ffffffff810e046b>] rcu_dump_cpu_stacks+0x8b/0xc0
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226307] [<ffffffff810e22d2>] print_cpu_stall+0xe2/0x140
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226310] [<ffffffff810e2b15>] check_cpu_stall.isra.56+0xa5/0xb0
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226313] [<ffffffff810e2b58>] __rcu_pending+0x38/0x180
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226316] [<ffffffff810e498d>] rcu_check_callbacks+0x9d/0x190
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226319] [<ffffffff810ea2a8>] update_process_times+0x38/0x70
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226324] [<ffffffff810fa6c3>] tick_sched_handle.isra.12+0x33/0x70
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226327] [<ffffffff810fa7fb>] tick_sched_timer+0x4b/0x80
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226329] [<ffffffff810eada6>] __run_hrtimer+0x76/0x230
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226332] [<ffffffff810fa7b0>] ? tick_nohz_handler+0xb0/0xb0
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226335] [<ffffffff810eb573>] hrtimer_interrupt+0xf3/0x220
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226340] [<ffffffff81052179>] local_apic_timer_interrupt+0x39/0x60
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226345] [<ffffffff81806995>] smp_apic_timer_interrupt+0x45/0x60
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226347] [<ffffffff81804a3e>] apic_timer_interrupt+0x6e/0x80
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226349] <EOI> [<ffffffff81803310>] ? _raw_write_lock+0x30/0x30
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226364] [<ffffffffc04ad2f8>] ? iscsit_remove_cmd_from_immediate_queue+0xe8/0x120 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226372] [<ffffffffc04ae630>] __iscsit_free_cmd+0x50/0xa0 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226379] [<ffffffffc04ae6fb>] ? iscsit_free_cmd+0x7b/0x160 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226385] [<ffffffffc04ae70b>] iscsit_free_cmd+0x8b/0x160 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226393] [<ffffffffc04b78dc>] iscsit_close_connection+0x47c/0x770 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226400] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226408] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226411] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226418] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226422] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226425] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226428] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226431] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
un 3 05:06:53 roc-4r-scd212 kernel: [70291.226213] INFO: rcu_sched self-detected stall on CPU { 3} (t=60003 jiffies g=1021511 c=1021510 q=0)
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226274] Task dump for CPU 3:
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226276] iscsi_trx R running task 0 7052 2 0x0000000c
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226280] ffffffff81e51940 ffff88085fac3d18 ffffffff810ab341 0000000000000003
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226283] ffffffff81e51940 ffff88085fac3d38 ffffffff810ae2bf 0000000000000087
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226286] ffffffff81e51a40 ffff88085fac3d68 ffffffff810e046b ffffffff81e51940
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226289] Call Trace:
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226291] <IRQ> [<ffffffff810ab341>] sched_show_task+0xb1/0x120
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226301] [<ffffffff810ae2bf>] dump_cpu_task+0x3f/0x50
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226305] [<ffffffff810e046b>] rcu_dump_cpu_stacks+0x8b/0xc0
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226307] [<ffffffff810e22d2>] print_cpu_stall+0xe2/0x140
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226310] [<ffffffff810e2b15>] check_cpu_stall.isra.56+0xa5/0xb0
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226313] [<ffffffff810e2b58>] __rcu_pending+0x38/0x180
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226316] [<ffffffff810e498d>] rcu_check_callbacks+0x9d/0x190
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226319] [<ffffffff810ea2a8>] update_process_times+0x38/0x70
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226324] [<ffffffff810fa6c3>] tick_sched_handle.isra.12+0x33/0x70
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226327] [<ffffffff810fa7fb>] tick_sched_timer+0x4b/0x80
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226329] [<ffffffff810eada6>] __run_hrtimer+0x76/0x230
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226332] [<ffffffff810fa7b0>] ? tick_nohz_handler+0xb0/0xb0
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226335] [<ffffffff810eb573>] hrtimer_interrupt+0xf3/0x220
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226340] [<ffffffff81052179>] local_apic_timer_interrupt+0x39/0x60
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226345] [<ffffffff81806995>] smp_apic_timer_interrupt+0x45/0x60
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226347] [<ffffffff81804a3e>] apic_timer_interrupt+0x6e/0x80
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226349] <EOI> [<ffffffff81803310>] ? _raw_write_lock+0x30/0x30
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226364] [<ffffffffc04ad2f8>] ? iscsit_remove_cmd_from_immediate_queue+0xe8/0x120 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226372] [<ffffffffc04ae630>] __iscsit_free_cmd+0x50/0xa0 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226379] [<ffffffffc04ae6fb>] ? iscsit_free_cmd+0x7b/0x160 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226385] [<ffffffffc04ae70b>] iscsit_free_cmd+0x8b/0x160 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226393] [<ffffffffc04b78dc>] iscsit_close_connection+0x47c/0x770 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226400] [<ffffffffc04a2c83>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226408] [<ffffffffc04b6947>] iscsi_target_rx_thread+0x207/0x2b0 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226411] [<ffffffff817ff4fe>] ? __schedule+0x38e/0x8db
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226418] [<ffffffffc04b6740>] ? iscsi_target_tx_thread+0x1d0/0x1d0 [iscsi_target_mod]
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226422] [<ffffffff8109cdc9>] kthread+0xc9/0xe0
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226425] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226428] [<ffffffff81803fe2>] ret_from_fork+0x42/0x70
Jun 3 05:06:53 roc-4r-scd212 kernel: [70291.226431] [<ffffffff8109cd00>] ? flush_kthread_worker+0x90/0x90
Jun 3 05:07:25 roc-4r-scd212 kernel: [70323.782380] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
Jun 3 05:07:25 roc-4r-scd212 kernel: [70323.817163] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION.
This then goes on and on until Alex restarted.
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
