Hi, I have a CentOS 7.1 box with vanilla 3.18.16 kernel exporting a single lun to a ~20 node ESXi 6.0 cluster. I have run into the soft lockup issue multiple times in the past few weeks. There was no IO except for VMware heartbeat, when the soft lockup happened, so I know this is not a load issue. This looks exactly like a duplicate of issue described in http://www.spinics.net/lists/target-devel/msg09204.html and I have the patches suggested there http://www.spinics.net/lists/target-devel/msg07060.html applied The problem manifests on 3.18.21 too. In all cases, the machine eventually dies due to memory leak caused by rcu stall. After the first soft lockup, all the nodes in the cluster lost connection to the lun. Oddly enough, this happens only when the machine is barely doing any IO. Let me know, if you need any further details Nov 15 10:16:22 sanhost kernel: Detected MISCOMPARE for addr: ffff88083b432000 buf: ffff880843038600 Nov 15 10:16:22 sanhost kernel: Target/iblock: Send MISCOMPARE check condition and sense Nov 15 12:10:05 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3289777 Nov 15 12:10:07 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 3289777 Nov 15 12:10:26 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435674 Nov 15 12:10:26 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435674 Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496040 Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496040 Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483359 Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483359 Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798798 Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798798 Nov 15 12:10:35 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552310 Nov 15 12:10:35 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552310 Nov 15 12:10:35 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552310 Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 380154 Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 380154 Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380154 Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380154 Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 380155 Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 380155 Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435681 Nov 15 12:10:39 sanhost kernel: ABORT_TASK: ref_tag: 435681 already complete, skipping Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435681 Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289781 Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289780 Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289781 Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483366 Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1798804 Nov 15 12:10:42 sanhost kernel: ABORT_TASK: ref_tag: 1798804 already complete, skipping Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798804 Nov 15 12:10:55 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496050 Nov 15 12:10:55 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289788 Nov 15 12:10:56 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798810 Nov 15 12:11:02 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552322 Nov 15 12:11:06 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380167 Nov 15 12:11:07 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435690 Nov 15 12:11:09 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289794 Nov 15 12:11:09 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483375 Nov 15 12:11:23 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798819 Nov 15 12:11:29 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552330 Nov 15 12:11:33 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380176 Nov 15 12:11:34 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435698 Nov 15 12:11:34 sanhost kernel: ABORT_TASK: ref_tag: 435698 already complete, skipping Nov 15 12:11:34 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435698 Nov 15 12:11:36 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3289803 Nov 15 12:11:36 sanhost kernel: ABORT_TASK: ref_tag: 3289803 already complete, skipping Nov 15 12:11:36 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289803 Nov 15 12:11:37 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798824 Nov 15 12:11:39 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3496055 Nov 15 12:11:39 sanhost kernel: Unable to locate ITT: 0x00355877 on CID: 0 Nov 15 12:11:39 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 3496055 Nov 15 12:11:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496055 Nov 15 12:11:39 sanhost kernel: Unable to locate RefTaskTag: 0x00355877 on CID: 0. Nov 15 12:11:43 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552336 Nov 15 12:11:47 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 380182 Nov 15 12:11:47 sanhost kernel: ABORT_TASK: ref_tag: 380182 already complete, skipping Nov 15 12:11:47 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380182 Nov 15 12:11:47 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435704 Nov 15 12:11:53 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483379 Nov 15 12:11:53 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483379 Nov 15 12:11:53 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483379 Nov 15 12:12:03 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289811 Nov 15 12:12:04 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798840 Nov 15 12:12:06 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496066 Nov 15 12:12:10 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552345 Nov 15 12:12:14 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380190 Nov 15 12:12:15 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435713 Nov 15 12:12:15 sanhost kernel: ABORT_TASK: ref_tag: 435713 already complete, skipping Nov 15 12:12:15 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435713 Nov 15 12:12:17 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289817 Nov 15 12:12:18 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798845 Nov 15 12:12:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380196 Nov 15 12:12:31 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798852 Nov 15 12:12:34 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496081 Nov 15 12:12:37 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3552353 Nov 15 12:12:37 sanhost kernel: ABORT_TASK: ref_tag: 3552353 already complete, skipping Nov 15 12:12:37 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552353 Nov 15 12:12:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435721 Nov 15 12:12:44 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289832 Nov 15 12:12:45 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1798857 Nov 15 12:12:45 sanhost kernel: ABORT_TASK: ref_tag: 1798857 already complete, skipping Nov 15 12:12:45 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798857 Nov 15 12:12:47 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3496087 Nov 15 12:12:47 sanhost kernel: ABORT_TASK: ref_tag: 3496087 already complete, skipping Nov 15 12:12:47 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496087 Nov 15 12:12:50 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1483391 Nov 15 12:12:50 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 1483391 Nov 15 12:12:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483391 Nov 15 12:12:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483391 Nov 15 12:12:55 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380204 Nov 15 12:13:04 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483399 Nov 15 12:13:05 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552362 Nov 15 12:13:09 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380210 Nov 15 12:13:11 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289840 Nov 15 12:13:12 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1798866 Nov 15 12:13:12 sanhost kernel: ABORT_TASK: ref_tag: 1798866 already complete, skipping Nov 15 12:13:12 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798866 Nov 15 12:13:14 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496096 Nov 15 12:13:18 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3552368 Nov 15 12:13:18 sanhost kernel: ABORT_TASK: ref_tag: 3552368 already complete, skipping Nov 15 12:13:18 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552368 Nov 15 12:13:23 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435735 Nov 15 12:13:23 sanhost kernel: ABORT_TASK: ref_tag: 435735 already complete, skipping Nov 15 12:13:23 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435735 Nov 15 12:13:25 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289846 Nov 15 12:13:26 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798872 Nov 15 12:13:36 sanhost kernel: Unexpected ret: -32 send data 48 Nov 15 12:13:36 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 380225 Nov 15 12:13:36 sanhost kernel: ABORT_TASK: ref_tag: 380225 already complete, skipping Nov 15 12:13:36 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380225 Nov 15 12:13:41 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496104 Nov 15 12:13:45 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552377 Nov 15 12:13:48 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1483403 Nov 15 12:13:48 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 1483403 Nov 15 12:13:48 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483403 Nov 15 12:13:48 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483403 Nov 15 12:13:50 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435743 Nov 15 12:13:50 sanhost kernel: ABORT_TASK: ref_tag: 435743 already complete, skipping Nov 15 12:13:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435743 Nov 15 12:13:52 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3289855 Nov 15 12:13:52 sanhost kernel: ABORT_TASK: ref_tag: 3289855 already complete, skipping Nov 15 12:13:52 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289855 Nov 15 12:13:53 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1798881 Nov 15 12:13:53 sanhost kernel: ABORT_TASK: ref_tag: 1798881 already complete, skipping Nov 15 12:13:53 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798881 Nov 15 12:14:01 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483410 Nov 15 12:14:03 sanhost kernel: Unexpected ret: -32 send data 48 Nov 15 12:14:03 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 380234 Nov 15 12:14:03 sanhost kernel: ABORT_TASK: ref_tag: 380234 already complete, skipping Nov 15 12:14:03 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380234 Nov 15 12:14:03 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435749 Nov 15 12:14:03 sanhost kernel: ABORT_TASK: ref_tag: 435749 already complete, skipping Nov 15 12:14:03 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435749 Nov 15 12:14:07 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798886 Nov 15 12:14:08 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496113 Nov 15 12:14:12 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552385 Nov 15 12:14:20 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3289863 Nov 15 12:14:20 sanhost kernel: ABORT_TASK: ref_tag: 3289863 already complete, skipping Nov 15 12:14:20 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289863 Nov 15 12:14:22 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3496119 Nov 15 12:14:22 sanhost kernel: ABORT_TASK: ref_tag: 3496119 already complete, skipping Nov 15 12:14:22 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496119 Nov 15 12:14:26 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552397 Nov 15 12:14:30 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380246 Nov 15 12:14:31 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435765 Nov 15 12:14:31 sanhost kernel: ABORT_TASK: ref_tag: 435765 already complete, skipping Nov 15 12:14:31 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435765 Nov 15 12:14:33 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289869 Nov 15 12:14:34 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798894 Nov 15 12:14:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483424 Nov 15 12:14:44 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380252 Nov 15 12:14:49 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496127 Nov 15 12:14:53 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3552407 Nov 15 12:14:53 sanhost kernel: ABORT_TASK: ref_tag: 3552407 already complete, skipping Nov 15 12:14:53 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552407 Nov 15 12:14:57 sanhost kernel: TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000001 Nov 15 12:14:58 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435773 Nov 15 12:15:00 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3289877 Nov 15 12:15:00 sanhost kernel: ABORT_TASK: ref_tag: 3289877 already complete, skipping Nov 15 12:15:00 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289877 Nov 15 12:15:01 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1798903 Nov 15 12:15:01 sanhost kernel: ABORT_TASK: ref_tag: 1798903 already complete, skipping Nov 15 12:15:01 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798903 Nov 15 12:15:09 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483432 Nov 15 12:15:11 sanhost kernel: TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000001 Nov 16 12:17:56 sanhost kernel: Unexpected ret: -32 send data 48 Nov 16 12:17:57 sanhost kernel: Unexpected ret: -32 send data 48 Nov 16 12:17:58 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3331838 Nov 16 12:17:59 sanhost kernel: Unexpected ret: -32 send data 48 Nov 16 12:18:01 sanhost kernel: Unexpected ret: -32 send data 48 Nov 16 12:18:03 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1526495 Nov 16 12:18:03 sanhost kernel: ABORT_TASK: ref_tag: 1526495 already complete, skipping Nov 16 12:18:03 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1526495 Nov 16 12:18:05 sanhost kernel: Unexpected ret: -32 send data 48 Nov 16 12:18:07 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1837659 Nov 16 12:18:08 sanhost kernel: Unexpected ret: -32 send data 48 Nov 16 12:18:08 sanhost kernel: iSCSI Login timeout on Network Portal 192.168.10.2:3260 Nov 16 12:18:21 sanhost kernel: NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [iscsi_trx:23523] Nov 16 12:18:21 sanhost kernel: Modules linked in: binfmt_misc target_core_pscsi target_core_file target_core_iblock dm_round_robin iscsi_tcp libiscsi_tcp raid0 raid1 mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase bonding dm_service_time xprtrdma sunrpc vfat fat ib_isert iscsi_target_mod intel_rapl x86_pkg_temp_thermal intel_powerclamp ib_iser coretemp libiscsi kvm_intel scsi_transport_iscsi kvm ib_srpt target_core_mod bnx2x crct10dif_pclmul crc32_pclmul iTCO_wdt crc32c_intel iTCO_vendor_support ghash_clmulni_intel cryptd pcspkr sb_edac mei_me edac_core mei ioatdma ib_srp libcrc32c lpc_ich i2c_i801 mfd_core scsi_transport_srp ipmi_devintf ib_ipoib ipmi_si ipmi_msghandler rdma_ucm ib_ucm ib_uverbs wmi shpchp ib_umad acpi_power_meter acpi_pad rdma_cm ib_cm iw_cm dm_multipath ext4 mbcache jbd2 mlx4_ib Nov 16 12:18:21 sanhost kernel: mlx4_en ib_sa ib_mad ib_core vxlan ip6_udp_tunnel udp_tunnel ib_addr sd_mod ast syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ahci libahci mlx4_core drm ixgbe libata megaraid_sas mdio ptp i2c_core pps_core dca dm_mirror dm_region_hash dm_log dm_mod Nov 16 12:18:21 sanhost kernel: CPU: 10 PID: 23523 Comm: iscsi_trx Not tainted 3.18.16 #2 Nov 16 12:18:21 sanhost kernel: Hardware name: xxxxxx, BIOS 1.0b 12/23/2014 Nov 16 12:18:21 sanhost kernel: task: ffff880829206d00 ti: ffff88082a02c000 task.ti: ffff88082a02c000 Nov 16 12:18:21 sanhost kernel: RIP: 0010:[<ffffffffa098c5b6>] [<ffffffffa098c5b6>] iscsit_stop_dataout_timer+0x6/0x80 [iscsi_target_mod] Nov 16 12:18:21 sanhost kernel: RSP: 0018:ffff88082a02fca8 EFLAGS: 00000246 Nov 16 12:18:21 sanhost kernel: RAX: 0000000000000001 RBX: ffff88080eee7408 RCX: 00000000f507d5c8 Nov 16 12:18:21 sanhost kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88080eee7300 Nov 16 12:18:21 sanhost kernel: RBP: ffff88082a02fcf8 R08: 0000000000000296 R09: 0000000000000101 Nov 16 12:18:21 sanhost kernel: R10: 0000000000002710 R11: 0000000000000000 R12: ffff88080eee7500 Nov 16 12:18:21 sanhost kernel: R13: 0000000000000101 R14: 0000000000002710 R15: 0000000000000000 Nov 16 12:18:21 sanhost kernel: FS: 0000000000000000(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000 Nov 16 12:18:21 sanhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 16 12:18:21 sanhost kernel: CR2: 00007f54678ebe90 CR3: 0000000001998000 CR4: 00000000001407e0 Nov 16 12:18:21 sanhost kernel: Stack: Nov 16 12:18:21 sanhost kernel: ffff88082a02fcf8 ffffffffa0996cf0 ffff88082a02fcf8 ffffffff00000001 Nov 16 12:18:21 sanhost kernel: ffff88080eee7300 ffff88080eee7300 0000000000000001 ffff88080eee7510 Nov 16 12:18:21 sanhost kernel: 0000000000000001 ffff8808470fb800 ffff88082a02fd28 ffffffffa0996dec Nov 16 12:18:21 sanhost kernel: Call Trace: Nov 16 12:18:21 sanhost kernel: [<ffffffffa0996cf0>] ? __iscsit_free_cmd+0x1f0/0x250 [iscsi_target_mod] Nov 16 12:18:21 sanhost kernel: [<ffffffffa0996dec>] iscsit_free_cmd+0x9c/0x150 [iscsi_target_mod] Nov 16 12:18:21 sanhost kernel: [<ffffffffa099e253>] iscsit_close_connection+0x393/0x6f0 [iscsi_target_mod] Nov 16 12:18:21 sanhost kernel: [<ffffffff810a3600>] ? wake_up_state+0x20/0x20 Nov 16 12:18:21 sanhost kernel: [<ffffffffa098c203>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod] Nov 16 12:18:21 sanhost kernel: [<ffffffffa099cd95>] iscsi_target_rx_thread+0x235/0xf50 [iscsi_target_mod] Nov 16 12:18:21 sanhost kernel: [<ffffffff810b17cc>] ? pick_next_task_fair+0x1ac/0x870 Nov 16 12:18:21 sanhost kernel: [<ffffffff810125a4>] ? __switch_to+0xe4/0x580 Nov 16 12:18:21 sanhost kernel: [<ffffffff81669b2b>] ? __schedule+0x2eb/0x810 Nov 16 12:18:21 sanhost kernel: [<ffffffffa099cb60>] ? iscsi_target_tx_thread+0x240/0x240 [iscsi_target_mod] Nov 16 12:18:21 sanhost kernel: [<ffffffff810955c8>] kthread+0xd8/0xf0 Nov 16 12:18:21 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 16 12:18:21 sanhost kernel: [<ffffffff8166e518>] ret_from_fork+0x58/0x90 Nov 16 12:18:21 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 16 12:18:21 sanhost kernel: Code: 80 00 00 00 00 29 b3 88 00 00 00 e9 ea fe ff ff b8 ff ff ff ff e9 21 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 <48> 89 e5 41 54 4c 8d a7 e4 00 00 00 53 48 89 fb 4c 89 e7 e8 b2 Nov 16 12:18:35 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38782 Nov 16 12:18:35 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38781 Nov 16 12:18:40 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37724 Nov 16 12:18:40 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37723 Nov 16 12:18:40 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37724 Nov 16 12:18:46 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38898 Nov 16 12:18:46 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38898 Nov 16 12:18:49 sanhost kernel: NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [iscsi_trx:23523] Nov 16 12:18:49 sanhost kernel: Modules linked in: binfmt_misc target_core_pscsi target_core_file target_core_iblock dm_round_robin iscsi_tcp libiscsi_tcp raid0 raid1 mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase bonding dm_service_time xprtrdma sunrpc vfat fat ib_isert iscsi_target_mod intel_rapl x86_pkg_temp_thermal intel_powerclamp ib_iser coretemp libiscsi kvm_intel scsi_transport_iscsi kvm ib_srpt target_core_mod bnx2x crct10dif_pclmul crc32_pclmul iTCO_wdt crc32c_intel iTCO_vendor_support ghash_clmulni_intel cryptd pcspkr sb_edac mei_me edac_core mei ioatdma ib_srp libcrc32c lpc_ich i2c_i801 mfd_core scsi_transport_srp ipmi_devintf ib_ipoib ipmi_si ipmi_msghandler rdma_ucm ib_ucm ib_uverbs wmi shpchp ib_umad acpi_power_meter acpi_pad rdma_cm ib_cm iw_cm dm_multipath ext4 mbcache jbd2 mlx4_ib Nov 16 12:18:49 sanhost kernel: mlx4_en ib_sa ib_mad ib_core vxlan ip6_udp_tunnel udp_tunnel ib_addr sd_mod ast syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ahci libahci mlx4_core drm ixgbe libata megaraid_sas mdio ptp i2c_core pps_core dca dm_mirror dm_region_hash dm_log dm_mod Nov 16 12:18:49 sanhost kernel: CPU: 10 PID: 23523 Comm: iscsi_trx Tainted: G L 3.18.16 #2 Nov 16 12:18:49 sanhost kernel: Hardware name: xxxxxx, BIOS 1.0b 12/23/2014 Nov 16 12:18:49 sanhost kernel: task: ffff880829206d00 ti: ffff88082a02c000 task.ti: ffff88082a02c000 Nov 16 12:18:49 sanhost kernel: RIP: 0010:[<ffffffff8166e002>] [<ffffffff8166e002>] _raw_spin_unlock_bh+0x12/0x40 Nov 16 12:18:49 sanhost kernel: RSP: 0018:ffff88082a02fca8 EFLAGS: 00000202 Nov 16 12:18:49 sanhost kernel: RAX: 0000000000000000 RBX: ffff88080eee7500 RCX: ffff88080eee7408 Nov 16 12:18:49 sanhost kernel: RDX: 0000000000003506 RSI: 00000000fffffe01 RDI: ffff880488ca7bf0 Nov 16 12:18:49 sanhost kernel: RBP: ffff88082a02fca8 R08: 0000000000000296 R09: 0000000000000101 Nov 16 12:18:49 sanhost kernel: R10: 0000000000002710 R11: 000000000000000a R12: 0000000000002710 Nov 16 12:18:49 sanhost kernel: R13: 000000000000000a R14: 0000000000002710 R15: ffff88082a02fc88 Nov 16 12:18:49 sanhost kernel: FS: 0000000000000000(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000 Nov 16 12:18:49 sanhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 16 12:18:49 sanhost kernel: CR2: 00007f54678ebe90 CR3: 0000000001998000 CR4: 00000000001407e0 Nov 16 12:18:49 sanhost kernel: Stack: Nov 16 12:18:49 sanhost kernel: ffff88082a02fcf8 ffffffffa0996d28 ffff88082a02fcf8 ffffffff00000001 Nov 16 12:18:49 sanhost kernel: 00000000c614c614 ffff88080eee7300 0000000000000001 ffff88080eee7510 Nov 16 12:18:49 sanhost kernel: 0000000000000001 ffff8808470fb800 ffff88082a02fd28 ffffffffa0996dec Nov 16 12:18:49 sanhost kernel: Call Trace: Nov 16 12:18:49 sanhost kernel: [<ffffffffa0996d28>] __iscsit_free_cmd+0x228/0x250 [iscsi_target_mod] Nov 16 12:18:49 sanhost kernel: [<ffffffffa0996dec>] iscsit_free_cmd+0x9c/0x150 [iscsi_target_mod] Nov 16 12:18:49 sanhost kernel: [<ffffffffa099e253>] iscsit_close_connection+0x393/0x6f0 [iscsi_target_mod] Nov 16 12:18:49 sanhost kernel: [<ffffffff810a3600>] ? wake_up_state+0x20/0x20 Nov 16 12:18:49 sanhost kernel: [<ffffffffa098c203>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod] Nov 16 12:18:49 sanhost kernel: [<ffffffffa099cd95>] iscsi_target_rx_thread+0x235/0xf50 [iscsi_target_mod] Nov 16 12:18:49 sanhost kernel: [<ffffffff810b17cc>] ? pick_next_task_fair+0x1ac/0x870 Nov 16 12:18:49 sanhost kernel: [<ffffffff810125a4>] ? __switch_to+0xe4/0x580 Nov 16 12:18:49 sanhost kernel: [<ffffffff81669b2b>] ? __schedule+0x2eb/0x810 Nov 16 12:18:49 sanhost kernel: [<ffffffffa099cb60>] ? iscsi_target_tx_thread+0x240/0x240 [iscsi_target_mod] Nov 16 12:18:49 sanhost kernel: [<ffffffff810955c8>] kthread+0xd8/0xf0 Nov 16 12:18:49 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 16 12:18:49 sanhost kernel: [<ffffffff8166e518>] ret_from_fork+0x58/0x90 Nov 16 12:18:49 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 16 12:18:49 sanhost kernel: Code: 0f b1 0f 85 c0 74 05 e8 6d 04 a5 ff 48 83 c4 08 48 89 d8 5b 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 0f 1f 44 00 00 66 83 07 02 <48> 8b 7d 08 be 00 02 00 00 e8 40 cf a0 ff 5d c3 66 0f 1f 44 00 Nov 16 12:18:49 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 41726 Nov 16 12:18:49 sanhost kernel: ABORT_TASK: ref_tag: 41726 already complete, skipping Nov 16 12:18:49 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 41726 Nov 16 12:18:49 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 41725 Nov 16 12:18:49 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 41726 Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 39698 Nov 16 12:18:50 sanhost kernel: Unexpected ret: -32 send data 48 Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 39698 Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 39700 Nov 16 12:18:50 sanhost kernel: ABORT_TASK: ref_tag: 39700 already complete, skipping Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 39700 Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 111155 Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 111153 Nov 16 12:18:51 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38107 Nov 16 12:18:51 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38106 Nov 16 12:18:51 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38107 Nov 16 12:18:53 sanhost kernel: INFO: rcu_sched self-detected stall on CPU { 10} (t=60000 jiffies g=117571148 c=117571147 q=0) Nov 16 12:18:53 sanhost kernel: Task dump for CPU 10: Nov 16 12:18:53 sanhost kernel: iscsi_trx R running task 0 23523 2 0x0000008c Nov 16 12:18:53 sanhost kernel: ffff880829206d00 000000001497afca ffff88087fd03d68 ffffffff810a2a98 Nov 16 12:18:53 sanhost kernel: 000000000000000a ffffffff819f8140 ffff88087fd03d88 ffffffff810a612d Nov 16 12:18:53 sanhost kernel: ffff88087fd03d88 000000000000000b ffff88087fd03db8 ffffffff810d3cf0 Nov 16 12:18:53 sanhost kernel: Call Trace: Nov 16 12:18:53 sanhost kernel: <IRQ> [<ffffffff810a2a98>] sched_show_task+0xa8/0x110 Nov 16 12:18:53 sanhost kernel: [<ffffffff810a612d>] dump_cpu_task+0x3d/0x50 Nov 16 12:18:53 sanhost kernel: [<ffffffff810d3cf0>] rcu_dump_cpu_stacks+0x90/0xd0 Nov 16 12:18:53 sanhost kernel: [<ffffffff810d7a37>] rcu_check_callbacks+0x497/0x710 Nov 16 12:18:53 sanhost kernel: [<ffffffff810dc96b>] update_process_times+0x4b/0x80 Nov 16 12:18:53 sanhost kernel: [<ffffffff810ec5f5>] tick_sched_handle.isra.19+0x25/0x60 Nov 16 12:18:53 sanhost kernel: [<ffffffff810ec675>] tick_sched_timer+0x45/0x80 Nov 16 12:18:53 sanhost kernel: [<ffffffff810dd6b7>] __run_hrtimer+0x77/0x1d0 Nov 16 12:18:53 sanhost kernel: [<ffffffff810ec630>] ? tick_sched_handle.isra.19+0x60/0x60 Nov 16 12:18:53 sanhost kernel: [<ffffffff810ddaa7>] hrtimer_interrupt+0xf7/0x240 Nov 16 12:18:53 sanhost kernel: [<ffffffff8104bc3b>] local_apic_timer_interrupt+0x3b/0x70 Nov 16 12:18:53 sanhost kernel: [<ffffffff81671365>] smp_apic_timer_interrupt+0x45/0x60 Nov 16 12:18:53 sanhost kernel: [<ffffffff8166f43d>] apic_timer_interrupt+0x6d/0x80 Nov 16 12:18:53 sanhost kernel: <EOI> [<ffffffff8166dd4b>] ? _raw_spin_unlock_irqrestore+0x1b/0x40 Nov 16 12:18:53 sanhost kernel: [<ffffffffa04f965b>] transport_wait_for_tasks+0xbb/0x150 [target_core_mod] Nov 16 12:18:53 sanhost kernel: [<ffffffffa04fabed>] transport_generic_free_cmd+0x23d/0x310 [target_core_mod] Nov 16 12:18:53 sanhost kernel: [<ffffffffa0996dc3>] iscsit_free_cmd+0x73/0x150 [iscsi_target_mod] Nov 16 12:18:53 sanhost kernel: [<ffffffffa099e253>] iscsit_close_connection+0x393/0x6f0 [iscsi_target_mod] Nov 16 12:18:53 sanhost kernel: [<ffffffff810a3600>] ? wake_up_state+0x20/0x20 Nov 16 12:18:53 sanhost kernel: [<ffffffffa098c203>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod] Nov 16 12:18:53 sanhost kernel: [<ffffffffa099cd95>] iscsi_target_rx_thread+0x235/0xf50 [iscsi_target_mod] Nov 16 12:18:53 sanhost kernel: [<ffffffff810b17cc>] ? pick_next_task_fair+0x1ac/0x870 Nov 16 12:18:53 sanhost kernel: [<ffffffff810125a4>] ? __switch_to+0xe4/0x580 Nov 16 12:18:53 sanhost kernel: [<ffffffff81669b2b>] ? __schedule+0x2eb/0x810 Nov 16 12:18:53 sanhost kernel: [<ffffffffa099cb60>] ? iscsi_target_tx_thread+0x240/0x240 [iscsi_target_mod] Nov 16 12:18:53 sanhost kernel: [<ffffffff810955c8>] kthread+0xd8/0xf0 Nov 16 12:18:53 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 16 12:18:53 sanhost kernel: [<ffffffff8166e518>] ret_from_fork+0x58/0x90 Nov 16 12:18:53 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 16 12:19:16 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 38828 Nov 16 12:19:16 sanhost kernel: ABORT_TASK: ref_tag: 38828 already complete, skipping Nov 16 12:19:16 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38828 Nov 16 12:19:16 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38826 Nov 16 12:19:16 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38828 Nov 16 12:19:20 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37757 Nov 16 12:19:20 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37754 Nov 16 12:19:21 sanhost kernel: NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [iscsi_trx:23523] Nov 16 12:19:21 sanhost kernel: Modules linked in: binfmt_misc target_core_pscsi target_core_file target_core_iblock dm_round_robin iscsi_tcp libiscsi_tcp raid0 raid1 mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase bonding dm_service_time xprtrdma sunrpc vfat fat ib_isert iscsi_target_mod intel_rapl x86_pkg_temp_thermal intel_powerclamp ib_iser coretemp libiscsi kvm_intel scsi_transport_iscsi kvm ib_srpt target_core_mod bnx2x crct10dif_pclmul crc32_pclmul iTCO_wdt crc32c_intel iTCO_vendor_support ghash_clmulni_intel cryptd pcspkr sb_edac mei_me edac_core mei ioatdma ib_srp libcrc32c lpc_ich i2c_i801 mfd_core scsi_transport_srp ipmi_devintf ib_ipoib ipmi_si ipmi_msghandler rdma_ucm ib_ucm ib_uverbs wmi shpchp ib_umad acpi_power_meter acpi_pad rdma_cm ib_cm iw_cm dm_multipath ext4 mbcache jbd2 mlx4_ib Nov 16 12:19:21 sanhost kernel: mlx4_en ib_sa ib_mad ib_core vxlan ip6_udp_tunnel udp_tunnel ib_addr sd_mod ast syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ahci libahci mlx4_core drm ixgbe libata megaraid_sas mdio ptp i2c_core pps_core dca dm_mirror dm_region_hash dm_log dm_mod Nov 16 12:19:21 sanhost kernel: CPU: 10 PID: 23523 Comm: iscsi_trx Tainted: G L 3.18.16 #2 Nov 16 12:19:21 sanhost kernel: Hardware name: xxxxxx, BIOS 1.0b 12/23/2014 Nov 16 12:19:21 sanhost kernel: task: ffff880829206d00 ti: ffff88082a02c000 task.ti: ffff88082a02c000 Nov 16 12:19:21 sanhost kernel: RIP: 0010:[<ffffffff8166e09e>] [<ffffffff8166e09e>] _raw_spin_lock_bh+0x1e/0x60 Nov 16 12:19:21 sanhost kernel: RSP: 0018:ffff88082a02fca8 EFLAGS: 00000286 Nov 16 12:19:21 sanhost kernel: RAX: 000000008ce88ce8 RBX: 00000000dcd8a6dc RCX: ffff88080eee7408 Nov 16 12:19:21 sanhost kernel: RDX: 0000000000008a7a RSI: 00000000fffffe01 RDI: ffff880488ca7bf0 Nov 16 12:19:21 sanhost kernel: RBP: ffff88082a02fca8 R08: ffff88080eee7500 R09: 0000000000000101 Nov 16 12:19:21 sanhost kernel: R10: 0000000000002710 R11: 0000000000000000 R12: 0000000000000296 Nov 16 12:19:21 sanhost kernel: R13: 0000000000000101 R14: 0000000000002710 R15: 0000000000000000 Nov 16 12:19:21 sanhost kernel: FS: 0000000000000000(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000 Nov 16 12:19:21 sanhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 16 12:19:21 sanhost kernel: CR2: 00007f54678ebe90 CR3: 0000000001998000 CR4: 00000000001407e0 Nov 16 12:19:21 sanhost kernel: Stack: Nov 16 12:19:21 sanhost kernel: ffff88082a02fcf8 ffffffffa0996be8 ffff88082a02fcf8 ffffffff00000001 Nov 16 12:19:21 sanhost kernel: 0000000000000000 ffff88080eee7300 0000000000000001 ffff88080eee7510 Nov 16 12:19:21 sanhost kernel: 0000000000000001 ffff8808470fb800 ffff88082a02fd28 ffffffffa0996db8 Nov 16 12:19:21 sanhost kernel: Call Trace: Nov 16 12:19:21 sanhost kernel: [<ffffffffa0996be8>] __iscsit_free_cmd+0xe8/0x250 [iscsi_target_mod] Nov 16 12:19:21 sanhost kernel: [<ffffffffa0996db8>] iscsit_free_cmd+0x68/0x150 [iscsi_target_mod] Nov 16 12:19:21 sanhost kernel: [<ffffffffa099e253>] iscsit_close_connection+0x393/0x6f0 [iscsi_target_mod] Nov 16 12:19:21 sanhost kernel: [<ffffffff810a3600>] ? wake_up_state+0x20/0x20 Nov 16 12:19:21 sanhost kernel: [<ffffffffa098c203>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod] Nov 16 12:19:21 sanhost kernel: [<ffffffffa099cd95>] iscsi_target_rx_thread+0x235/0xf50 [iscsi_target_mod] Nov 16 12:19:21 sanhost kernel: [<ffffffff810b17cc>] ? pick_next_task_fair+0x1ac/0x870 Nov 16 12:19:21 sanhost kernel: [<ffffffff810125a4>] ? __switch_to+0xe4/0x580 Nov 16 12:19:21 sanhost kernel: [<ffffffff81669b2b>] ? __schedule+0x2eb/0x810 Nov 16 12:19:21 sanhost kernel: [<ffffffffa099cb60>] ? iscsi_target_tx_thread+0x240/0x240 [iscsi_target_mod] Nov 16 12:19:21 sanhost kernel: [<ffffffff810955c8>] kthread+0xd8/0xf0 Nov 16 12:19:21 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 16 12:19:21 sanhost kernel: [<ffffffff8166e518>] ret_from_fork+0x58/0x90 Nov 16 12:19:21 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 Nov 16 12:19:21 sanhost kernel: Code: 80 00 00 00 00 eb da 66 0f 1f 44 00 00 0f 1f 44 00 00 55 65 81 04 25 a0 b7 00 00 00 02 00 00 48 89 e5 b8 00 00 02 00 f0 0f c1 07 <89> c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 Nov 16 12:19:25 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 37625 Nov 16 12:19:25 sanhost kernel: ABORT_TASK: ref_tag: 37625 already complete, skipping Nov 16 12:19:25 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37625 Nov 16 12:19:25 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 37621 Nov 16 12:19:25 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 37621 Nov 16 12:19:31 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 38106 Nov 16 12:19:31 sanhost kernel: ABORT_TASK: ref_tag: 38106 already complete, skipping Nov 16 12:19:31 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38106 Nov 16 12:19:31 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38105 Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38724 Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38724 Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 38890 Nov 16 12:19:32 sanhost kernel: Unexpected ret: -32 send data 48 Nov 16 12:19:32 sanhost kernel: ABORT_TASK: ref_tag: 38890 already complete, skipping Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38890 Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 38891 Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 38891 Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 38894 Nov 16 12:19:32 sanhost kernel: ABORT_TASK: ref_tag: 38894 already complete, skipping Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38894 Nov 16 12:19:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38826 Nov 16 12:19:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38823 Nov 16 12:19:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38826 Nov 16 12:19:49 sanhost kernel: NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [iscsi_trx:23523] thanks, Vyas -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
