Hi Vedavyas, Apologies for the delayed follow-up. Comments below. On Tue, 2015-11-17 at 19:51 -0500, Vedavyas Duggirala wrote: > Hi, > > I have a CentOS 7.1 box with vanilla 3.18.16 kernel exporting a single > lun to a ~20 node ESXi 6.0 cluster. I have run into the soft lockup > issue multiple times in the past few weeks. > > There was no IO except for VMware heartbeat, when the soft lockup > happened, so I know this is not a load issue. > > This looks exactly like a duplicate of issue described in > http://www.spinics.net/lists/target-devel/msg09204.html and I have the > patches suggested there > http://www.spinics.net/lists/target-devel/msg07060.html applied > This particular bug, I believe, was identified as a rbd client reconnect bug, outside of target code. Can you please confirm which storage backend + configuration that you are using..? > The problem manifests on 3.18.21 too. In all cases, the machine > eventually dies due to memory leak caused by rcu stall. After the > first soft lockup, all the nodes in the cluster lost connection to the > lun. Oddly enough, this happens only when the machine is barely doing > any IO. > > Let me know, if you need any further details > So really there are a couple of different things going on. First, if you start to see constant ABORT_TASK from ESX with iSCSI, it means that your storage backend is unable to keep up with the workload. That is, ESX iSCSI has a hard-coded 5 second I/O timeout, that if a outstanding I/O is not completed within that time while the iSCSI path is still active, ESX starts to generate ABORT_TASKs. Seeing these occasionally on a loaded system is not unusual. Seeing them ongoing at multiple times a second as in your logs below, means your backend is having trouble completing I/Os within that 5 second latency requirement for ESX iSCSI. As mentioned in the thread above, you can try reducing the default_cmdsn_depth or NodeACL cmdsn_depth on the target side, to reduce the number of active I/Os each initiator can keep in flight in parallel. Since your using such a larger number of ESX hosts (~20) connected to a single target, the default_cmdsn_depth = 64 value is certainly too high for you. Wrt to the RCU stalls + hung task, the logs below would tend to indicate that your backend has stopped completing outstanding I/Os back to target_core_mod. Please confirm your backend storage configuration. --nab > Nov 15 10:16:22 sanhost kernel: Detected MISCOMPARE for addr: ffff88083b432000 buf: ffff880843038600 > Nov 15 10:16:22 sanhost kernel: Target/iblock: Send MISCOMPARE check condition and sense > > Nov 15 12:10:05 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3289777 > Nov 15 12:10:07 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 3289777 > Nov 15 12:10:26 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435674 > Nov 15 12:10:26 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435674 > Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496040 > Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496040 > Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483359 > Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483359 > Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798798 > Nov 15 12:10:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798798 > Nov 15 12:10:35 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552310 > Nov 15 12:10:35 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552310 > Nov 15 12:10:35 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552310 > Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 380154 > Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 380154 > Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380154 > Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380154 > Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 380155 > Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 380155 > Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435681 > Nov 15 12:10:39 sanhost kernel: ABORT_TASK: ref_tag: 435681 already complete, skipping > Nov 15 12:10:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435681 > Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289781 > Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289780 > Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289781 > Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483366 > Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1798804 > Nov 15 12:10:42 sanhost kernel: ABORT_TASK: ref_tag: 1798804 already complete, skipping > Nov 15 12:10:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798804 > Nov 15 12:10:55 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496050 > Nov 15 12:10:55 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289788 > Nov 15 12:10:56 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798810 > Nov 15 12:11:02 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552322 > Nov 15 12:11:06 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380167 > Nov 15 12:11:07 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435690 > Nov 15 12:11:09 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289794 > Nov 15 12:11:09 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483375 > Nov 15 12:11:23 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798819 > Nov 15 12:11:29 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552330 > Nov 15 12:11:33 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380176 > Nov 15 12:11:34 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435698 > Nov 15 12:11:34 sanhost kernel: ABORT_TASK: ref_tag: 435698 already complete, skipping > Nov 15 12:11:34 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435698 > Nov 15 12:11:36 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3289803 > Nov 15 12:11:36 sanhost kernel: ABORT_TASK: ref_tag: 3289803 already complete, skipping > Nov 15 12:11:36 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289803 > Nov 15 12:11:37 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798824 > Nov 15 12:11:39 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3496055 > Nov 15 12:11:39 sanhost kernel: Unable to locate ITT: 0x00355877 on CID: 0 > Nov 15 12:11:39 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 3496055 > Nov 15 12:11:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496055 > Nov 15 12:11:39 sanhost kernel: Unable to locate RefTaskTag: 0x00355877 on CID: 0. > Nov 15 12:11:43 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552336 > Nov 15 12:11:47 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 380182 > Nov 15 12:11:47 sanhost kernel: ABORT_TASK: ref_tag: 380182 already complete, skipping > Nov 15 12:11:47 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380182 > Nov 15 12:11:47 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435704 > Nov 15 12:11:53 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483379 > Nov 15 12:11:53 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483379 > Nov 15 12:11:53 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483379 > Nov 15 12:12:03 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289811 > Nov 15 12:12:04 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798840 > Nov 15 12:12:06 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496066 > Nov 15 12:12:10 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552345 > Nov 15 12:12:14 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380190 > Nov 15 12:12:15 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435713 > Nov 15 12:12:15 sanhost kernel: ABORT_TASK: ref_tag: 435713 already complete, skipping > Nov 15 12:12:15 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435713 > Nov 15 12:12:17 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289817 > Nov 15 12:12:18 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798845 > Nov 15 12:12:28 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380196 > Nov 15 12:12:31 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798852 > Nov 15 12:12:34 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496081 > Nov 15 12:12:37 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3552353 > Nov 15 12:12:37 sanhost kernel: ABORT_TASK: ref_tag: 3552353 already complete, skipping > Nov 15 12:12:37 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552353 > Nov 15 12:12:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435721 > Nov 15 12:12:44 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289832 > Nov 15 12:12:45 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1798857 > Nov 15 12:12:45 sanhost kernel: ABORT_TASK: ref_tag: 1798857 already complete, skipping > Nov 15 12:12:45 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798857 > Nov 15 12:12:47 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3496087 > Nov 15 12:12:47 sanhost kernel: ABORT_TASK: ref_tag: 3496087 already complete, skipping > Nov 15 12:12:47 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496087 > Nov 15 12:12:50 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1483391 > Nov 15 12:12:50 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 1483391 > Nov 15 12:12:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483391 > Nov 15 12:12:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483391 > Nov 15 12:12:55 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38020 > Nov 15 12:13:04 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483399 > Nov 15 12:13:05 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552362 > Nov 15 12:13:09 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380210 > Nov 15 12:13:11 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289840 > Nov 15 12:13:12 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1798866 > Nov 15 12:13:12 sanhost kernel: ABORT_TASK: ref_tag: 1798866 already complete, skipping > Nov 15 12:13:12 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798866 > Nov 15 12:13:14 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496096 > Nov 15 12:13:18 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3552368 > Nov 15 12:13:18 sanhost kernel: ABORT_TASK: ref_tag: 3552368 already complete, skipping > Nov 15 12:13:18 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552368 > Nov 15 12:13:23 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435735 > Nov 15 12:13:23 sanhost kernel: ABORT_TASK: ref_tag: 435735 already complete, skipping > Nov 15 12:13:23 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435735 > Nov 15 12:13:25 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289846 > Nov 15 12:13:26 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798872 > Nov 15 12:13:36 sanhost kernel: Unexpected ret: -32 send data 48 > Nov 15 12:13:36 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 380225 > Nov 15 12:13:36 sanhost kernel: ABORT_TASK: ref_tag: 380225 already complete, skipping > Nov 15 12:13:36 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380225 > Nov 15 12:13:41 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496104 > Nov 15 12:13:45 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552377 > Nov 15 12:13:48 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1483403 > Nov 15 12:13:48 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 1483403 > Nov 15 12:13:48 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483403 > Nov 15 12:13:48 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483403 > Nov 15 12:13:50 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435743 > Nov 15 12:13:50 sanhost kernel: ABORT_TASK: ref_tag: 435743 already complete, skipping > Nov 15 12:13:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435743 > Nov 15 12:13:52 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3289855 > Nov 15 12:13:52 sanhost kernel: ABORT_TASK: ref_tag: 3289855 already complete, skipping > Nov 15 12:13:52 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289855 > Nov 15 12:13:53 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1798881 > Nov 15 12:13:53 sanhost kernel: ABORT_TASK: ref_tag: 1798881 already complete, skipping > Nov 15 12:13:53 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798881 > Nov 15 12:14:01 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483410 > Nov 15 12:14:03 sanhost kernel: Unexpected ret: -32 send data 48 > Nov 15 12:14:03 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 380234 > Nov 15 12:14:03 sanhost kernel: ABORT_TASK: ref_tag: 380234 already complete, skipping > Nov 15 12:14:03 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380234 > Nov 15 12:14:03 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435749 > Nov 15 12:14:03 sanhost kernel: ABORT_TASK: ref_tag: 435749 already complete, skipping > Nov 15 12:14:03 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435749 > Nov 15 12:14:07 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798886 > Nov 15 12:14:08 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496113 > Nov 15 12:14:12 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552385 > Nov 15 12:14:20 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3289863 > Nov 15 12:14:20 sanhost kernel: ABORT_TASK: ref_tag: 3289863 already complete, skipping > Nov 15 12:14:20 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289863 > Nov 15 12:14:22 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3496119 > Nov 15 12:14:22 sanhost kernel: ABORT_TASK: ref_tag: 3496119 already complete, skipping > Nov 15 12:14:22 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496119 > Nov 15 12:14:26 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552397 > Nov 15 12:14:30 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380246 > Nov 15 12:14:31 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 435765 > Nov 15 12:14:31 sanhost kernel: ABORT_TASK: ref_tag: 435765 already complete, skipping > Nov 15 12:14:31 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435765 > Nov 15 12:14:33 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289869 > Nov 15 12:14:34 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798894 > Nov 15 12:14:42 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483424 > Nov 15 12:14:44 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 380252 > Nov 15 12:14:49 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3496127 > Nov 15 12:14:53 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3552407 > Nov 15 12:14:53 sanhost kernel: ABORT_TASK: ref_tag: 3552407 already complete, skipping > Nov 15 12:14:53 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3552407 > Nov 15 12:14:57 sanhost kernel: TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000001 > Nov 15 12:14:58 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 435773 > Nov 15 12:15:00 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 3289877 > Nov 15 12:15:00 sanhost kernel: ABORT_TASK: ref_tag: 3289877 already complete, skipping > Nov 15 12:15:00 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3289877 > Nov 15 12:15:01 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1798903 > Nov 15 12:15:01 sanhost kernel: ABORT_TASK: ref_tag: 1798903 already complete, skipping > Nov 15 12:15:01 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1798903 > Nov 15 12:15:09 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1483432 > Nov 15 12:15:11 sanhost kernel: TARGET_CORE[iSCSI]: Detected NON_EXISTENT_LUN Access for 0x00000001 > Nov 16 12:17:56 sanhost kernel: Unexpected ret: -32 send data 48 > Nov 16 12:17:57 sanhost kernel: Unexpected ret: -32 send data 48 > Nov 16 12:17:58 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 3331838 > Nov 16 12:17:59 sanhost kernel: Unexpected ret: -32 send data 48 > Nov 16 12:18:01 sanhost kernel: Unexpected ret: -32 send data 48 > Nov 16 12:18:03 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 1526495 > Nov 16 12:18:03 sanhost kernel: ABORT_TASK: ref_tag: 1526495 already complete, skipping > Nov 16 12:18:03 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1526495 > Nov 16 12:18:05 sanhost kernel: Unexpected ret: -32 send data 48 > Nov 16 12:18:07 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 1837659 > Nov 16 12:18:08 sanhost kernel: Unexpected ret: -32 send data 48 > Nov 16 12:18:08 sanhost kernel: iSCSI Login timeout on Network Portal 192.168.10.2:3260 > Nov 16 12:18:21 sanhost kernel: NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [iscsi_trx:23523] > Nov 16 12:18:21 sanhost kernel: Modules linked in: binfmt_misc target_core_pscsi target_core_file target_core_iblock dm_round_robin > iscsi_tcp libiscsi_tcp raid0 raid1 mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase bonding dm_service_time xprtrdma > sunrpc vfat fat ib_isert iscsi_target_mod intel_rapl x86_pkg_temp_thermal intel_powerclamp ib_iser coretemp libiscsi > kvm_intel scsi_transport_iscsi kvm ib_srpt target_core_mod bnx2x crct10dif_pclmul crc32_pclmul iTCO_wdt crc32c_intel > iTCO_vendor_support ghash_clmulni_intel cryptd pcspkr sb_edac mei_me edac_core mei ioatdma ib_srp libcrc32c lpc_ich i2c_i801 mfd_core > scsi_transport_srp ipmi_devintf ib_ipoib ipmi_si ipmi_msghandler rdma_ucm ib_ucm ib_uverbs wmi shpchp ib_umad acpi_power_meter acpi_pad > rdma_cm ib_cm iw_cm dm_multipath ext4 mbcache jbd2 mlx4_ib > Nov 16 12:18:21 sanhost kernel: mlx4_en ib_sa ib_mad ib_core vxlan ip6_udp_tunnel udp_tunnel ib_addr sd_mod ast syscopyarea sysfillrect > sysimgblt i2c_algo_bit drm_kms_helper ttm ahci libahci mlx4_core drm ixgbe libata megaraid_sas mdio ptp i2c_core pps_core dca dm_mirror > dm_region_hash dm_log dm_mod > Nov 16 12:18:21 sanhost kernel: CPU: 10 PID: 23523 Comm: iscsi_trx Not tainted 3.18.16 #2 > Nov 16 12:18:21 sanhost kernel: Hardware name: xxxxxx, BIOS 1.0b 12/23/2014 > Nov 16 12:18:21 sanhost kernel: task: ffff880829206d00 ti: ffff88082a02c000 task.ti: ffff88082a02c000 > Nov 16 12:18:21 sanhost kernel: RIP: 0010:[<ffffffffa098c5b6>][<ffffffffa098c5b6>] iscsit_stop_dataout_timer+0x6/0x80 [iscsi_target_mod] > Nov 16 12:18:21 sanhost kernel: RSP: 0018:ffff88082a02fca8 EFLAGS: 00000246 > Nov 16 12:18:21 sanhost kernel: RAX: 0000000000000001 RBX: ffff88080eee7408 RCX: 00000000f507d5c8 > Nov 16 12:18:21 sanhost kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88080eee7300 > Nov 16 12:18:21 sanhost kernel: RBP: ffff88082a02fcf8 R08: 0000000000000296 R09: 0000000000000101 > Nov 16 12:18:21 sanhost kernel: R10: 0000000000002710 R11: 0000000000000000 R12: ffff88080eee7500 > Nov 16 12:18:21 sanhost kernel: R13: 0000000000000101 R14: 0000000000002710 R15: 0000000000000000 > Nov 16 12:18:21 sanhost kernel: FS: 0000000000000000(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000 > Nov 16 12:18:21 sanhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > Nov 16 12:18:21 sanhost kernel: CR2: 00007f54678ebe90 CR3: 0000000001998000 CR4: 00000000001407e0 > Nov 16 12:18:21 sanhost kernel: Stack: > Nov 16 12:18:21 sanhost kernel: ffff88082a02fcf8 ffffffffa0996cf0 ffff88082a02fcf8 ffffffff00000001 > Nov 16 12:18:21 sanhost kernel: ffff88080eee7300 ffff88080eee7300 0000000000000001 ffff88080eee7510 > Nov 16 12:18:21 sanhost kernel: 0000000000000001 ffff8808470fb800 ffff88082a02fd28 ffffffffa0996dec > Nov 16 12:18:21 sanhost kernel: Call Trace: > Nov 16 12:18:21 sanhost kernel: [<ffffffffa0996cf0>] ? __iscsit_free_cmd+0x1f0/0x250 [iscsi_target_mod] > Nov 16 12:18:21 sanhost kernel: [<ffffffffa0996dec>] iscsit_free_cmd+0x9c/0x150 [iscsi_target_mod] > Nov 16 12:18:21 sanhost kernel: [<ffffffffa099e253>] iscsit_close_connection+0x393/0x6f0 [iscsi_target_mod] > Nov 16 12:18:21 sanhost kernel: [<ffffffff810a3600>] ? wake_up_state+0x20/0x20 > Nov 16 12:18:21 sanhost kernel: [<ffffffffa098c203>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod] > Nov 16 12:18:21 sanhost kernel: [<ffffffffa099cd95>] iscsi_target_rx_thread+0x235/0xf50 [iscsi_target_mod] > Nov 16 12:18:21 sanhost kernel: [<ffffffff810b17cc>] ? pick_next_task_fair+0x1ac/0x870 > Nov 16 12:18:21 sanhost kernel: [<ffffffff810125a4>] ? __switch_to+0xe4/0x580 > Nov 16 12:18:21 sanhost kernel: [<ffffffff81669b2b>] ? __schedule+0x2eb/0x810 > Nov 16 12:18:21 sanhost kernel: [<ffffffffa099cb60>] ? iscsi_target_tx_thread+0x240/0x240 [iscsi_target_mod] > Nov 16 12:18:21 sanhost kernel: [<ffffffff810955c8>] kthread+0xd8/0xf0 > Nov 16 12:18:21 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 > Nov 16 12:18:21 sanhost kernel: [<ffffffff8166e518>] ret_from_fork+0x58/0x90 > Nov 16 12:18:21 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 > Nov 16 12:18:21 sanhost kernel: Code: 80 00 00 00 00 29 b3 88 00 00 00 e9 ea fe ff ff b8 ff ff ff ff e9 21 ff ff ff 66 66 2e 0f 1f 84 00 00 > 00 00 00 0f 1f 44 00 00 55 <48> 89 e5 41 54 4c 8d a7 e4 00 00 00 53 48 89 fb 4c 89 e7 e8 b2 > Nov 16 12:18:35 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38782 > Nov 16 12:18:35 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38781 > Nov 16 12:18:40 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37724 > Nov 16 12:18:40 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37723 > Nov 16 12:18:40 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37724 > Nov 16 12:18:46 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38898 > Nov 16 12:18:46 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38898 > Nov 16 12:18:49 sanhost kernel: NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [iscsi_trx:23523] > Nov 16 12:18:49 sanhost kernel: Modules linked in: binfmt_misc target_core_pscsi target_core_file target_core_iblock dm_round_robin > iscsi_tcp libiscsi_tcp raid0 raid1 mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase bonding dm_service_time xprtrdma > sunrpc vfat fat ib_isert iscsi_target_mod intel_rapl x86_pkg_temp_thermal intel_powerclamp ib_iser coretemp libiscsi > kvm_intel scsi_transport_iscsi kvm ib_srpt target_core_mod bnx2x crct10dif_pclmul crc32_pclmul iTCO_wdt crc32c_intel > iTCO_vendor_support ghash_clmulni_intel cryptd pcspkr sb_edac mei_me edac_core mei ioatdma ib_srp libcrc32c lpc_ich i2c_i801 mfd_core > scsi_transport_srp ipmi_devintf ib_ipoib ipmi_si ipmi_msghandler rdma_ucm ib_ucm ib_uverbs wmi shpchp ib_umad acpi_power_meter acpi_pad > rdma_cm ib_cm iw_cm dm_multipath ext4 mbcache jbd2 mlx4_ib > Nov 16 12:18:49 sanhost kernel: mlx4_en ib_sa ib_mad ib_core vxlan ip6_udp_tunnel udp_tunnel ib_addr sd_mod ast syscopyarea sysfillrect > sysimgblt i2c_algo_bit drm_kms_helper ttm ahci libahci mlx4_core drm ixgbe libata megaraid_sas mdio ptp i2c_core pps_core dca dm_mirror > dm_region_hash dm_log dm_mod > Nov 16 12:18:49 sanhost kernel: CPU: 10 PID: 23523 Comm: iscsi_trx Tainted: G L 3.18.16 #2 > Nov 16 12:18:49 sanhost kernel: Hardware name: xxxxxx, BIOS 1.0b 12/23/2014 > Nov 16 12:18:49 sanhost kernel: task: ffff880829206d00 ti: ffff88082a02c000 task.ti: ffff88082a02c000 > Nov 16 12:18:49 sanhost kernel: RIP: 0010:[<ffffffff8166e002>] [<ffffffff8166e002>] _raw_spin_unlock_bh+0x12/0x40 > Nov 16 12:18:49 sanhost kernel: RSP: 0018:ffff88082a02fca8 EFLAGS: 00000202 > Nov 16 12:18:49 sanhost kernel: RAX: 0000000000000000 RBX: ffff88080eee7500 RCX: ffff88080eee7408 > Nov 16 12:18:49 sanhost kernel: RDX: 0000000000003506 RSI: 00000000fffffe01 RDI: ffff880488ca7bf0 > Nov 16 12:18:49 sanhost kernel: RBP: ffff88082a02fca8 R08: 0000000000000296 R09: 0000000000000101 > Nov 16 12:18:49 sanhost kernel: R10: 0000000000002710 R11: 000000000000000a R12: 0000000000002710 > Nov 16 12:18:49 sanhost kernel: R13: 000000000000000a R14: 0000000000002710 R15: ffff88082a02fc88 > Nov 16 12:18:49 sanhost kernel: FS: 0000000000000000(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000 > Nov 16 12:18:49 sanhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > Nov 16 12:18:49 sanhost kernel: CR2: 00007f54678ebe90 CR3: 0000000001998000 CR4: 00000000001407e0 > Nov 16 12:18:49 sanhost kernel: Stack: > Nov 16 12:18:49 sanhost kernel: ffff88082a02fcf8 ffffffffa0996d28 ffff88082a02fcf8 ffffffff00000001 > Nov 16 12:18:49 sanhost kernel: 00000000c614c614 ffff88080eee7300 0000000000000001 ffff88080eee7510 > Nov 16 12:18:49 sanhost kernel: 0000000000000001 ffff8808470fb800 ffff88082a02fd28 ffffffffa0996dec > Nov 16 12:18:49 sanhost kernel: Call Trace: > Nov 16 12:18:49 sanhost kernel: [<ffffffffa0996d28>] __iscsit_free_cmd+0x228/0x250 [iscsi_target_mod] > Nov 16 12:18:49 sanhost kernel: [<ffffffffa0996dec>] iscsit_free_cmd+0x9c/0x150 [iscsi_target_mod] > Nov 16 12:18:49 sanhost kernel: [<ffffffffa099e253>] iscsit_close_connection+0x393/0x6f0 [iscsi_target_mod] > Nov 16 12:18:49 sanhost kernel: [<ffffffff810a3600>] ? wake_up_state+0x20/0x20 > Nov 16 12:18:49 sanhost kernel: [<ffffffffa098c203>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod] > Nov 16 12:18:49 sanhost kernel: [<ffffffffa099cd95>] iscsi_target_rx_thread+0x235/0xf50 [iscsi_target_mod] > Nov 16 12:18:49 sanhost kernel: [<ffffffff810b17cc>] ? pick_next_task_fair+0x1ac/0x870 > Nov 16 12:18:49 sanhost kernel: [<ffffffff810125a4>] ? __switch_to+0xe4/0x580 > Nov 16 12:18:49 sanhost kernel: [<ffffffff81669b2b>] ? __schedule+0x2eb/0x810 > Nov 16 12:18:49 sanhost kernel: [<ffffffffa099cb60>] ? iscsi_target_tx_thread+0x240/0x240 [iscsi_target_mod] > Nov 16 12:18:49 sanhost kernel: [<ffffffff810955c8>] kthread+0xd8/0xf0 > Nov 16 12:18:49 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 > Nov 16 12:18:49 sanhost kernel: [<ffffffff8166e518>] ret_from_fork+0x58/0x90 > Nov 16 12:18:49 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 > Nov 16 12:18:49 sanhost kernel: Code: 0f b1 0f 85 c0 74 05 e8 6d 04 a5 ff 48 83 c4 08 48 89 d8 5b 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 > 0f 1f 44 00 00 66 83 07 02 <48> 8b 7d 08 be 00 02 00 00 e8 40 cf a0 ff 5d c3 66 0f 1f 44 00 > Nov 16 12:18:49 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 41726 > Nov 16 12:18:49 sanhost kernel: ABORT_TASK: ref_tag: 41726 already complete, skipping > Nov 16 12:18:49 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 41726 > Nov 16 12:18:49 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 41725 > Nov 16 12:18:49 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 41726 > Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 39698 > Nov 16 12:18:50 sanhost kernel: Unexpected ret: -32 send data 48 > Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 39698 > Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 39700 > Nov 16 12:18:50 sanhost kernel: ABORT_TASK: ref_tag: 39700 already complete, skipping > Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 39700 > Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 111155 > Nov 16 12:18:50 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 111153 > Nov 16 12:18:51 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38107 > Nov 16 12:18:51 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38106 > Nov 16 12:18:51 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38107 > Nov 16 12:18:53 sanhost kernel: INFO: rcu_sched self-detected stall on CPU { 10} (t=60000 jiffies g=117571148 c=117571147 q=0) > Nov 16 12:18:53 sanhost kernel: Task dump for CPU 10: > Nov 16 12:18:53 sanhost kernel: iscsi_trx R running task 0 23523 2 0x0000008c > Nov 16 12:18:53 sanhost kernel: ffff880829206d00 000000001497afca ffff88087fd03d68 ffffffff810a2a98 > Nov 16 12:18:53 sanhost kernel: 000000000000000a ffffffff819f8140 ffff88087fd03d88 ffffffff810a612d > Nov 16 12:18:53 sanhost kernel: ffff88087fd03d88 000000000000000b ffff88087fd03db8 ffffffff810d3cf0 > Nov 16 12:18:53 sanhost kernel: Call Trace: > Nov 16 12:18:53 sanhost kernel: <IRQ> [<ffffffff810a2a98>] sched_show_task+0xa8/0x110 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810a612d>] dump_cpu_task+0x3d/0x50 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810d3cf0>] rcu_dump_cpu_stacks+0x90/0xd0 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810d7a37>] rcu_check_callbacks+0x497/0x710 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810dc96b>] update_process_times+0x4b/0x80 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810ec5f5>] tick_sched_handle.isra.19+0x25/0x60 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810ec675>] tick_sched_timer+0x45/0x80 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810dd6b7>] __run_hrtimer+0x77/0x1d0 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810ec630>] ? tick_sched_handle.isra.19+0x60/0x60 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810ddaa7>] hrtimer_interrupt+0xf7/0x240 > Nov 16 12:18:53 sanhost kernel: [<ffffffff8104bc3b>] local_apic_timer_interrupt+0x3b/0x70 > Nov 16 12:18:53 sanhost kernel: [<ffffffff81671365>] smp_apic_timer_interrupt+0x45/0x60 > Nov 16 12:18:53 sanhost kernel: [<ffffffff8166f43d>] apic_timer_interrupt+0x6d/0x80 > Nov 16 12:18:53 sanhost kernel: <EOI> [<ffffffff8166dd4b>] ? _raw_spin_unlock_irqrestore+0x1b/0x40 > Nov 16 12:18:53 sanhost kernel: [<ffffffffa04f965b>] transport_wait_for_tasks+0xbb/0x150 [target_core_mod] > Nov 16 12:18:53 sanhost kernel: [<ffffffffa04fabed>] transport_generic_free_cmd+0x23d/0x310 [target_core_mod] > Nov 16 12:18:53 sanhost kernel: [<ffffffffa0996dc3>] iscsit_free_cmd+0x73/0x150 [iscsi_target_mod] > Nov 16 12:18:53 sanhost kernel: [<ffffffffa099e253>] iscsit_close_connection+0x393/0x6f0 [iscsi_target_mod] > Nov 16 12:18:53 sanhost kernel: [<ffffffff810a3600>] ? wake_up_state+0x20/0x20 > Nov 16 12:18:53 sanhost kernel: [<ffffffffa098c203>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod] > Nov 16 12:18:53 sanhost kernel: [<ffffffffa099cd95>] iscsi_target_rx_thread+0x235/0xf50 [iscsi_target_mod] > Nov 16 12:18:53 sanhost kernel: [<ffffffff810b17cc>] ? pick_next_task_fair+0x1ac/0x870 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810125a4>] ? __switch_to+0xe4/0x580 > Nov 16 12:18:53 sanhost kernel: [<ffffffff81669b2b>] ? __schedule+0x2eb/0x810 > Nov 16 12:18:53 sanhost kernel: [<ffffffffa099cb60>] ? iscsi_target_tx_thread+0x240/0x240 [iscsi_target_mod] > Nov 16 12:18:53 sanhost kernel: [<ffffffff810955c8>] kthread+0xd8/0xf0 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 > Nov 16 12:18:53 sanhost kernel: [<ffffffff8166e518>] ret_from_fork+0x58/0x90 > Nov 16 12:18:53 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 > Nov 16 12:19:16 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 38828 > Nov 16 12:19:16 sanhost kernel: ABORT_TASK: ref_tag: 38828 already complete, skipping > Nov 16 12:19:16 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38828 > Nov 16 12:19:16 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38826 > Nov 16 12:19:16 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38828 > Nov 16 12:19:20 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37757 > Nov 16 12:19:20 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37754 > Nov 16 12:19:21 sanhost kernel: NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [iscsi_trx:23523] > Nov 16 12:19:21 sanhost kernel: Modules linked in: binfmt_misc target_core_pscsi target_core_file target_core_iblock dm_round_robin > iscsi_tcp libiscsi_tcp raid0 raid1 mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase bonding dm_service_time xprtrdma > sunrpc vfat fat ib_isert iscsi_target_mod intel_rapl x86_pkg_temp_thermal intel_powerclamp ib_iser coretemp libiscsi > kvm_intel scsi_transport_iscsi kvm ib_srpt target_core_mod bnx2x crct10dif_pclmul crc32_pclmul iTCO_wdt crc32c_intel > iTCO_vendor_support ghash_clmulni_intel cryptd pcspkr sb_edac mei_me edac_core mei ioatdma ib_srp libcrc32c lpc_ich i2c_i801 mfd_core > scsi_transport_srp ipmi_devintf ib_ipoib ipmi_si ipmi_msghandler rdma_ucm ib_ucm ib_uverbs wmi shpchp ib_umad acpi_power_meter acpi_pad > rdma_cm ib_cm iw_cm dm_multipath ext4 mbcache jbd2 mlx4_ib > Nov 16 12:19:21 sanhost kernel: mlx4_en ib_sa ib_mad ib_core vxlan ip6_udp_tunnel udp_tunnel ib_addr sd_mod ast syscopyarea sysfillrect > sysimgblt i2c_algo_bit drm_kms_helper ttm ahci libahci mlx4_core drm ixgbe libata megaraid_sas mdio ptp i2c_core pps_core dca dm_mirror > dm_region_hash dm_log dm_mod > Nov 16 12:19:21 sanhost kernel: CPU: 10 PID: 23523 Comm: iscsi_trx Tainted: G L 3.18.16 #2 > Nov 16 12:19:21 sanhost kernel: Hardware name: xxxxxx, BIOS 1.0b 12/23/2014 > Nov 16 12:19:21 sanhost kernel: task: ffff880829206d00 ti: ffff88082a02c000 task.ti: ffff88082a02c000 > Nov 16 12:19:21 sanhost kernel: RIP: 0010:[<ffffffff8166e09e>] [<ffffffff8166e09e>] _raw_spin_lock_bh+0x1e/0x60 > Nov 16 12:19:21 sanhost kernel: RSP: 0018:ffff88082a02fca8 EFLAGS: 00000286 > Nov 16 12:19:21 sanhost kernel: RAX: 000000008ce88ce8 RBX: 00000000dcd8a6dc RCX: ffff88080eee7408 > Nov 16 12:19:21 sanhost kernel: RDX: 0000000000008a7a RSI: 00000000fffffe01 RDI: ffff880488ca7bf0 > Nov 16 12:19:21 sanhost kernel: RBP: ffff88082a02fca8 R08: ffff88080eee7500 R09: 0000000000000101 > Nov 16 12:19:21 sanhost kernel: R10: 0000000000002710 R11: 0000000000000000 R12: 0000000000000296 > Nov 16 12:19:21 sanhost kernel: R13: 0000000000000101 R14: 0000000000002710 R15: 0000000000000000 > Nov 16 12:19:21 sanhost kernel: FS: 0000000000000000(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000 > Nov 16 12:19:21 sanhost kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > Nov 16 12:19:21 sanhost kernel: CR2: 00007f54678ebe90 CR3: 0000000001998000 CR4: 00000000001407e0 > Nov 16 12:19:21 sanhost kernel: Stack: > Nov 16 12:19:21 sanhost kernel: ffff88082a02fcf8 ffffffffa0996be8 ffff88082a02fcf8 ffffffff00000001 > Nov 16 12:19:21 sanhost kernel: 0000000000000000 ffff88080eee7300 0000000000000001 ffff88080eee7510 > Nov 16 12:19:21 sanhost kernel: 0000000000000001 ffff8808470fb800 ffff88082a02fd28 ffffffffa0996db8 > Nov 16 12:19:21 sanhost kernel: Call Trace: > Nov 16 12:19:21 sanhost kernel: [<ffffffffa0996be8>] __iscsit_free_cmd+0xe8/0x250 [iscsi_target_mod] > Nov 16 12:19:21 sanhost kernel: [<ffffffffa0996db8>] iscsit_free_cmd+0x68/0x150 [iscsi_target_mod] > Nov 16 12:19:21 sanhost kernel: [<ffffffffa099e253>] iscsit_close_connection+0x393/0x6f0 [iscsi_target_mod] > Nov 16 12:19:21 sanhost kernel: [<ffffffff810a3600>] ? wake_up_state+0x20/0x20 > Nov 16 12:19:21 sanhost kernel: [<ffffffffa098c203>] iscsit_take_action_for_connection_exit+0x83/0x110 [iscsi_target_mod] > Nov 16 12:19:21 sanhost kernel: [<ffffffffa099cd95>] iscsi_target_rx_thread+0x235/0xf50 [iscsi_target_mod] > Nov 16 12:19:21 sanhost kernel: [<ffffffff810b17cc>] ? pick_next_task_fair+0x1ac/0x870 > Nov 16 12:19:21 sanhost kernel: [<ffffffff810125a4>] ? __switch_to+0xe4/0x580 > Nov 16 12:19:21 sanhost kernel: [<ffffffff81669b2b>] ? __schedule+0x2eb/0x810 > Nov 16 12:19:21 sanhost kernel: [<ffffffffa099cb60>] ? iscsi_target_tx_thread+0x240/0x240 [iscsi_target_mod] > Nov 16 12:19:21 sanhost kernel: [<ffffffff810955c8>] kthread+0xd8/0xf0 > Nov 16 12:19:21 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 > Nov 16 12:19:21 sanhost kernel: [<ffffffff8166e518>] ret_from_fork+0x58/0x90 > Nov 16 12:19:21 sanhost kernel: [<ffffffff810954f0>] ? kthread_create_on_node+0x1b0/0x1b0 > Nov 16 12:19:21 sanhost kernel: Code: 80 00 00 00 00 eb da 66 0f 1f 44 00 00 0f 1f 44 00 00 55 65 81 04 25 a0 b7 00 00 00 02 00 00 48 > 89 e5 b8 00 00 02 00 f0 0f c1 07 <89> c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 f2 b8 00 80 > Nov 16 12:19:25 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 37625 > Nov 16 12:19:25 sanhost kernel: ABORT_TASK: ref_tag: 37625 already complete, skipping > Nov 16 12:19:25 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 37625 > Nov 16 12:19:25 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 37621 > Nov 16 12:19:25 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 37621 > Nov 16 12:19:31 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 38106 > Nov 16 12:19:31 sanhost kernel: ABORT_TASK: ref_tag: 38106 already complete, skipping > Nov 16 12:19:31 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38106 > Nov 16 12:19:31 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38105 > Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38724 > Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38724 > Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 38890 > Nov 16 12:19:32 sanhost kernel: Unexpected ret: -32 send data 48 > Nov 16 12:19:32 sanhost kernel: ABORT_TASK: ref_tag: 38890 already complete, skipping > Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38890 > Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 38891 > Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 38891 > Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Found referenced iSCSI task_tag: 38894 > Nov 16 12:19:32 sanhost kernel: ABORT_TASK: ref_tag: 38894 already complete, skipping > Nov 16 12:19:32 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38894 > Nov 16 12:19:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38826 > Nov 16 12:19:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38823 > Nov 16 12:19:39 sanhost kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 38826 > Nov 16 12:19:49 sanhost kernel: NMI watchdog: BUG: soft lockup -CPU#10 stuck for 23s! [iscsi_trx:23523] > thanks, > Vyas > -- > To unsubscribe from this list: send the line "unsubscribe target-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
