For what it's worth, I'm most interested in this topic, as I'm seeing the same behaviour with a slightly different setup. We're using an OpenSUSE distribution to start with on the Storage servers, using LSI MegaRAID 9286 Raid controllers, Intel 10GbE NICs, DRBD, etc.. We're running ESX 5.5 on the Virtualization servers, connected to the storage servers with Intel x540-T2 cards (10GbE) through a pair of matched Arista 7050 Switches. Each ESX server has two paths to each storage array, on two different subnets, each travelling through a separate NIC, to the Arista switches, to two different NICs on the storage servers (again, separate subnets so there's no routing confusion). Under heavy load, we see iSCSI timeouts, with the same log entries you've listed. We see things like these entries fairly commonly when under load: [357494.203252] ABORT_TASK: Found referenced iSCSI task_tag: 9402184 [357494.203255] ABORT_TASK: ref_tag: 9402184 already complete, skipping [357494.203256] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 9402184 Once we see these entries, we typically see ESX mark the path as OFFLINE, then ONLINE again, sometimes after performing a LUN scan. Which I interpret (and please correct me if I'm wrong in this) to mean that the ESX server saw a task as timed out, sent the command to abort the command, but the Storage server had already completed the command. This makes me question if the response to the command is being 'queued' somewhere after the iSCSI code has handed it off. We also have large frames enabled, and have tuned the NIC for a TX queue of 10000, as per Intel's 10GbE tuning recommendations. As I write this, I wonder if the TXQueue is possibly to blame? I'll be happy to provide any info I can on our setup. I would dearly love some help in getting to the bottom of it. We've tried tuning the network, the CPUs, the NICs, the storage controllers, changing and tuning the storage queuing mechanism, holding our tongues against our upper right molar, crossing our fingers, etc. We've been looking for a virgin to sacrifice, but haven't yet located one. Any other ideas you folks have would be appreciated. We're currently using the deadline scheduler on each of the arrays (we have 4 arrays per storage server). # modinfo iscsi_target_mod filename: /lib/modules/3.14.1-1.geafcebd-default/kernel/drivers/target/iscsi/iscsi_target_mod.ko license: GPL author: nab@xxxxxxxxxxxxxxx version: 4.1.x description: iSCSI-Target Driver for mainline target infrastructure srcversion: E0226DD23D8EB7FBC028F2A depends: target_core_mod,configfs vermagic: 3.14.1-1.geafcebd-default SMP mod_unload modversions # modinfo ixgbe filename: /lib/modules/3.14.1-1.geafcebd-default/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko version: 3.19.1-k license: GPL description: Intel(R) 10 Gigabit PCI Express Network Driver author: Intel Corporation, <linux.nics@xxxxxxxxx> srcversion: 71477A3DD5379805CF3D489 alias: pci:v00008086d00001560sv*sd*bc*sc*i* alias: pci:v00008086d0000154Asv*sd*bc*sc*i* alias: pci:v00008086d00001557sv*sd*bc*sc*i* alias: pci:v00008086d00001558sv*sd*bc*sc*i* alias: pci:v00008086d0000154Fsv*sd*bc*sc*i* alias: pci:v00008086d0000154Dsv*sd*bc*sc*i* alias: pci:v00008086d00001528sv*sd*bc*sc*i* alias: pci:v00008086d000010F8sv*sd*bc*sc*i* alias: pci:v00008086d0000151Csv*sd*bc*sc*i* alias: pci:v00008086d00001529sv*sd*bc*sc*i* alias: pci:v00008086d0000152Asv*sd*bc*sc*i* alias: pci:v00008086d000010F9sv*sd*bc*sc*i* alias: pci:v00008086d00001514sv*sd*bc*sc*i* alias: pci:v00008086d00001507sv*sd*bc*sc*i* alias: pci:v00008086d000010FBsv*sd*bc*sc*i* alias: pci:v00008086d00001517sv*sd*bc*sc*i* alias: pci:v00008086d000010FCsv*sd*bc*sc*i* alias: pci:v00008086d000010F7sv*sd*bc*sc*i* alias: pci:v00008086d00001508sv*sd*bc*sc*i* alias: pci:v00008086d000010DBsv*sd*bc*sc*i* alias: pci:v00008086d000010F4sv*sd*bc*sc*i* alias: pci:v00008086d000010E1sv*sd*bc*sc*i* alias: pci:v00008086d000010F1sv*sd*bc*sc*i* alias: pci:v00008086d000010ECsv*sd*bc*sc*i* alias: pci:v00008086d000010DDsv*sd*bc*sc*i* alias: pci:v00008086d0000150Bsv*sd*bc*sc*i* alias: pci:v00008086d000010C8sv*sd*bc*sc*i* alias: pci:v00008086d000010C7sv*sd*bc*sc*i* alias: pci:v00008086d000010C6sv*sd*bc*sc*i* alias: pci:v00008086d000010B6sv*sd*bc*sc*i* depends: mdio,ptp,dca intree: Y vermagic: 3.14.1-1.geafcebd-default SMP mod_unload modversions parm: max_vfs:Maximum number of virtual functions to allocate per physical function - default is zero and maximum value is 63. (Deprecated) (uint) parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint) parm: debug:Debug level (0=none,...,16=all) (int) ...Steve... -----Original Message----- From: target-devel-owner@xxxxxxxxxxxxxxx [mailto:target-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Govindarajan Sent: March 12, 2015 11:09 PM To: target-devel@xxxxxxxxxxxxxxx Subject: Re: LIO iscsi connections issues with ESXi I forgot to mention ubuntu version and linux kernel details. The storage box is running ubuntu server 14.10 with a more recent kernel. Kernel info: Linux storage1 3.19.0-031900-generic #201502091451 SMP Mon Feb 9 14:52:52 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux On Thu, Mar 12, 2015 at 5:05 PM, Govindarajan <govind.rajan@xxxxxxxxx> wrote: > Hi, > I have been using LIO target connected to 4 ESXi hosts for testing > purposes. On the ESXi hosts I have a total of about 30 virtual > machines. Whenever I run heavy IO workload in the VMs for some tests > ESXi seems to flip-flop between marking the iscsi connection online > and offline. This can be seen in the log snippet given at the bottom. > > A brief note about the set up. > > Ubuntu installed on a supermicro box. > 8 x 3TB disks in raid10 /dev/md0 > 4 x 480GB SSD raid10 /dev/md1 > bcache0 created out of /dev/md0 and /dev/md1 (cache) LVMs created on > top of bcache0 > 10 Gbps Intel 82599ES NIC > 10 Gbps on all ESXi hosts for iscsi portgroups switch(dell > powerconnect), esxi, and linux enabled for jumbo frames > > Every time I see the connection flap in ESXi I see a lot of task > aborts being sent by lio target. I have looked around in the mailing > list archives but I am not able understand the root cause of the > problem. If I restart target service, things are back in order until I > run heavy IO again. I have taken care of switch related settings as > per Dell's whitepaper meant for ESXi best practices. > > Today I increased the MaxConnections param to 2, not sure if it would help. > > Any clues are welcome that could help me root cause and solve this > issue. Please do let me know if you need any other log files from > either the linux storage box or the esxi hosts, I'll pull them out for > you. > > Thanks, > Govind > > > > targetcli output -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html ��.n��������+%������w��{.n����j� ����{ay� ʇڙ���f���h������_�(�階�ݢj"��������G����?���&��
