Hi Moussa & Co, On Mon, 2013-11-11 at 19:05 +0000, Moussa Ba (moussaba) wrote: > My system setup is as follows: > > Target: CentOS 6.4 - LIO Target running 3.12 kernel, 8 PCIe SSD in one volume group, 8 logical volumes, 8 targets, 1 LUN/target. > Initiator: CentOS 6.4 running 3.11 Kernel (Also ran 2.6.32-358), ISER based initiator over Mellanox 40Gbit ConnectX HCA > > When running performance tests on the initiator, I am running into fio timeouts that lead to ABORT_TASK commands on the target. In other words, fio fails to "reap" all the io threads almost as if it is waiting for lost IOs to complete. This is happening on random write IO operations. Some context, we are generating about 576KIOPS 4KB block sizes using 8 LUNS. The PCIe SSD have a write buffer that can absorb writes with an end to end latency on the initiator of 44us. We are not currently seeing any errors on read IOs, which tend to have 2X+ the latency of writes. > > See below for the dmesg on the target side. > Timeout Condition occurs at 154 which is the Protocol Error after fio is interrupted or runs to completion. > [ 154.453663] Received CmdSN: 0x000fcbb7 is greater than MaxCmdSN: 0x000fcbb6, protocol error. > [ 154.453673] Received CmdSN: 0x000fcbb8 is greater than MaxCmdSN: 0x000fcbb6, protocol error. > (CC'ing Mike) As mentioned off-list, this would tend to indicate some manner of open-iscsi bug, as it's never legal for an initiator to send a CmdSN greater than the MaxCmdSN that's currently being sent in target outgoing response PDUs. Mike, any idea as to how this might be happening on the initiator side..? --nab > AFTER a about 90 seconds the abort task commands are received > > [ 216.239598] Unable to locate ITT: 0x00000074 on CID: 0 > [ 216.239601] Unable to locate RefTaskTag: 0x00000074 on CID: 0. > [ 216.239609] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 0 > [ 216.239630] Unable to locate ITT: 0x00000018 on CID: 0 > [ 216.239631] Unable to locate RefTaskTag: 0x00000018 on CID: 0. > [ 216.239635] ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 0 > [ 216.239654] ABORT_TASK: Found referenced iSCSI task_tag: 124 > [ 216.239657] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 124 > [ 216.239677] ABORT_TASK: Found referenced iSCSI task_tag: 69 > [ 216.239680] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 69 > > > These errors are intermittent but reproducible. > > Moussa. > > -- > To unsubscribe from this list: send the line "unsubscribe target-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
