We are trying to test if LIO target sets ResidualCount correctly in case of ResidualOverflow for Data-In by following the below Steps:
1. Send SCSI Inquiry command to iSCSI target, if there are no exceptions in the response (and no residual overflow) we are able to determine how many bytes the target SCSI layer on target side attempted to transfer to target iSCSI layer. On the initiator side when response is received: in the absence of any exceptions or residual overflow we assume all the data was successfully transferred and is equal to iSCSI DataSegmentLength. We will use this value as SPDTL for SCSI Inquiry command.
Note: SPDTL is detailed in RFC 7143 11.4.5.2 (and you may also refer to discussion on T10 mailing list: http://www.t10.org/pipermail/t10/2014-June/017367.html).
2. Send out the same SCSI inquiry command as in Step 1. but set iSCSI EDTL to a value lower than SPDTL.
Assumption: Since we are not making any changes to iSCSI target LU configuration in our setup b/w Step 1. and 2, we assume amount of Inquiry Data target generates will be same of 1. and 2. since inquiry command is the same. During our testing we find this assumption to be true.
Expected Result:
Since in Step 2, SPDTL > EDTL, iSCSI target is expected to set ResidualOverflow flag and ResidualCount to SPDTL - EDTL.
In the attached PCAP text file, frame no. 11 (SCSI Inquiry Req.) and 12 (Inquiry Response) are for Step 1. As explained, SPDTL = 0x24 or 36D.
Frame no. 14 is for (SCSI Inq. Req.) Step 2. EDTL = 0x08.
Hence, expected ResidualCount = 0x1C (28D) i.e. 0x24 (36D) - 0x08 (8D)
Observed Result:
iSCSI target sets the Residual Overflow flag correctly but value of ResidualCount doesn’t match expected value.
Target is setting, ResidualCount to 0x3f8 (1016D) as seen in frame no 16. In the frame no. 14, in the SCSI Inquiry CDB allocation length (SPC 5r01 Section 4.2.5.6) was set to 0x0400 (1024D) and target seems to be using this value to calculate ResidualCount i.e. 0x0400 (1024D) - 0x08.
To further verify this we changed the allocation length value in SCSI Inquiry CDB to 0x0200 (512D) bytes instead, we get ResidualCount of 0x1f8 (504D).
Allocation Length is the maximum value of the SPDTL and not substitute for it, hence it shouldn’t be used to calculate ResidualCount except for cases where SPDTL > Allocation Length and Data is truncated (in that case both Alloc Len and SPDTL are same). (SPC 5r01 Section 4.2.5.6).
LIO Version: Datera Inc. iSCSI Target v4.1.0
Linux Kernel: 4.4.0-22-generic
RFC 7143:
11.4.5.2. Residuals Concepts Overview
"SCSI-Presented Data Transfer Length (SPDTL)" is the term this
document uses (see
Section 2.2
for definition) to represent the
aggregate data length that the target SCSI layer attempts to transfer
using the local iSCSI layer for a task. "Expected Data Transfer
Length (EDTL)" is the iSCSI term that represents the length of data
that the iSCSI layer expects to transfer for a task. EDTL is
specified in the SCSI Command PDU.
When SPDTL = EDTL for a task, the target iSCSI layer completes the
task with no residuals. Whenever SPDTL differs from EDTL for a task,
that task is said to have a residual.
If SPDTL > EDTL for a task, iSCSI Overflow MUST be signaled in the
SCSI Response PDU as specified in
Section 11.4.5.1
. The Residual
Count MUST be set to the numerical value of (SPDTL - EDTL).
If SPDTL < EDTL for a task, iSCSI Underflow MUST be signaled in the
SCSI Response PDU as specified in
Section 11.4.5.1
. The Residual
Count MUST be set to the numerical value of (EDTL - SPDTL).
Note that the Overflow and Underflow scenarios are independent of
Data-In and Data-Out. Either scenario is logically possible in
SPC 5r01
4.2.5.6 Allocation length
The ALLOCATION LENGTH field specifies the maximum number of bytes or blocks that an application client has allocated in the Data-In Buffer. The ALLOCATION LENGTH field specifies bytes unless a different requirement is stated in the command definition.
An allocation length of zero specifies that no data shall be transferred. This condition shall not be considered an error.
The device server shall terminate transfers to the Data-In Buffer when the number of bytes or blocks specified by the ALLOCATION LENGTH field have been transferred or when all available data have been transferred, whichever is less. The allocation length is used to limit the maximum amount of variable length data (e.g., mode data, log data, diagnostic data) returned to an application client. If the information being transferred to the Data-In Buffer includes fields containing counts of the number of bytes in some or all of the data (e.g., a PARAMETER DATA LENGTH field, a PAGE LENGTH field, a DESCRIPTOR LENGTH field, an AVAILABLE DATA field), then the contents of these fields shall not be altered to reflect the truncation, if any, that results from an insufficient ALLOCATION LENGTH value, unless the standard that describes the Data-In Buffer format states otherwise.
If the amount of information that is available to be transferred exceeds the maximum value that the ALLOCATION LENGTH field in combination with other fields in the CDB is capable of specifying, then no data shall be transferred and the command shall be terminated with CHECK CONDITION status, with the sense key set to ILLEGAL REQUEST, and the additional sense code set to INVALID FIELD IN CDB.
Regards,
Sumit Rai
Principal Development Engineer
Calsoft Pvt. Ltd., Pune, India
4 0.001086 172.16.62.150 172.16.62.161 iSCSI 276 Login Command
Frame 4: 276 bytes on wire (2208 bits), 276 bytes captured (2208 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.62.150, Dst: 172.16.62.161
Transmission Control Protocol, Src Port: 34107 (34107), Dst Port: 3260 (3260), Seq: 1, Ack: 1, Len: 208
iSCSI (Login Command)
Opcode: Login Command (0x03)
1... .... = T: Transit to next login stage
.0.. .... = C: Text is complete
.... 00.. = CSG: Security negotiation (0x00)
.... ..01 = NSG: Operational negotiation (0x01)
VersionMax: 0x00
VersionMin: 0x00
TotalAHSLength: 0x00
DataSegmentLength: 157 (0x0000009d)
ISID: 00230d030000
TSIH: 0x0000
InitiatorTaskTag: 0xa3a3a3a3
CID: 0x0000
CmdSN: 0x00000000
ExpStatSN: 0x00000000
Key/Value Pairs
KeyValue: InitiatorName=iqn.2016-01.org.calsoft:nochap
KeyValue: TargetName=iqn.2003-01.org.linux-iscsi.srai-virtualbox.x8664:sn.619fe26c7e52
KeyValue: SessionType=Normal
KeyValue: AuthMethod=None
Padding: 000000
6 0.002616 172.16.62.161 172.16.62.150 iSCSI 180 Login Response (Success)
Frame 6: 180 bytes on wire (1440 bits), 180 bytes captured (1440 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.62.161, Dst: 172.16.62.150
Transmission Control Protocol, Src Port: 3260 (3260), Dst Port: 34107 (34107), Seq: 1, Ack: 209, Len: 112
iSCSI (Login Response)
Opcode: Login Response (0x23)
1... .... = T: Transit to next login stage
.0.. .... = C: Text is complete
.... 00.. = CSG: Security negotiation (0x00)
.... ..01 = NSG: Operational negotiation (0x01)
VersionMax: 0x00
VersionActive: 0x00
TotalAHSLength: 0x00
DataSegmentLength: 62 (0x0000003e)
ISID: 00230d030000
TSIH: 0x0000
InitiatorTaskTag: 0xa3a3a3a3
StatSN: 0x79b89678
ExpCmdSN: 0x00000000
MaxCmdSN: 0x00000000
Status: Success (0x0000)
Key/Value Pairs
KeyValue: AuthMethod=None
KeyValue: TargetAlias=LIO Target
KeyValue: TargetPortalGroupTag=1
Padding: 0000
8 0.003927 172.16.62.150 172.16.62.161 iSCSI 380 Login Command
Frame 8: 380 bytes on wire (3040 bits), 380 bytes captured (3040 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.62.150, Dst: 172.16.62.161
Transmission Control Protocol, Src Port: 34107 (34107), Dst Port: 3260 (3260), Seq: 209, Ack: 113, Len: 312
iSCSI (Login Command)
Opcode: Login Command (0x03)
1... .... = T: Transit to next login stage
.0.. .... = C: Text is complete
.... 01.. = CSG: Operational negotiation (0x01)
.... ..11 = NSG: Full feature phase (0x03)
VersionMax: 0x00
VersionMin: 0x00
TotalAHSLength: 0x00
DataSegmentLength: 263 (0x00000107)
ISID: 00230d030000
TSIH: 0x0000
InitiatorTaskTag: 0xa3a3a3a3
CID: 0x0000
CmdSN: 0x00000000
ExpStatSN: 0x79b89679
Key/Value Pairs
KeyValue: MaxBurstLength=1024
KeyValue: MaxRecvDataSegmentLength=1024
KeyValue: ErrorRecoveryLevel=0
KeyValue: ImmediateData=Yes
KeyValue: MaxOutstandingR2T=1
KeyValue: InitialR2T=Yes
KeyValue: DataDigest=None
KeyValue: HeaderDigest=None
KeyValue: FirstBurstLength=512
KeyValue: DataSequenceInOrder=Yes
KeyValue: DataPDUInOrder=Yes
KeyValue: DefaultTime2Retain=20
KeyValue: DefaultTime2Wait=2
Padding: 00
9 0.004118 172.16.62.161 172.16.62.150 iSCSI 340 Login Response (Success)
Frame 9: 340 bytes on wire (2720 bits), 340 bytes captured (2720 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.62.161, Dst: 172.16.62.150
Transmission Control Protocol, Src Port: 3260 (3260), Dst Port: 34107 (34107), Seq: 113, Ack: 521, Len: 272
iSCSI (Login Response)
Opcode: Login Response (0x23)
1... .... = T: Transit to next login stage
.0.. .... = C: Text is complete
.... 01.. = CSG: Operational negotiation (0x01)
.... ..11 = NSG: Full feature phase (0x03)
VersionMax: 0x00
VersionActive: 0x00
TotalAHSLength: 0x00
DataSegmentLength: 224 (0x000000e0)
ISID: 00230d030000
TSIH: 0x0011
InitiatorTaskTag: 0xa3a3a3a3
StatSN: 0x79b89679
ExpCmdSN: 0x00000000
MaxCmdSN: 0x00000000
Status: Success (0x0000)
Key/Value Pairs
KeyValue: HeaderDigest=None
KeyValue: DataDigest=None
KeyValue: MaxConnections=1
KeyValue: ImmediateData=Yes
KeyValue: MaxRecvDataSegmentLength=262144
KeyValue: MaxBurstLength=1024
KeyValue: FirstBurstLength=512
KeyValue: DefaultTime2Wait=2
KeyValue: DefaultTime2Retain=20
KeyValue: MaxOutstandingR2T=1
KeyValue: ErrorRecoveryLevel=0
10 0.005526 172.16.62.150 172.16.62.161 iSCSI 116 SCSI: Inquiry LUN: 0x05
Frame 10: 116 bytes on wire (928 bits), 116 bytes captured (928 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.62.150, Dst: 172.16.62.161
Transmission Control Protocol, Src Port: 34107 (34107), Dst Port: 3260 (3260), Seq: 521, Ack: 385, Len: 48
iSCSI (SCSI Command)
Opcode: SCSI Command (0x01)
.0.. .... = I: Queued delivery
TotalAHSLength: 0x00
DataSegmentLength: 0 (0x00000000)
LUNs
LUN 0
InitiatorTaskTag: 0xa3a3a3a3
ExpectedDataTransferLength: 0x00000800
CmdSN: 0x00000000
ExpStatSN: 0x79b8967a
Data In in: 12
Response in: 12
Flags: 0xc1, F, R, Attr: Simple
1... .... = F: Final PDU in sequence
.1.. .... = R: Data will be read from target
..0. .... = W: No data will be written to target
.... .001 = Attr: Simple (0x01)
SCSI CDB Inquiry
[LUN: 0x0005]
[Command Set:Direct Access Device (0x00) ]
[Response in: 12]
Opcode: Inquiry (0x12)
CMDT = 0, EVPD = 0
Allocation Length: 1024
Control: 0x00
12 0.005787 172.16.62.161 172.16.62.150 iSCSI 104 SCSI: Data In LUN: 0x05 (Inquiry Response Data) SCSI: Response LUN: 0x05 (Inquiry) (Good)
Frame 12: 104 bytes on wire (832 bits), 104 bytes captured (832 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.62.161, Dst: 172.16.62.150
Transmission Control Protocol, Src Port: 3260 (3260), Dst Port: 34107 (34107), Seq: 433, Ack: 569, Len: 36
[2 Reassembled TCP Segments (84 bytes): #11(48), #12(36)]
iSCSI (SCSI Data In)
Opcode: SCSI Data In (0x25)
Flags: 0x83
1... .... = F: Final PDU in sequence
.0.. .... = A: Acknowledge not requested
.... .0.. = O: No residual overflow occurred
.... ..1. = U: Residual underflow occurred
.... ...1 = S: Response contains SCSI status
Status: Good (0x00)
TotalAHSLength: 0x00
DataSegmentLength: 36 (0x00000024)
InitiatorTaskTag: 0xa3a3a3a3
StatSN: 0x79b8967a
ExpCmdSN: 0x00000001
MaxCmdSN: 0x00000040
DataSN: 0x00000000
BufferOffset: 0x00000000
ResidualCount: 0x000007dc
Request in: 10
Time from request: 0.000261000 seconds
SCSI Payload (Inquiry Response Data)
[LUN: 0x0005]
[Command Set:Direct Access Device (0x00) ]
[SBC Opcode: Inquiry (0x12)]
[Request in: 10]
[Response in: 12]
Peripheral: 0x00, Qualifier: Device type is connected to logical unit, Device Type: Direct Access Device
Inquiry RMB Flags: 0x00
Version: Compliance to SPC-3 (0x05)
Inquiry ACA Flags: 0x02, Response Data Format: SPC-2/SPC-3/SPC-4
Additional Length: 31
Inquiry SCCS Flags: 0xb8, SCCS, TPGS: Both Implicit and Explicit LU Access supported, 3PC
Inquiry BQue Flags: 0x00
Inquiry RelAdr Flags: 0x02, CmdQue
Vendor Id: LIO-ORG
Product Id: FILEIO
Product Revision Level: 4.0
SCSI Response (Inquiry)
[LUN: 0x0005]
[Command Set:Direct Access Device (0x00) ]
[SBC Opcode: Inquiry (0x12)]
[Request in: 10]
[Time from request: 0.000261000 seconds]
[Status: Good (0x00)]
14 0.006861 172.16.62.150 172.16.62.161 iSCSI 116 SCSI: Inquiry LUN: 0x05
Frame 14: 116 bytes on wire (928 bits), 116 bytes captured (928 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.62.150, Dst: 172.16.62.161
Transmission Control Protocol, Src Port: 34107 (34107), Dst Port: 3260 (3260), Seq: 569, Ack: 469, Len: 48
iSCSI (SCSI Command)
Opcode: SCSI Command (0x01)
.0.. .... = I: Queued delivery
TotalAHSLength: 0x00
DataSegmentLength: 0 (0x00000000)
LUNs
LUN 0
InitiatorTaskTag: 0xa3a3a3a3
ExpectedDataTransferLength: 0x00000008
CmdSN: 0x00000001
ExpStatSN: 0x79b8967b
Data In in: 16
Response in: 16
Flags: 0xc1, F, R, Attr: Simple
1... .... = F: Final PDU in sequence
.1.. .... = R: Data will be read from target
..0. .... = W: No data will be written to target
.... .001 = Attr: Simple (0x01)
SCSI CDB Inquiry
[LUN: 0x0005]
[Command Set:Direct Access Device (0x00) ]
[Response in: 16]
Opcode: Inquiry (0x12)
CMDT = 0, EVPD = 0
Allocation Length: 1024
Control: 0x00
16 0.007032 172.16.62.161 172.16.62.150 iSCSI 76 SCSI: Data In LUN: 0x05 (Inquiry Response Data) SCSI: Response LUN: 0x05 (Inquiry) (Good)
Frame 16: 76 bytes on wire (608 bits), 76 bytes captured (608 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.62.161, Dst: 172.16.62.150
Transmission Control Protocol, Src Port: 3260 (3260), Dst Port: 34107 (34107), Seq: 517, Ack: 617, Len: 8
[2 Reassembled TCP Segments (56 bytes): #15(48), #16(8)]
iSCSI (SCSI Data In)
Opcode: SCSI Data In (0x25)
Flags: 0x85
1... .... = F: Final PDU in sequence
.0.. .... = A: Acknowledge not requested
.... .1.. = O: Residual overflow occurred
.... ..0. = U: No residual underflow occurred
.... ...1 = S: Response contains SCSI status
Status: Good (0x00)
TotalAHSLength: 0x00
DataSegmentLength: 8 (0x00000008)
InitiatorTaskTag: 0xa3a3a3a3
StatSN: 0x79b8967b
ExpCmdSN: 0x00000002
MaxCmdSN: 0x00000041
DataSN: 0x00000000
BufferOffset: 0x00000000
ResidualCount: 0x000003f8
Request in: 14
Time from request: 0.000171000 seconds
SCSI Payload (Inquiry Response Data)
[LUN: 0x0005]
[Command Set:Direct Access Device (0x00) ]
[SBC Opcode: Inquiry (0x12)]
[Request in: 14]
[Response in: 16]
Peripheral: 0x00, Qualifier: Device type is connected to logical unit, Device Type: Direct Access Device
Inquiry RMB Flags: 0x00
Version: Compliance to SPC-3 (0x05)
Inquiry ACA Flags: 0x02, Response Data Format: SPC-2/SPC-3/SPC-4
Additional Length: 31
Inquiry SCCS Flags: 0xb8, SCCS, TPGS: Both Implicit and Explicit LU Access supported, 3PC
Inquiry BQue Flags: 0x00
Inquiry RelAdr Flags: 0x02, CmdQue
SCSI Response (Inquiry)
[LUN: 0x0005]
[Command Set:Direct Access Device (0x00) ]
[SBC Opcode: Inquiry (0x12)]
[Request in: 14]
[Time from request: 0.000171000 seconds]
[Status: Good (0x00)]
18 0.007787 172.16.62.150 172.16.62.161 iSCSI 116 Logout Command (Close session)
Frame 18: 116 bytes on wire (928 bits), 116 bytes captured (928 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.62.150, Dst: 172.16.62.161
Transmission Control Protocol, Src Port: 34107 (34107), Dst Port: 3260 (3260), Seq: 617, Ack: 525, Len: 48
iSCSI (Logout Command)
Opcode: Logout Command (0x06)
.1.. .... = I: Immediate delivery
.000 0000 = Reason: Close session (0x00)
TotalAHSLength: 0x00
DataSegmentLength: 0 (0x00000000)
InitiatorTaskTag: 0x05000000
CID: 0x0000
CmdSN: 0x00000000
ExpStatSN: 0x00000000
19 0.007824 172.16.62.161 172.16.62.150 iSCSI 116 Logout Response
Frame 19: 116 bytes on wire (928 bits), 116 bytes captured (928 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 172.16.62.161, Dst: 172.16.62.150
Transmission Control Protocol, Src Port: 3260 (3260), Dst Port: 34107 (34107), Seq: 525, Ack: 665, Len: 48
iSCSI (Logout Response)
Opcode: Logout Response (0x26)
Response: Connection closed successfully (0x00)
TotalAHSLength: 0x00
DataSegmentLength: 0 (0x00000000)
InitiatorTaskTag: 0x05000000
StatSN: 0x79b8967c
ExpCmdSN: 0x00000002
MaxCmdSN: 0x00000041
Time2Wait: 0x0000
Time2Retain: 0x0000
