Hi Nicholas, I don't think your interpretation of RFC-3720, section 10.3.4 is correct :). The key words are "OR ONLY the immediate data, if any". Immediate data only needs to be <= the negotiated FirstBurstLength. This is from section 3.2.4.2: An initiator may send unsolicited data up to FirstBurstLength as immediate (up to the negotiated maximum PDU length), in a separate PDU sequence or both. All subsequent data MUST be solicited. The maximum length of an individual data PDU or the immediate-part of the first unsolicited burst MAY be negotiated at login. The handling of immediate data may be mute here since the failure is occurring on a subsequent DataOut PDU in the transfer. I believe this is being handled in iscsi_target.c:iscsit_handle_data_out. If you check the trace again that I sent you, the failure is on the first PDU after the R2T is sent. I re-checked Wireshark CRC32C validation with a script. It's correct. The target is locating the correct location and value for the digest since it is reported correctly in the /var/lo/messages file. I glanced through the CRC32C calculation portion of the code and did not see any systematic problems. I am running this test on a 10 GbE network and it make me wonder if we have a race going. Two DataOut PDU's make it into the target following the flagged PDU before the sends stop. What does iscsit_unmap_iovec(cmd) do prior to the CRC32C calculation? I really don't know the data path on this code base. Thanks, -Joe ----- Original Message ----- From: Nicholas A. Bellinger <nab@xxxxxxxxxxxxxxx> To: jrepac@xxxxxxxxx Cc: target-devel <target-devel@xxxxxxxxxxxxxxx> Sent: Wednesday, February 15, 2012 3:01 PM Subject: Re: Data Digest Errors On Tue, 2012-02-14 at 08:48 -0800, jrepac@xxxxxxxxx wrote: > Hi Nicholas, > I sent the trace and /var/log/messages off list. This trace is > executed with only one command outstanding. Here's what I see from > the data: > > 1. Error occurs on packet 69762 for a write (packet 69751) with > Immediate Data = 0x1f70 and Expected Data Transfer Length = 0xf000. > 2. Many prior write transactions(782)complete fine with Immediate Data > = 0x1f60 and Expected Data transfer Length = 0xf000. > 3. Packet 69751 is the first write using Immediate Data = 0x1f70. > Ok, looking at the immediate data WRITEs in the packet capture, I still think the DataSegmentLength values for immediate data payloads are incorrect. Namely the fact that the DataSegmentLength for the immediate data payload (in some cases) is *less* than what FirstBurstLength is negotiated during iSCSI login (see below). It's my understanding for Immediate data the Initiator MUST be sending the write buffer up to min(FirstBurstLength, MaxRecvDataSegmentLength) bytes, or if the Expected Data Transfer Length is less than that either, send the entire WRITE buffer as immediate data payload. I'm getting this MUST from RFC-3720, section 10.3.4. Expected Data Transfer Length: If the Expected Data Transfer Length is higher than the FirstBurstLength (the negotiated maximum amount of unsolicited data the target will accept), the initiator MUST send the maximum amount of unsolicited data OR ONLY the immediate data, if any. So following the packet capture and your observations above, the MSFT Emulex offload initiator is generating the following packets (some have been left out for simplicity): Packet 68786: Immediate data length 8032 bytes (0x1f60), EDTL = 61440 bytes (0xf000) -> target DataDigest PASS Packet 69016: Immediate data length 4096 bytes (0x1000), EDTL = 4096 bytes (0x1000) -> target DataDigest Pass Packet 69056: Immediate data length 8032 bytes (0x1f60), EDTL = 61440 bytes (0xf000) -> target DataDigest PASS Packet 69237: Immediate data length 4096 bytes (0x1000), EDTL = 4096 bytes (0x1000) -> target DataDigest PASS Packet 69277: Immediate data length 8032 bytes (0x1f60), EDTL = 61440 bytes (0xf000) -> target DataDigest PASS Packet 69350: Immediate data length 8032 bytes (0x1f60), EDTL = 49152 bytes (0xc000) -> target DataDigest PASS Packet 69439: Immediate data length 4096 bytes (0x1000), EDTL = 4096 bytes (0x1000) -> target DataDigest PASS Packet 69751: Immediate data length 8048 bytes (0x1f70), EDTL = 61440 bytes (0xf000) -> target DataDigest FAIL So taking a quick look at iscsi_target.c:iscsit_handle_immediate_data(), I don't see a reason why 8048 bytes of immediate data payload is failing the data digest checks, while 8032 byte payloads would be passing here.. > I will start varying the InitialR2T, ImmediateData, FirstBurstLength, > MaxBurstLength and MaxRecvDataSegmentLength keys. > > So looking at the Login parameter negotiation, I see the following key values being set: ImmediateData=Yes InitialR2T=Yes FirstBurstLength=8192 MaxBurstLength=262144 MaxRecvDataSegmentLength=65536 So it might be worthwhile to try increasing FirstBurstLength to 65536 to see if that makes the initiator honor the proper Immediate Data lengths (eg: up to FirstBurstLength). Otherwise disabling immediate data operation with ImmedidateData=No seems like it would likely work-around the problem. I would still recommend to try to fix this sub FirstBurstLength immediate data lengths on the initiator side, as according to my interpretation this is not is following rfc-3720, and I dont recall see any other initiators (in practice) doing this.. > No digest testing was performed with the initiator in a Linux system > though I did do long duration login/logout tests there with 512 > targets. I moved to Windows to do Iometer testing and did not see any > issues with no digest. I could not even initialize the drive with > both digests enabled. This is what I am using as a test case. > Another thing I need to try is doing a data integrity test with digest > disabled under Windows. > Ok, I was just curious if this was something you could reproduce with offloaded enabled on Linux clients as well. I also notice you're using a unh-iol TargetName for your testing. Have you tested with the old unh-target, and does something similar occur with ImmediateData data digest payload failures too..? Thanks Joe! --nab -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
