Hi Nicholas, I think you may be getting lost in the packet sequence. I using Wireshark filter iscsi.initiatortasktag ==0x00170000 Here's a summary: packet 69751 - Write command + immediate data packet 69755 - R2T from the target packet 69762 - DataOut PDU => PDU with mis-detected Data Digest error packet 69768 - Next DataOut PDU packet 69771 - Next DataOut PDU If you switch to filter tcp.port==3260.... packets 69722-69779, 69844 - Target TCP ACKs previously sent data packet 69844 - Target resets the TCP connection (Digest error detected fail path) Packet 69762 matches the first detected digest error in /var/log/messages: ITT: 0x00170000, Offset: 8048, Length: 8192, DataSN: 0x00000000, CRC32C DataDigest 0xa8517e77 does not match computed 0x4e289762 Offset 8048 = 0x1f70 Length 8192 = 0x2000 DataDigest 0xa8517e77 =~ 0x77 0x7e 0x51 0xa8 (From Wireshark trace and reverse order presentation) The initial R2T is definitely in the trace. Either something is wrong in the CRC32C calculation code or the data has changed by the time the calculation was performed. I see little difference between a packet with offset 0x1f70 that fails and those with offset 0x1f60 that digest works on. Puzzling! Thanks, -Joe ----- Original Message ----- From: Nicholas A. Bellinger <nab@xxxxxxxxxxxxxxx> To: jrepac@xxxxxxxxx Cc: target-devel <target-devel@xxxxxxxxxxxxxxx> Sent: Thursday, February 16, 2012 12:54 AM Subject: Re: Data Digest Errors On Wed, 2012-02-15 at 19:01 -0800, jrepac@xxxxxxxxx wrote: > Hi Nicholas, > > I don't think your interpretation of RFC-3720, section 10.3.4 is > correct :). The key words are "OR ONLY the immediate data, if any". > Immediate data only needs to be <= the negotiated FirstBurstLength. > This is from section 3.2.4.2: > > An initiator may send unsolicited data up to FirstBurstLength as > immediate (up to the negotiated maximum PDU length), in a separate > PDU sequence or both. All subsequent data MUST be solicited. The > maximum length of an individual data PDU or the immediate-part of > the first unsolicited burst MAY be negotiated at login. > Mmmmm, it's still preferred for iSCSI Initiators to round to the nearest sector for data SG I/O transfers following what's required for FirstBurstLength + MaxRecvDataSegmentLength keys (multiple of 512 bytes) during iSCSI login negotiation. > The handling of immediate data may be mute here since the failure is > occurring on a subsequent DataOut PDU in the transfer. I believe this > is being handled in iscsi_target.c:iscsit_handle_data_out. Ah yes, sorry. The solicited Data OUT PDU is what is failing CRC32C data digest, and not the unsolicited immediate data payload. So as you've pointed out, the starting Packet 69751: Immediate data length 8048 bytes (0x1f70), EDTL = 61440 > bytes (0xf000) is the leading packet here. It appears that Packet 69768 containing the ITT 0x00170000 solicited Data OUT PDU in question is coming across the wire with an Offset: 0x3f70 (instead of 0x1f70 starting from immediate data) as the first solicited data-out sequence for ITT 0x00170000 starting from Packet 69751. I think this should actually be failing on the target side because DataPDUInOrder=Yes has been negotiated, and we are expecting Packet 69768 Data OUT to contain the next starting Offset: 0x1f70 > If you check the trace again that I sent you, the failure is on the > first PDU after the R2T is sent. I re-checked Wireshark CRC32C > validation with a script. It's correct. The target is locating the > correct location and value for the digest since it is reported > correctly in the /var/lo/messages file. > > I glanced through the CRC32C calculation portion of the code and did > not see any systematic problems. I am running this test on a 10 GbE > network and it make me > wonder if we have a race going. Two DataOut PDU's make it into the > target following the flagged PDU before the sends stop. What does > iscsit_unmap_iovec(cmd) do > prior to the CRC32C calculation? I really don't know the data path on > this code base. > AFAICT there are no R2T being generated for ITT 0x0017000 in the packet catpure I am missing something..? The Packet # containing r2t_sn=0 for ITT 0x00170000 would be helpful to know in order to double check the R2T offset be generated by iscsi-target is correct against what's been received as unsolicited immediate data data length (0x1f70). InitialR2T=Yes is being negotiated, so we are expecting to see target generated R2Ts on the wire, yes..? --nab -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
