On 6/27/2014 9:53 AM, Nicholas A. Bellinger wrote:
> On Wed, 2014-06-25 at 18:21 +0530, Arshad Hussain wrote:
>> On 6/20/2014 11:17 PM, Nicholas A. Bellinger wrote:
>>> On Fri, 2014-06-20 at 15:54 +0530, Arshad Hussain wrote:
>>>> Hi Nab,
>>>> On 6/20/2014 12:32 PM, Nicholas A. Bellinger wrote:
>>>>> On Thu, 2014-06-19 at 23:43 -0700, Nicholas A. Bellinger wrote:
>>>>>> On Fri, 2014-06-20 at 11:30 +0530, Santosh Kulkarni wrote:
>>> <SNIP>
>>>
>>>>> One other quick thing to try is the following patch to dump the Data-out
>>>>> payload when a ITT received is not associated with a WRITE (following
>>>>> the log), instead of generating a Reject for the existing ITT.
>>>>>
>>>>> I can't quite tell from wireshark output why your test is sending
>>>>> Data-out with an ITT that is associated with a READ that has not been
>>>>> acknowledged by StatSN, but the following should at least leave the READ
>>>>> ITT alone.
>>>>>
>>>>> --nab
>>>>>
>>>>> diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
>>>>> index b87721a..049fe14 100644
>>>>> --- a/drivers/target/iscsi/iscsi_target.c
>>>>> +++ b/drivers/target/iscsi/iscsi_target.c
>>>>> @@ -1308,7 +1308,7 @@ iscsit_check_dataout_hdr(struct iscsi_conn *conn, unsigned char *buf,
>>>>> if (cmd->data_direction != DMA_TO_DEVICE) {
>>>>> pr_err("Command ITT: 0x%08x received DataOUT for a"
>>>>> " NON-WRITE command.\n", cmd->init_task_tag);
>>>>> - return iscsit_reject_cmd(cmd, ISCSI_REASON_PROTOCOL_ERROR, buf);
>>>>> + return iscsit_dump_data_payload(conn, payload_length, 1);
>>>>> }
>>>>> se_cmd = &cmd->se_cmd;
>>>>> iscsit_mod_dataout_timer(cmd);
>>>>>
>>>>>
>>>> Santosh is traveling and I will help out with any answers you need.
>>>>
>>>> The ITT is unique within a session, should READ here be a problem.? And
>>>> what we have observed that our the crash is intermittent as we have
>>>> mention before.
>>>>
>>> That is not what is indicated in the logs so far. The output (appears)
>>> to show that an ITT that was previously used by a READ is being reused
>>> by a WRITE + associated Data-Out, before the original READ is explicitly
>>> acknowledged via StatSN.
>>>
>>> This results in the lookup for the Data-out ITT to locate the original
>>> READ, and correctly fail due to the incorrect data_direction type of the
>>> original command.
>>>
>>>> Tried out your patch on 3.14.0-rc6+ and 3.15.0-rc3+. With 3.15.0-rc3
>>>> the target is not crashing. The dmesg shows the line below.
>>>>
>>>> [root@wfsc iscsi]# dmesg -c
>>>> [ 210.623207] Command ITT: 0xa3a3a3a3 received DataOUT for a NON-WRITE
>>>> command.
>>>>
>>> Thanks for confirming. I think this is the correct fix, but would like
>>> to verify some more on my end. Please send me the test off-list.
>>>
>>>> [root@wfsc iscsi]# uname -a
>>>> Linux wfsc 3.15.0-rc3+ #3 SMP Fri May 30 06:11:42 EDT 2014 x86_64 x86_64
>>>> x86_64 GNU/Linux
>>>>
>>>> With 3.14.0-rc6+. It is crashing with the below output.
>>>>
>>> This output for v3.14-rc6 would indicate that the patch is not applied.
>>> Eg:
>>>
>>> "ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set."
>>>
>>> means that iscsi_dump_data_payload() is not being called, because the
>>> subsequent Data-Out payload (still in the TCP stream) is processed, and
>>> fails at the WRITE + FINAL bits sanity checks.
>>>
>>> Please confirm this patch is applied with a printk() ahead of the call
>>> to iscsi_dump_data_payload().
>> Apologies for the delay.
>>
>> I double check your patch on 3.14.0-rc6+. It is applied and I have
>> added debug statement as you have advised to see if we are hitting the
>> correct area. I have also attached a .png, as I was getting a soft lock.
>> This is on same kernel (3.14.0-rc6+), but without your patch. I will
>> also revisit our test plan and get back.
>>
> Ok, I take the above to mean that the patch addressed the original OOPs
> on both v3.15-rc and v3.14-rc code, correct..?
Yes you are correct.
Thanks
Arshad
>
>> <Dmesg output With your patch on 3.14.0-rc6+>
>> root@wfsc target-pending]# dmesg
>> [ 8915.579866] Debug#1
>> [ 8915.607790] Debug#1
>> [ 8915.607862] Debug#4
>> [ 8915.607923] Command ITT: 0xa3a3a3a3 received DataOUT for a NON-WRITE
>> command.
> Thanks for confirming.
>
> --nab
>
>
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
