On Wed, 2014-06-25 at 18:21 +0530, Arshad Hussain wrote:
> On 6/20/2014 11:17 PM, Nicholas A. Bellinger wrote:
> > On Fri, 2014-06-20 at 15:54 +0530, Arshad Hussain wrote:
> >> Hi Nab,
> >> On 6/20/2014 12:32 PM, Nicholas A. Bellinger wrote:
> >>> On Thu, 2014-06-19 at 23:43 -0700, Nicholas A. Bellinger wrote:
> >>>> On Fri, 2014-06-20 at 11:30 +0530, Santosh Kulkarni wrote:
> > <SNIP>
> >
> >>> One other quick thing to try is the following patch to dump the Data-out
> >>> payload when a ITT received is not associated with a WRITE (following
> >>> the log), instead of generating a Reject for the existing ITT.
> >>>
> >>> I can't quite tell from wireshark output why your test is sending
> >>> Data-out with an ITT that is associated with a READ that has not been
> >>> acknowledged by StatSN, but the following should at least leave the READ
> >>> ITT alone.
> >>>
> >>> --nab
> >>>
> >>> diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
> >>> index b87721a..049fe14 100644
> >>> --- a/drivers/target/iscsi/iscsi_target.c
> >>> +++ b/drivers/target/iscsi/iscsi_target.c
> >>> @@ -1308,7 +1308,7 @@ iscsit_check_dataout_hdr(struct iscsi_conn *conn, unsigned char *buf,
> >>> if (cmd->data_direction != DMA_TO_DEVICE) {
> >>> pr_err("Command ITT: 0x%08x received DataOUT for a"
> >>> " NON-WRITE command.\n", cmd->init_task_tag);
> >>> - return iscsit_reject_cmd(cmd, ISCSI_REASON_PROTOCOL_ERROR, buf);
> >>> + return iscsit_dump_data_payload(conn, payload_length, 1);
> >>> }
> >>> se_cmd = &cmd->se_cmd;
> >>> iscsit_mod_dataout_timer(cmd);
> >>>
> >>>
> >> Santosh is traveling and I will help out with any answers you need.
> >>
> >> The ITT is unique within a session, should READ here be a problem.? And
> >> what we have observed that our the crash is intermittent as we have
> >> mention before.
> >>
> > That is not what is indicated in the logs so far. The output (appears)
> > to show that an ITT that was previously used by a READ is being reused
> > by a WRITE + associated Data-Out, before the original READ is explicitly
> > acknowledged via StatSN.
> >
> > This results in the lookup for the Data-out ITT to locate the original
> > READ, and correctly fail due to the incorrect data_direction type of the
> > original command.
> >
> >> Tried out your patch on 3.14.0-rc6+ and 3.15.0-rc3+. With 3.15.0-rc3
> >> the target is not crashing. The dmesg shows the line below.
> >>
> >> [root@wfsc iscsi]# dmesg -c
> >> [ 210.623207] Command ITT: 0xa3a3a3a3 received DataOUT for a NON-WRITE
> >> command.
> >>
> > Thanks for confirming. I think this is the correct fix, but would like
> > to verify some more on my end. Please send me the test off-list.
> >
> >> [root@wfsc iscsi]# uname -a
> >> Linux wfsc 3.15.0-rc3+ #3 SMP Fri May 30 06:11:42 EDT 2014 x86_64 x86_64
> >> x86_64 GNU/Linux
> >>
> >> With 3.14.0-rc6+. It is crashing with the below output.
> >>
> > This output for v3.14-rc6 would indicate that the patch is not applied.
> > Eg:
> >
> > "ISCSI_FLAG_CMD_WRITE & ISCSI_FLAG_CMD_FINAL not set."
> >
> > means that iscsi_dump_data_payload() is not being called, because the
> > subsequent Data-Out payload (still in the TCP stream) is processed, and
> > fails at the WRITE + FINAL bits sanity checks.
> >
> > Please confirm this patch is applied with a printk() ahead of the call
> > to iscsi_dump_data_payload().
> Apologies for the delay.
>
> I double check your patch on 3.14.0-rc6+. It is applied and I have
> added debug statement as you have advised to see if we are hitting the
> correct area. I have also attached a .png, as I was getting a soft lock.
> This is on same kernel (3.14.0-rc6+), but without your patch. I will
> also revisit our test plan and get back.
>
Ok, I take the above to mean that the patch addressed the original OOPs
on both v3.15-rc and v3.14-rc code, correct..?
> <Dmesg output With your patch on 3.14.0-rc6+>
> root@wfsc target-pending]# dmesg
> [ 8915.579866] Debug#1
> [ 8915.607790] Debug#1
> [ 8915.607862] Debug#4
> [ 8915.607923] Command ITT: 0xa3a3a3a3 received DataOUT for a NON-WRITE
> command.
Thanks for confirming.
--nab
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
