Hi Arshad,
On Tue, 2014-04-29 at 18:33 +0530, Arshad Hussain wrote:
> Hi Nab,
>
> I see a bug when running on the iSCSI target ( Datera Inc. iSCSI Target
> v4.1.0).
> At this moment I do not know what part of our code is causing it. I am
> looking
> into it. Please Let me know if you need more info.
>
> Thanks,
> Arshad
>
> [root@wfsc test_suite_iscsi]# uname -a
> Linux wfsc 3.14.0-rc6+ #2 SMP Tue Apr 29 04:55:14 EDT 2014 x86_64 x86_64
> x86_64 GNU/Linux
> [root@wfsc test_suite_iscsi]#
>
>
> Apr 29 08:33:48 wfsc kernel: [ 463.353742] ------------[ cut here
> ]------------
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] kernel BUG at
> drivers/target/iscsi/iscsi_target.c:1567!
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] invalid opcode: 0000 [#1] SMP
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] Modules linked in:
> target_core_pscsi target_core_file target_core_iblock iscsi_target_mod
>
> target_core_mod xt_CHECKSUM nf_conntrack_netbios_ns
> nf_conntrack_broadcast bnep bluetooth ip6t_REJECT 6lowpan_iphc rfkill
> nf_conntrack_ipv6
>
> nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack
> nf_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter
> ebtables
>
> ip6table_mangle ip6table_security ip6table_raw ip6table_filter
> ip6_tables iptable_mangle iptable_security iptable_raw be2iscsi
>
> iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi
> ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp
>
> libiscsi_tcp libiscsi scsi_transport_iscsi ppdev joydev snd_intel8x0
> snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm microcode e1000
>
> pcspkr snd_timer snd i2c_piix4 i2c_core serio_raw soundcore parport_pc
> parport binfmt_misc ata_generic pata_acpi uinput
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] CPU: 0 PID: 1637 Comm:
> iscsi_trx Not tainted 3.14.0-rc6+ #2
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] Hardware name: innotek GmbH
> VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] task: ffff88002914c950 ti:
> ffff88003bb82000 task.ti: ffff88003bb82000
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] RIP:
> 0010:[<ffffffffa042b2b3>] [<ffffffffa042b2b3>]
> iscsit_process_nop_out+0x123/0x130
>
> [iscsi_target_mod]
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] RSP: 0018:ffff88003bb83d78
> EFLAGS: 00010246
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] RAX: 0000000000000000 RBX:
> ffff88003c7c8000 RCX: 0000000030006c65
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] RDX: ffff88003bb83e58 RSI:
> 0000000000000000 RDI: ffff88003c7c8000
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] RBP: ffff88003bb83d98 R08:
> 00000000d51d0389 R09: 0000000000000030
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] R10: ffff880025074e00 R11:
> 0000000000000030 R12: ffff88003bb83e58
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] R13: 0000000000000000 R14:
> 0000000000000000 R15: 0000000000000000
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] FS: 0000000000000000(0000)
> GS:ffff88003fc00000(0000) knlGS:0000000000000000
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] CS: 0010 DS: 0000 ES: 0000
> CR0: 000000008005003b
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] CR2: 00007f733083af98 CR3:
> 000000003cd7d000 CR4: 00000000000006f0
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] Stack:
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] ffff88003c7c8000
> ffff88003bb83e58 0000000000000006 0000000000000000
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] ffff88003bb83eb8
> ffffffffa042ebc1 ffff88003bb83dc8 ffffffff810a40bf
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] ffff88003cce3148
> ffff880000000200 ffff880000000000 ffff8800290cf8a0
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] Call Trace:
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] [<ffffffffa042ebc1>]
> iscsi_target_rx_thread+0x4d1/0xf00 [iscsi_target_mod]
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] [<ffffffff810a40bf>] ?
> __dequeue_entity+0x2f/0x50
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] [<ffffffff8169130e>] ?
> __schedule+0x38e/0x7a0
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] [<ffffffffa042e6f0>] ?
> iscsi_target_tx_thread+0x210/0x210 [iscsi_target_mod]
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] [<ffffffff8108fe79>]
> kthread+0xc9/0xe0
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] [<ffffffff8108fdb0>] ?
> flush_kthread_worker+0x80/0x80
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] [<ffffffff8169d3ec>]
> ret_from_fork+0x7c/0xb0
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] [<ffffffff8108fdb0>] ?
> flush_kthread_worker+0x80/0x80
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] Code: 89 e7 41 c7 44 24 0c
> 29 00 00 00 ba 29 00 00 00 48 89 de e8 f0 cf ff ff 48 89 df e8 48 e2
>
> ff ff 31 c0 eb 8c b8 ea ff ff ff eb 85 <0f> 0b 66 66 2e 0f 1f 84 00 00
> 00 00 00 66 66 66 66 90 55 48 89
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] RIP [<ffffffffa042b2b3>]
> iscsit_process_nop_out+0x123/0x130 [iscsi_target_mod]
> Apr 29 08:33:48 wfsc kernel: [ 463.354091] RSP <ffff88003bb83d78>
> Apr 29 08:33:48 wfsc kernel: [ 463.420698] ---[ end trace
> e708346c72cf3103 ]---
So this BUG_ON is triggering due to a flood of zeros in the TCP
data-stream, that results in the block in iscsit_process_nop_out() to be
reached without a valid iscsi_cmd pointer.
Below is the patch that is being applied to target-pending/master, along
with a CC to v3.11.y stable to address the regression that was
originally introduced when support was added for handling NOPs PDUs over
iser.
Thanks for reporting!
--nab
commit 1868c477c0d408be000db0c35e41ec6f3fadea06
Author: Nicholas Bellinger <nab@xxxxxxxxxxxxxxx>
Date: Thu May 1 13:44:56 2014 -0700
iscsi-target: Change BUG_ON to REJECT in iscsit_process_nop_out
This patch changes an incorrect use of BUG_ON to instead generate a
REJECT + PROTOCOL_ERROR in iscsit_process_nop_out() code. This case
can occur with traditional TCP where a flood of zeros in the data
stream can reach this block for what is presumed to be a NOP-OUT with
a solicited reply, but without a valid iscsi_cmd pointer.
This incorrect BUG_ON was introduced during the v3.11-rc timeframe
with the following commit:
commit 778de368964c5b7e8100cde9f549992d521e9c89
Author: Nicholas Bellinger <nab@xxxxxxxxxxxxxxx>
Date: Fri Jun 14 16:07:47 2013 -0700
iscsi/isert-target: Refactor ISCSI_OP_NOOP RX handling
Reported-by: Arshad Hussain <arshad.hussain@xxxxxxxxxxxxxx>
Cc: stable@xxxxxxxxxxxxxxx # 3.11+
Signed-off-by: Nicholas Bellinger <nab@xxxxxxxxxxxxxxx>
diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index 78cab13..46588c8 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -1593,7 +1593,9 @@ int iscsit_process_nop_out(struct iscsi_conn *conn, struct iscsi_cmd *c
* Initiator is expecting a NopIN ping reply..
*/
if (hdr->itt != RESERVED_ITT) {
- BUG_ON(!cmd);
+ if (!cmd)
+ return iscsit_add_reject(conn, ISCSI_REASON_PROTOCOL_ERROR,
+ (unsigned char *)hdr);
spin_lock_bh(&conn->cmd_lock);
list_add_tail(&cmd->i_conn_node, &conn->conn_cmd_list);
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
