I noted something interesting with the digest error problem noted over the past two months with the Emulex initiator and the target. It seems that problem occurs just after the target sends out a sequence of multiple tightly spaced R2Ts for outstanding write commands. The problem occurs while simply trying to initialize or format the drive with the MS iSCSI initiator (using the Emulex adapter instance). Errors are reported in /var/log/messages as data digest failures but the wireshark trace shows the digest is actually good. The MS drive initialization completes fine when both data and header digest are disabled.
This problem only seems to occur when a group of R2T's are sent out with <20us between PDUs. I did four trace captures on this last night and the behavior seems to be consistent. The magic number so far is three R2Ts. Nicholas fixed a list corruption issue (noted below) that was occurring when the first mis-detected digest error report. However, the mis-detected digest error remains. It seems like data for another outstanding write command (released by one of the R2Ts) is interfering with a digest calculation in progress.
Can someone familiar with this area of the take a look at this? I plan to cut the queue depth on the target to 1 today to see if the problem goes away.
Thanks,
-Joe
________________________________
From: Nicholas A. Bellinger <nab@xxxxxxxxxxxxxxx>
To: target-devel <target-devel@xxxxxxxxxxxxxxx>
Cc: Nicholas Bellinger <nab@xxxxxxxxxxxxxxx>; jrepac@xxxxxxxxx
Sent: Monday, January 16, 2012 6:15 PM
Subject: [PATCH 1/2] iscsi-target: Fix reject release handling in iscsit_free_cmd()
From: Nicholas Bellinger <nab@xxxxxxxxxxxxxxx>
This patch addresses a bug where iscsit_free_cmd() was incorrectly calling
iscsit_release_cmd() for ISCSI_OP_REJECT because iscsi_add_reject*() will
overwrite the original iscsi_cmd->iscsi_opcode assignment. This bug was
introduced with the following commit:
commit 0be67f2ed8f577d2c72d917928394c5885fa9134
Author: Nicholas Bellinger <nab@xxxxxxxxxxxxxxx>
Date: Sun Oct 9 01:48:14 2011 -0700
iscsi-target: Remove SCF_SE_LUN_CMD flag abuses
and was manifesting itself as list corruption with the following:
[ 131.191092] ------------[ cut here ]------------
[ 131.191092] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 131.191092] Hardware name: VMware Virtual Platform
[ 131.191092] list_del corruption. prev->next should be ffff880022d3c100, but was 6b6b6b6b6b6b6b6b
[ 131.191092] Modules linked in: tcm_vhost ib_srpt ib_cm ib_sa ib_mad ib_core tcm_qla2xxx qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc crc32c iscsi_target_mod target_core_stgt scsi_tgt target_core_pscsi target_core_file target_core_iblock target_core_mod configfs ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sr_mod cdrom sd_mod e1000 ata_piix libata mptspi mptscsih mptbase [last unloaded: scsi_wait_scan]
[ 131.191092] Pid: 2250, comm: iscsi_ttx Tainted: G W 3.2.0-rc4+ #42
[ 131.191092] Call Trace:
[ 131.191092] [<ffffffff8103b553>] warn_slowpath_common+0x80/0x98
[ 131.191092] [<ffffffff8103b5ff>] warn_slowpath_fmt+0x41/0x43
[ 131.191092] [<ffffffff811d0279>] __list_del_entry+0x8d/0x98
[ 131.191092] [<ffffffffa01395c9>] transport_lun_remove_cmd+0x9b/0xb7 [target_core_mod]
[ 131.191092] [<ffffffffa013a55c>] transport_generic_free_cmd+0x5d/0x71 [target_core_mod]
[ 131.191092] [<ffffffffa01a012b>] iscsit_free_cmd+0x1e/0x27 [iscsi_target_mod]
[ 131.191092] [<ffffffffa01a13be>] iscsit_close_connection+0x14d/0x5b2 [iscsi_target_mod]
[ 131.191092] [<ffffffffa0196a0c>] iscsit_take_action_for_connection_exit+0xdb/0xe0 [iscsi_target_mod]
[ 131.191092] [<ffffffffa01a55d4>] iscsi_target_tx_thread+0x15cb/0x1608 [iscsi_target_mod]
[ 131.191092] [<ffffffff8103609a>] ? check_preempt_wakeup+0x121/0x185
[ 131.191092] [<ffffffff81030801>] ? __dequeue_entity+0x2e/0x33
[ 131.191092] [<ffffffffa01a4009>] ? iscsit_send_text_rsp+0x25f/0x25f [iscsi_target_mod]
[ 131.191092] [<ffffffffa01a4009>] ? iscsit_send_text_rsp+0x25f/0x25f [iscsi_target_mod]
[ 131.191092] [<ffffffff8138f706>] ? schedule+0x55/0x57
[ 131.191092] [<ffffffff81056c7d>] kthread+0x7d/0x85
[ 131.191092] [<ffffffff81399534>] kernel_thread_helper+0x4/0x10
[ 131.191092] [<ffffffff81056c00>] ? kthread_worker_fn+0x16d/0x16d
[ 131.191092] [<ffffffff81399530>] ? gs_change+0x13/0x13
Reported-by: <jrepac@xxxxxxxxx>
Cc: <jrepac@xxxxxxxxx>
Signed-off-by: Nicholas Bellinger <nab@xxxxxxxxxxxxxxx>
---
drivers/target/iscsi/iscsi_target_util.c | 11 +++++++++++
1 files changed, 11 insertions(+), 0 deletions(-)
diff --git a/drivers/target/iscsi/iscsi_target_util.c b/drivers/target/iscsi/iscsi_target_util.c
index a05ca1c..11287e1 100644
--- a/drivers/target/iscsi/iscsi_target_util.c
+++ b/drivers/target/iscsi/iscsi_target_util.c
@@ -849,6 +849,17 @@ void iscsit_free_cmd(struct iscsi_cmd *cmd)
case ISCSI_OP_SCSI_TMFUNC:
transport_generic_free_cmd(&cmd->se_cmd, 1);
break;
+ case ISCSI_OP_REJECT:
+ /*
+ * Handle special case for REJECT when iscsi_add_reject*() has
+ * overwritten the original iscsi_opcode assignment, and the
+ * associated cmd->se_cmd needs to be released.
+ */
+ if (cmd->se_cmd.se_tfo != NULL) {
+ transport_generic_free_cmd(&cmd->se_cmd, 1);
+ break;
+ }
+ /* Fall-through */
default:
iscsit_release_cmd(cmd);
break;
--
1.7.2.5
--
You received this message because you are subscribed to the Google Groups "Linux-iSCSI.org Target Development" group.
To post to this group, send email to linux-iscsi-target-dev@xxxxxxxxxxxxxxxx.
To unsubscribe from this group, send email to linux-iscsi-target-dev+unsubscribe@xxxxxxxxxxxxxxxx.
For more options, visit this group at http://groups.google.com/group/linux-iscsi-target-dev?hl=en.
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
