On Wed, 2015-02-11 at 10:17 +0200, Sagi Grimberg wrote:
> Hey Nic,
>
> So Our QA guys recently stepped on this bug when performing stress
> login-logout from a single initiator to 10 targets each exposed over
> 4 portals, so overall 40 sessions (needless to say we are talking on
> iser...). So there are lots of logins in parallel with lots of logouts.
>
> It seems that the connection termination causes iscsi_tx_thread to
> access the connection after it is freed or something (list corruption
> probably coming from iscsit_handle_immediate_queue or
> iscsit_handle_response_queue, and NULL deref coming from
> iscsit_take_action_for_connection_exit).
>
> Note, isert_wait_conn waits for session commands and QP flush which is
> normally pretty fast, the conn termination is done in a work that waits
> for DISCONNECTED event which might take longer (which is why we do it
> outside wait_conn context to avoid blocking it).
>
> I didn't get too far with this until now, do you have any idea on what
> might have happened?
Mmm, it looks like iscsit_take_action_for_connection_exit() in TX thread
context is calling iscsi_close_connection() after hitting the following
check in iscsi_target_erl0.c:
if (conn->conn_state == TARG_CONN_STATE_IN_LOGOUT) {
spin_unlock_bh(&conn->state_lock);
iscsit_close_connection(conn);
return;
}
.. once iscsi_close_connection() has already being called earlier by
iser-target code.
AFAICT, this check is specific to traditional iscsi-target during the
iscsit_handle_logout_cmd() failure case in iscsi_target_rx_opcode().
Here's a quick patch to make this case ISCSI_TCP only.
Thanks Sagi.
--nab
diff --git a/drivers/target/iscsi/iscsi_target_erl0.c b/drivers/target/iscsi/iscsi_target_erl0.c
index bdd8731..1c197ba 100644
--- a/drivers/target/iscsi/iscsi_target_erl0.c
+++ b/drivers/target/iscsi/iscsi_target_erl0.c
@@ -22,6 +22,7 @@
#include <target/target_core_fabric.h>
#include <target/iscsi/iscsi_target_core.h>
+#include <target/iscsi/iscsi_transport.h>
#include "iscsi_target_seq_pdu_list.h"
#include "iscsi_target_tq.h"
#include "iscsi_target_erl0.h"
@@ -939,7 +940,8 @@ void iscsit_take_action_for_connection_exit(struct iscsi_conn *conn)
if (conn->conn_state == TARG_CONN_STATE_IN_LOGOUT) {
spin_unlock_bh(&conn->state_lock);
- iscsit_close_connection(conn);
+ if (conn->conn_transport->transport_type == ISCSI_TCP)
+ iscsit_close_connection(conn);
return;
}
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
