Patch "IB/isert: Fix dead lock in ib_isert" has been added to the 6.1-stable tree

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



This is a note to let you know that I've just added the patch titled

    IB/isert: Fix dead lock in ib_isert

to the 6.1-stable tree which can be found at:
    http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary

The filename of the patch is:
     ib-isert-fix-dead-lock-in-ib_isert.patch
and it can be found in the queue-6.1 subdirectory.

If you, or anyone else, feels it should not be added to the stable tree,
please let <stable@xxxxxxxxxxxxxxx> know about it.



commit eb58e9d5cf0ab04a64006dca7b43d9d910077b2c
Author: Saravanan Vajravel <saravanan.vajravel@xxxxxxxxxxxx>
Date:   Tue Jun 6 03:25:29 2023 -0700

    IB/isert: Fix dead lock in ib_isert
    
    [ Upstream commit 691b0480933f0ce88a81ed1d1a0aff340ff6293a ]
    
    - When a iSER session is released, ib_isert module is taking a mutex
      lock and releasing all pending connections. As part of this, ib_isert
      is destroying rdma cm_id. To destroy cm_id, rdma_cm module is sending
      CM events to CMA handler of ib_isert. This handler is taking same
      mutex lock. Hence it leads to deadlock between ib_isert & rdma_cm
      modules.
    
    - For fix, created local list of pending connections and release the
      connection outside of mutex lock.
    
    Calltrace:
    ---------
    [ 1229.791410] INFO: task kworker/10:1:642 blocked for more than 120 seconds.
    [ 1229.791416]       Tainted: G           OE    --------- -  - 4.18.0-372.9.1.el8.x86_64 #1
    [ 1229.791418] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    [ 1229.791419] task:kworker/10:1    state:D stack:    0 pid:  642 ppid:     2 flags:0x80004000
    [ 1229.791424] Workqueue: ib_cm cm_work_handler [ib_cm]
    [ 1229.791436] Call Trace:
    [ 1229.791438]  __schedule+0x2d1/0x830
    [ 1229.791445]  ? select_idle_sibling+0x23/0x6f0
    [ 1229.791449]  schedule+0x35/0xa0
    [ 1229.791451]  schedule_preempt_disabled+0xa/0x10
    [ 1229.791453]  __mutex_lock.isra.7+0x310/0x420
    [ 1229.791456]  ? select_task_rq_fair+0x351/0x990
    [ 1229.791459]  isert_cma_handler+0x224/0x330 [ib_isert]
    [ 1229.791463]  ? ttwu_queue_wakelist+0x159/0x170
    [ 1229.791466]  cma_cm_event_handler+0x25/0xd0 [rdma_cm]
    [ 1229.791474]  cma_ib_handler+0xa7/0x2e0 [rdma_cm]
    [ 1229.791478]  cm_process_work+0x22/0xf0 [ib_cm]
    [ 1229.791483]  cm_work_handler+0xf4/0xf30 [ib_cm]
    [ 1229.791487]  ? move_linked_works+0x6e/0xa0
    [ 1229.791490]  process_one_work+0x1a7/0x360
    [ 1229.791491]  ? create_worker+0x1a0/0x1a0
    [ 1229.791493]  worker_thread+0x30/0x390
    [ 1229.791494]  ? create_worker+0x1a0/0x1a0
    [ 1229.791495]  kthread+0x10a/0x120
    [ 1229.791497]  ? set_kthread_struct+0x40/0x40
    [ 1229.791499]  ret_from_fork+0x1f/0x40
    
    [ 1229.791739] INFO: task targetcli:28666 blocked for more than 120 seconds.
    [ 1229.791740]       Tainted: G           OE    --------- -  - 4.18.0-372.9.1.el8.x86_64 #1
    [ 1229.791741] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    [ 1229.791742] task:targetcli       state:D stack:    0 pid:28666 ppid:  5510 flags:0x00004080
    [ 1229.791743] Call Trace:
    [ 1229.791744]  __schedule+0x2d1/0x830
    [ 1229.791746]  schedule+0x35/0xa0
    [ 1229.791748]  schedule_preempt_disabled+0xa/0x10
    [ 1229.791749]  __mutex_lock.isra.7+0x310/0x420
    [ 1229.791751]  rdma_destroy_id+0x15/0x20 [rdma_cm]
    [ 1229.791755]  isert_connect_release+0x115/0x130 [ib_isert]
    [ 1229.791757]  isert_free_np+0x87/0x140 [ib_isert]
    [ 1229.791761]  iscsit_del_np+0x74/0x120 [iscsi_target_mod]
    [ 1229.791776]  lio_target_np_driver_store+0xe9/0x140 [iscsi_target_mod]
    [ 1229.791784]  configfs_write_file+0xb2/0x110
    [ 1229.791788]  vfs_write+0xa5/0x1a0
    [ 1229.791792]  ksys_write+0x4f/0xb0
    [ 1229.791794]  do_syscall_64+0x5b/0x1a0
    [ 1229.791798]  entry_SYSCALL_64_after_hwframe+0x65/0xca
    
    Fixes: bd3792205aae ("iser-target: Fix pending connections handling in target stack shutdown sequnce")
    Reviewed-by: Sagi Grimberg <sagi@xxxxxxxxxxx>
    Signed-off-by: Selvin Xavier <selvin.xavier@xxxxxxxxxxxx>
    Signed-off-by: Saravanan Vajravel <saravanan.vajravel@xxxxxxxxxxxx>
    Link: https://lore.kernel.org/r/20230606102531.162967-2-saravanan.vajravel@xxxxxxxxxxxx
    Signed-off-by: Leon Romanovsky <leon@xxxxxxxxxx>
    Signed-off-by: Sasha Levin <sashal@xxxxxxxxxx>

diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index 7cca171478a22..50d7373b425c0 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -2432,6 +2432,7 @@ isert_free_np(struct iscsi_np *np)
 {
 	struct isert_np *isert_np = np->np_context;
 	struct isert_conn *isert_conn, *n;
+	LIST_HEAD(drop_conn_list);
 
 	if (isert_np->cm_id)
 		rdma_destroy_id(isert_np->cm_id);
@@ -2451,7 +2452,7 @@ isert_free_np(struct iscsi_np *np)
 					 node) {
 			isert_info("cleaning isert_conn %p state (%d)\n",
 				   isert_conn, isert_conn->state);
-			isert_connect_release(isert_conn);
+			list_move_tail(&isert_conn->node, &drop_conn_list);
 		}
 	}
 
@@ -2462,11 +2463,16 @@ isert_free_np(struct iscsi_np *np)
 					 node) {
 			isert_info("cleaning isert_conn %p state (%d)\n",
 				   isert_conn, isert_conn->state);
-			isert_connect_release(isert_conn);
+			list_move_tail(&isert_conn->node, &drop_conn_list);
 		}
 	}
 	mutex_unlock(&isert_np->mutex);
 
+	list_for_each_entry_safe(isert_conn, n, &drop_conn_list, node) {
+		list_del_init(&isert_conn->node);
+		isert_connect_release(isert_conn);
+	}
+
 	np->np_context = NULL;
 	kfree(isert_np);
 }



[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux