On 10/04/2023 21:13, Guoqing Jiang wrote:
>
>
> On 4/10/23 14:43, Li Zhijian wrote:
>> Currently, con will be destroyed when wait_event_interruptible_timeout()
>> returns ERESTARTSYS. But the in-flight event handler
>> rtrs_clt_rdma_cm_handler() could be rescheduled/wakeup which
>> may cause a use-after-free.
>>
>> WARNING: CPU: 0 PID: 14766 at drivers/infiniband/ulp/rtrs/rtrs-clt.c:1687 rtrs_clt_rdma_cm_handler+0x620/0x640 [rtrs_client]
>> Modules linked in: rnbd_client rtrs_client rtrs_core rdma_cm iw_cm ib_cm rdma_rxe ib_uverbs ib_core libiscsi scsi_transport_iscsi crc32_generic udp_tunnel dax_pmem nd_pmem nd_btt virtiofs crc32c_intel nvme fuse nvme_core nfit
>> libnvdimm dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_core]
>> CPU: 0 PID: 14766 Comm: kworker/u2:3 Kdump: loaded Tainted: G W 6.2.0-rc6-roce-flush+ #56
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
>> Workqueue: ib_addr process_one_req [ib_core]
>> RIP: 0010:rtrs_clt_rdma_cm_handler+0x620/0x640 [rtrs_client]
>> Code: 00 0f 85 5f fd ff ff 4c 8b 23 41 bd f4 ff ff ff e9 95 fb ff ff 0f 0b 4c 89 f7 41 bd ea ff ff ff e8 75 c8 92 ec e9 4b ff ff ff <0f> 0b 4c 89 f7 41 bd ea ff ff ff e8 60 c8 92 ec e9 36 ff ff ff e8
>> RSP: 0018:ffffa4ef41cdbc60 EFLAGS: 00010246
>> RAX: 0000000000000000 RBX: ffff9372c394e600 RCX: 0000000000000001
>> RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffffad634277
>> RBP: ffffa4ef41cdbd00 R08: 0000000000000000 R09: 0000000000000001
>> R10: 0000000000003ff3 R11: 0000000000000000 R12: ffff9372c3164800
>> R13: ffff9372c3164800 R14: ffff9372c394e640 R15: ffff9372c5219020
>> FS: 0000000000000000(0000) GS:ffff9372fbc00000(0000) knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 00007f35bb7d5de0 CR3: 0000000020c2a006 CR4: 00000000001706f0
>> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> Call Trace:
>> <TASK>
>> ? mark_held_locks+0x49/0x80
>> ? lock_is_held_type+0xd7/0x130
>> ? cma_cm_event_handler+0x49/0x200 [rdma_cm]
>> cma_cm_event_handler+0x49/0x200 [rdma_cm]
>> addr_handler+0xf1/0x1e0 [rdma_cm]
>> ? lock_acquire+0xca/0x2f0
>> ? lock_acquire+0xda/0x2f0
>> process_one_req+0x43/0x170 [ib_core]
>> process_one_work+0x274/0x590
>> worker_thread+0x4f/0x3d0
>> ? __pfx_worker_thread+0x10/0x10
>> kthread+0xe7/0x110
>> ? __pfx_kthread+0x10/0x10
>> ret_from_fork+0x2c/0x50
>> </TASK>
>> irq event stamp: 1432669
>> hardirqs last enabled at (1432683): [<ffffffffac508eb2>] __up_console_sem+0x52/0x60
>> hardirqs last disabled at (1432698): [<ffffffffac508e97>] __up_console_sem+0x37/0x60
>> softirqs last enabled at (1432518): [<ffffffffac48c985>] __irq_exit_rcu+0xc5/0x120
>> softirqs last disabled at (1432509): [<ffffffffac48c985>] __irq_exit_rcu+0xc5/0x120
>> ---[ end trace 0000000000000000 ]---
>>
>> Signed-off-by: Li Zhijian <lizhijian@xxxxxxxxxxx>
>> ---
>> drivers/infiniband/ulp/rtrs/rtrs-clt.c | 14 ++++++++++----
>> 1 file changed, 10 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
>> index 4c8f42e46e2f..760a7eb51297 100644
>> --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c
>> +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c
>> @@ -2074,6 +2074,7 @@ static int create_cm(struct rtrs_clt_con *con)
>> rtrs_err(s, "Failed to resolve address, err: %d\n", err);
>> goto destroy_cm;
>> }
>> +again:
>> /*
>> * Combine connection status and session events. This is needed
>> * for waiting two possible cases: cm_err has something meaningful
>> @@ -2083,10 +2084,15 @@ static int create_cm(struct rtrs_clt_con *con)
>> clt_path->state_wq,
>> con->cm_err || clt_path->state != RTRS_CLT_CONNECTING,
>> msecs_to_jiffies(RTRS_CONNECT_TIMEOUT_MS));
>> - if (err == 0 || err == -ERESTARTSYS) {
>> - if (err == 0)
>> - err = -ETIMEDOUT;
>> - /* Timedout or interrupted */
>> + if (err == -ERESTARTSYS) {
>> + /* interrupted,
>> + * try again to avoid the in-flight rtrs_clt_rdma_cm_handler()
>> + * getting a use-after-free
>> + */
>> + goto again;
>> + } else if (err == 0) {
>> + err = -ETIMEDOUT;
>> + /* Timedout */
>> goto errr;
>> }
>
> Can event handler still be triggered in case of timeout?
I have never hit such race.
>
> And I guess either stop_cm -> rdma_disconnect or destroy_cm -> rdma_destroy_id
> should prevent this kind of racy issue.
In practice, they are possible that rtrs_clt_rdma_cm_handler() is in-flight during
'either stop_cm -> rdma_disconnect or destroy_cm -> rdma_destroy_id'. rtrs_clt_rdma_cm_handler() and
cm's cleanup path need to hold mutex_lock(&con->con_mutex), once cm's cleanup path get this lock first
rtrs_clt_rdma_cm_handler has to sleep, when rtrs_clt_rdma_cm_handler is wakeup again, some resources has been
freed by cm's cleanup path.
Thanks
>
> Thanks,
> Guoqing
