Hi
After testing this patch, No sending request failure occurred, the issue has not been reproduced yet.
It may take a long time to test.
best wishes
zhang.guanghui@xxxxxxxx
发件人： Sagi Grimberg
发送时间： 2025-02-17 15:46
收件人： zhang.guanghui@xxxxxxxx; mgurtovoy; kbusch; sashal; chunguang.xu
抄送： linux-kernel; linux-nvme; linux-block
主题： Re: nvme-tcp: fix a possible UAF when failing to send request【请注意，邮件由sagigrim@xxxxxxxxx代发】
On 10/02/2025 9:41, zhang.guanghui@xxxxxxxx wrote:
> Hello
>
>
>
> When using the nvme-tcp driver in a storage cluster, the driver may trigger a null pointer causing the host to crash several times.
>
>
>
> By analyzing the vmcore, we know the direct cause is that the request->mq_hctx was used after free.
>
>
>
>
>
> CPU1 CPU2
>
>
>
> nvme_tcp_poll nvme_tcp_try_send --failed to send reqrest 13
>
>
>
> nvme_tcp_try_recv nvme_tcp_fail_request
>
>
>
> nvme_tcp_recv_skb nvme_tcp_end_request
>
>
>
> nvme_tcp_recv_pdu nvme_complete_rq
>
>
>
> nvme_tcp_handle_comp nvme_retry_req -- request->mq_hctx have been freed, is NULL.
>
>
>
> nvme_tcp_process_nvme_cqe
>
>
>
> nvme_complete_rq
>
>
>
> nvme_end_req
>
>
>
> blk_mq_end_request
>
>
>
>
>
>
>
> when nvme_tcp_try_send failed to send reqrest 13, it maybe be resulted by selinux or other reasons, this is a problem. then the nvme_tcp_fail_request would execute。
>
>
>
> but the nvme_tcp_recv_pdu may have received the responding pdu and the nvme_tcp_process_nvme_cqe would have completed the request. request->mq_hctx was used after free.
>
>
>
> the follow patch is to solve it.
Zhang, your email client needs fixing - it is impossible to follow your
emails.
>
>
>
> can you give some suggestions? thanks!
The problem is the C2HTerm that the host is unable to handle correctly.
And it also appears that nvme_tcp_poll() does not signal correctly to
blk-mq to stop
calling poll.
One thing to do is to handle C2HTerm PDU correctly, and, here is a
possible fix to try for the UAF issue:
--
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index c637ff04a052..0e390e98aaf9 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -2673,6 +2673,7 @@ static int nvme_tcp_poll(struct blk_mq_hw_ctx
*hctx, struct io_comp_batch *iob)
{
struct nvme_tcp_queue *queue = hctx->driver_data;
struct sock *sk = queue->sock->sk;
+ int ret;
if (!test_bit(NVME_TCP_Q_LIVE, &queue->flags))
return 0;
@@ -2680,9 +2681,9 @@ static int nvme_tcp_poll(struct blk_mq_hw_ctx
*hctx, struct io_comp_batch *iob)
set_bit(NVME_TCP_Q_POLLING, &queue->flags);
if (sk_can_busy_loop(sk) &&
skb_queue_empty_lockless(&sk->sk_receive_queue))
sk_busy_loop(sk, true);
- nvme_tcp_try_recv(queue);
+ ret = nvme_tcp_try_recv(queue);
clear_bit(NVME_TCP_Q_POLLING, &queue->flags);
- return queue->nr_cqe;
+ return ret < 0 ? ret : queue->nr_cqe;
}
static int nvme_tcp_get_address(struct nvme_ctrl *ctrl, char *buf, int
size)
--
Does this help?
