On 2019/2/26 20:32, Håkon Bugge wrote:
On 14 Feb 2019, at 01:03, Yanjun Zhu <yanjun.zhu@xxxxxxxxxx> wrote:
On 2019/1/23 0:15, Jason Gunthorpe wrote:
On Tue, Jan 22, 2019 at 02:18:21AM -0500, Zhu Yanjun wrote:
When the interface related with IB device is set to down/up over and
over again, the following call trace will pop out.
"
Call Trace:
[<ffffffffa039ff8d>] ib_mad_completion_handler+0x7d/0xa0 [ib_mad]
[<ffffffff810a1a41>] process_one_work+0x151/0x4b0
[<ffffffff810a1ec0>] worker_thread+0x120/0x480
[<ffffffff810a709e>] kthread+0xce/0xf0
[<ffffffff816e9962>] ret_from_fork+0x42/0x70
RIP [<ffffffffa039f926>] ib_mad_recv_done_handler+0x26/0x610 [ib_mad]
"
From vmcore, we can find the following:
"
crash7lates> struct ib_mad_list_head ffff881fb3713400
struct ib_mad_list_head {
list = {
next = 0xffff881fb3713800,
prev = 0xffff881fe01395c0
},
mad_queue = 0x0
}
"
Before the call trace, a lot of ib_cancel_mad is sent to the sender.
So it is necessary to check mad_queue in struct ib_mad_list_head to avoid
"kernel NULL pointer" error.
Signed-off-by: Zhu Yanjun <yanjun.zhu@xxxxxxxxxx>
drivers/infiniband/core/mad.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c
index 7870823bac47..ab5a7d1152ca 100644
+++ b/drivers/infiniband/core/mad.c
@@ -2250,6 +2250,17 @@ static void ib_mad_recv_done(struct ib_cq *cq, struct ib_wc *wc)
return;
}
+ if (unlikely(!mad_list->mad_queue)) {
+ /*
+ * When the interface related with IB device is set to down/up,
+ * a lot of ib_cancel_mad packets are sent to the sender. In
+ * sender, the mad packets are cancelled. The receiver will
+ * find mad_queue NULL. If the receiver does not test mad_queue,
+ * the receiver will crash with "kernel NULL pointer" error.
+ */
How does it become null here?
Hi, Jason
After upgrading IB switch to version 2.2.9-3, this problem disappears. It seems that IB switch results in this problem.
Hi Yanjun,
I would like to rephrase: After changing the fw in the IB switch (where SA and OpenSM runs), we are not exposed to the bug any more.
This seems very much to be a bug. A kernel shall not crash - and that shall hold true independent of external conditions, such as fw versions in switches.
So, I have two requests for you:
1. Please see if the bug can be reproed with an upstream kernel.
Hi, Haakon
Thanks a lot for your review this patch.
I have no test environment in LAB to reproduce this bug. When the bug
occurred, I made tests in the customer's host with this patch. And this
patch can fix this bug and do not introduce new problems. So I think
this patch is a fix or workaround to this bug.
And I analyzed the vmcore. Then I found that the packet that caused this
bug was from a IB switch. And I checked the source code (kernel 2.6.x)
of IB switch. I find that there is difference in mad between IB switch
and the hosts(kernel 4.x). This is why I suspect IB switch.
The customer's hosts are the production machine. I suspect that the
customer can borrow his hosts for us to make tests.
Any way, this patch can fix this problem since this problem disappears
and no new problem appears after this patch is applied.
As to the root cause, I will make further investigations to find it.:-)
Zhu Yanjun
2. (Assuming yes to the above), your commit doesn't fix the problem, it just diminishes it. If mad_list->mad_queue can become NULL asynchronously to ib_mad_recv_done() being called, it can become NULL just after you tested it to be non-NULL, right?
Thxs, Håkon
Thanks,
Zhu Yanjun
Jason
