Hi Or,
We're testing our rdma kernel module, the tests is load module, create
RDMA connection, do some traffic, and unload module.
No mlx4_en involved, in fact we disable mlx4_en in kernel build,
because we don't need that.
I did some debug with gdb:
(gdb)list *mlx4_test_interrupts+0x84a
0xb0ea is in mlx4_eq_int (drivers/net/ethernet/mellanox/mlx4/eq.c:517).
512 in drivers/net/ethernet/mellanox/mlx4/eq.c
513 switch (eqe->type) {
514 case MLX4_EVENT_TYPE_COMP:
515 cqn = be32_to_cpu(eqe->event.comp.cqn) & 0xffffff;
516 mlx4_cq_completion(dev, cqn);
517 break;
(gdb) list *mlx4_cq_completion+0x96
0x9486 is in mlx4_cq_completion (drivers/net/ethernet/mellanox/mlx4/cq.c:117).
(gdb) list *mlx4_ib_destroy_ah+0x37
0x4e7 is in mlx4_ib_cq_comp (drivers/infiniband/hw/mlx4/cq.c:50).
static void mlx4_ib_cq_comp(struct mlx4_cq *cq)
47 {
48 struct ib_cq *ibcq = &to_mibcq(cq)->ibcq;
49 ibcq->comp_handler(ibcq, ibcq->cq_context);
50 }
Looks like cq use-after-free? I have no idea where.
Regards
Jack
2015-07-08 14:19 GMT+02:00 Or Gerlitz <ogerlitz@xxxxxxxxxxxx>:
> On 7/8/2015 12:42 PM, Jack Wang wrote:
>
>> We're using MLX OFED 2.4-1.0.4 together on top of 3.18.14.
>
>
> So this list is for upstream things.. still, let's see
>
>
>> We hit bug below spontaneously, our test trigger this bug around 1 in 5
>> times.
>
>
> and what is your test if I may ask?!
>
>
>
>> HCA 'mlx4_0'
>> CA type: MT26428
>> Number of ports: 2
>> Firmware version: 2.9.1000
>> Hardware version: b0
>>
>> Could you offer some insight, could this be a old bug already fixed,
>> if so, could you point me the link, I can port to our kernel. thanks.
>>
>> [ 657.723842] BUG: unable to handle kernel at ffffffffa02be210
>> [ 657.724245] IP: [<ffffffffa02be210>] 0xffffffffa02be210
>> [ 657.724539] PGD 1c15067
>> [ 657.725162] Oops: 0010 [#1]
>> [ 657.725657] Modules linked in: ib_ipoib ib_uverbs ib_umad mlx4_ib
>> rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr ipv6 null_blk loop
>> amd64_edac_mod k10temp fam15h_power edac
>> _core button microcode hid_generic usbhid hid igb hwmon i2c_algo_bit
>> i2c_core dca ahci ptp libahci ohci_pci pps_core mlx4_core ohci_hcd
>> libata [last unloaded: ibtrs_server]
>> [ 657.731897] CPU: 0 PID: 337 Comm: kworker/u128:1 Tainted: G
>> O 3.18.14-1-ibnbd-debug #1
>> [ 657.732049] Hardware name: Supermicro BHQGE/BHQGE, BIOS 3.00
>> 10/24/2012
>> [ 657.732199] Workqueue: ib_mad1 ib_mad_complete_send_wr [ib_mad]
>> [ 657.732464] task: ffff880415bea1f0 ti: ffff880415420000 task.ti:
>> ffff880415420000
>> [ 657.732610] RIP: 0010:[<ffffffffa02be210>] [<ffffffffa02be210>]
>> 0xffffffffa02be210
>> [ 657.732959] RSP: 0018:ffff880417c03d00 EFLAGS: 00010006
>> [ 657.733193] RAX: ffff8803bc5fc4d8 RBX: ffff8803bc5fc4d8 RCX:
>> 0000000000000000
>> [ 657.733416] RDX: ffff880415bea9e0 RSI: ffff8803d8dcd388 RDI:
>> ffff8803bc5fc4a8
>> [ 657.736094] RBP: ffff880417c03d08 R08: 0000000000000000 R09:
>> ffff880415bea9b8
>> [ 657.736317] R10: 0000000000000000 R11: 0000000000000000 R12:
>> ffff8800d3b00000
>> [ 657.736543] R13: 00000000000000c5 R14: 0000000000000000 R15:
>> 0000000000000020
>> [ 657.736800] FS: 00007f2f05b5f700(0000) GS:ffff880417c00000(0000)
>> knlGS:0000000000000000
>> [ 657.737109] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 657.737330] CR2: ffffffffa02be210 CR3: 000000180d76f000 CR4:
>> 00000000000407f0
>> [ 657.737555] Stack:
>> [ 657.737758] ffffffffa01d84b7 ffff880417c03d48 ffffffffa004a486
>> ffffffffa004a3f5
>> [ 657.738546] ffffffff81c194e0 0000000000000000 00000000c5000000
>> ffff8804136001c0
>> [ 657.739360] ffff8800d3b00000 ffff880417c03e18 ffffffffa004c0ea
>> 0000000000000002
>> [ 657.740149] Call Trace:
>> [ 657.740385] <IRQ>
>> [ 657.740514] [<ffffffffa01d84b7>] ? mlx4_ib_destroy_ah+0x37/0x360
>> [mlx4_ib]
>> [ 657.741093] [<ffffffffa004a486>] mlx4_cq_completion+0x96/0xe0
>> [mlx4_core]
>> [ 657.741330] [<ffffffffa004a3f5>] ? mlx4_cq_completion+0x5/0xe0
>> [mlx4_core]
>> [ 657.741594] [<ffffffffa004c0ea>] mlx4_test_interrupts+0x84a/0x1100
>> [mlx4_core]
>
>
> mlx4_test_interrupts is called from the mlx4_en ethtool selftest handler, so
> you are
> calling it while X (what?) is done in parallel?
>
>
>
>
>> [ 657.741908] [<ffffffff8109f37a>] ? __lock_acquire.isra.28+0x3aa/0xcb0
>>
>> [ 657.742142] [<ffffffffa004c904>]
>> mlx4_test_interrupts+0x1064/0x1100 [mlx4_core]
>> [ 657.742457] [<ffffffff810aa678>] handle_irq_event_percpu+0x78/0x2b0
>> [ 657.742685] [<ffffffff810aa8f8>] handle_irq_event+0x48/0x70
>> [ 657.742934] [<ffffffff810adf58>] handle_edge_irq+0xc8/0x160
>> [ 657.743160] [<ffffffff8100515e>] handle_irq+0x14e/0x200
>> [ 657.743384] [<ffffffff815fea3e>] do_IRQ+0x5e/0x110
>> [ 657.743603] [<ffffffff815fcf6a>] common_interrupt+0x6a/0x6a
>> [ 657.743826] <EOI>
>> [ 657.743957] [<ffffffff81197295>] ? __slab_alloc+0x615/0x710
>> [ 657.744513] [<ffffffffa01d80de>] ? mlx4_ib_create_ah+0x2e/0x2a0
>> [mlx4_ib]
>> [ 657.744738] [<ffffffffa0195603>] ? ib_create_send_mad+0xf3/0x330
>> [ib_mad]
>> [ 657.744968] [<ffffffff81198f12>] __kmalloc+0x162/0x2e0
>> [ 657.745191] [<ffffffffa0195603>] ? ib_create_send_mad+0xf3/0x330
>> [ib_mad]
>> [ 657.745420] [<ffffffffa01d8100>] ? mlx4_ib_create_ah+0x50/0x2a0
>> [mlx4_ib]
>> [ 657.745650] [<ffffffffa0195603>] ib_create_send_mad+0xf3/0x330
>> [ib_mad]
>> [ 657.745875] [<ffffffffa019985b>] agent_send_response+0xbb/0x270
>> [ib_mad]
>> [ 657.746103] [<ffffffffa0198bf4>] ?
>> ib_mad_complete_send_wr+0x844/0xfa0 [ib_mad]
>> [ 657.746413] [<ffffffffa0198f96>]
>> ib_mad_complete_send_wr+0xbe6/0xfa0 [ib_mad]
>> [ 657.746729] [<ffffffff8109f37a>] ? __lock_acquire.isra.28+0x3aa/0xcb0
>> [ 657.746959] [<ffffffff8106c82d>] process_one_work+0x33d/0x6d0
>> [ 657.747181] [<ffffffff8106c7a4>] ? process_one_work+0x2b4/0x6d0
>> [ 657.747434] [<ffffffff8106d015>] worker_thread+0x55/0x6d0
>> [ 657.751224] [<ffffffff8106cfc0>] ? rescuer_thread+0x3c0/0x3c0
>> [ 657.751482] [<ffffffff81073e84>] kthread+0xe4/0x100
>> [ 657.751705] [<ffffffff810792b4>] ? finish_task_switch+0x84/0x140
>> [ 657.751935] [<ffffffff81073da0>] ? kthread_create_on_node+0x280/0x280
>> [ 657.752165] [<ffffffff815fc3c8>] ret_from_fork+0x58/0x90
>> [ 657.752391] [<ffffffff81073da0>] ? kthread_create_on_node+0x280/0x280
>> [ 657.752640] Code: Bad RIP value.
>> [ 657.753095] RIP [<ffffffffa02be210>] 0xffffffffa02be210
>> [ 657.753434] RSP <ffff880417c03d00>
>> [ 657.753645] CR2: ffffffffa02be210
>> [ 657.753878] ---[ end trace 9c9225f5e490f806 ]---
>> [ 657.765754] Kernel panic - not syncing: Fatal exception in interrupt
>> [ 657.766089] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation
>> range: 0xffffffff80000000-0xffffffff9fffffff)
>> [ 657.778084] ---[ end Kernel panic - not syncing: Fatal exception in
>> interrupt
>>
>> Best regards,
>> Jack Wang
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
