Hi Parav,
I have built and installed on 3 servers the same kernel version with
your patch applied. So far, so good - IPoIB is working, no kernel errors
in the logs.
Please let me know when you have "proper fix", we need to push it to Fedora.
BTW There is another qp = kmalloc(...) in the code. Does it need to be
changed?
$ grep -n -C6 'qp = kmalloc' drivers/infiniband/hw/mthca/mthca_provider.c
596- case IB_QPT_GSI:
597- {
598- /* Don't allow userspace to create special QPs */
599- if (pd->uobject)
600- return ERR_PTR(-EINVAL);
601-
602: qp = kmalloc(sizeof (struct mthca_sqp), GFP_KERNEL);
603- if (!qp)
604- return ERR_PTR(-ENOMEM);
605-
606- qp->ibqp.qp_num = init_attr->qp_type == IB_QPT_SMI ? 0 : 1;
607-
608- err = mthca_alloc_sqp(to_mdev(pd->device), to_mpd(pd),
Best regards,
Alex Murashkin
On 12/30/18 8:21 PM, Parav Pandit wrote:
> Hi Alexander,
>
>> -----Original Message-----
>> From: linux-rdma-owner@xxxxxxxxxxxxxxx <linux-rdma-
>> owner@xxxxxxxxxxxxxxx> On Behalf Of Alexander Murashkin
>> Sent: Monday, December 31, 2018 3:26 AM
>> To: linux-rdma@xxxxxxxxxxxxxxx
>> Subject: ib_ipoib: general protection fault in ib_destroy_qp ->
>> rdma_put_gid_attr+0x9/0x30 [ib_core]
>>
>> ipoib crashes in rdma_put_gid_attr. It happens every time, often during boot
>> process or soon after it, occasionally after few hours since a reboot.
>>
>> After the crash, IPoIB stops working for new connections. Interesting fact is
>> that TCP sessions created before the crash continue to work.
>>
>> The problem occurs on four (4) servers. The servers are running Fedora 29
>> with kernel 4.19.10-300.fc29.x86_64. Note that 4.19.8-300.fc29.x86_64 has
>> the same problem.
>>
>> The servers use the same Infiniband controller model, OS, kernel, and drivers
>>
>> Device: 02:00.0 InfiniBand: Mellanox Technologies MT25208 [InfiniHost III
>> Ex] (rev a0)
>> Firmware: 5.3.0
>> Driver: ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4, 2008)
>>
>> More details at https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.redhat.com%2Fshow_bug.cgi%3Fid%3D1661864&data=02%7C01%7C%7Caaa37633966646eea1cc08d66ec69f57%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636818196681805142&sdata=c7F76JB1t2c7le4VfL65yH%2BBv1Lrdd6ZtIiuWFUs4bs%3D&reserved=0
>>
>> -------------------------------------------------------------------------
>> Additional info:
>> reporter: libreport-2.9.7
>> general protection fault: 0000 [#1] SMP NOPTI
>> CPU: 3 PID: 74 Comm: kworker/u16:1 Not tainted 4.19.10-300.fc29.x86_64 #1
>> Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A99X EVO,
>> BIOS 0402 05/16/2011
>> Workqueue: ipoib_wq ipoib_cm_tx_reap [ib_ipoib]
>> RIP: 0010:rdma_put_gid_attr+0x9/0x30 [ib_core]
>> Code: 96 5b 5d 41 5c 41 5d 41 5e 41 5f c3 48 8b 7b 30 e8 cc 0d c6 f1 48 89 df
>> e8 c4 0d c6 f1 eb c3 c3 90 0f 1f 44 00 00 48 8d 57 d8 <f0> ff 4f d8 0f 88 78 65
>> 01 00 74 01 c3 48 8b 35 2b d0 02 00 48 83
>> RSP: 0018:ffffb7ad819dbde8 EFLAGS: 00010202
>> RAX: 0000000000000000 RBX: ffff8d1bdf5a2e00 RCX: 0000000000002699
>> RDX: 206c656e72656af8 RSI: ffff8d1bf7ae6160 RDI: 206c656e72656b20
>> RBP: 0000000000000000 R08: 0000000000026160 R09: ffffffffc06b45bf
>> R10: ffffe849887da000 R11: 0000000000000002 R12: ffff8d1be30cb400
>> R13: ffff8d1bdf681800 R14: ffff8d1be2272400 R15: ffff8d1be30ca000
>> FS: 0000000000000000(0000) GS:ffff8d1bf7ac0000(0000)
>> knlGS:0000000000000000
>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 00007f4f99d5dc80 CR3: 000000021878e000 CR4: 00000000000006e0 Call
>> Trace:
>> ib_destroy_qp+0xc9/0x240 [ib_core]
>> ipoib_cm_tx_reap+0x1f9/0x4e0 [ib_ipoib]
>> process_one_work+0x1a1/0x3a0
>> worker_thread+0x30/0x380
>> ? pwq_unbound_release_workfn+0xd0/0xd0
>> kthread+0x112/0x130
>> ? kthread_create_worker_on_cpu+0x70/0x70
>> ret_from_fork+0x22/0x40
>> Modules linked in: nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_multiport
>> 8021q garp mrp stp llc ip6t_REJECT nf_reject_ipv6 xt_state xt_conntrack
>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c it87 hwmon_vid
>> ip6table_filter ip6_tables ib_isert iscsi_target_mod ib_srpt target_core_mod
>> ib_srp scsi_transport_srp rpcrdma rdma_ucm ib_uverbs ib_iser ib_umad
>> rdma_cm iw_cm ib_ipoib libiscsi scsi_transport_iscsi ib_cm eeepc_wmi
>> amd64_edac_mod asus_wmi edac_mce_amd sparse_keymap rfkill kvm_amd
>> video wmi_bmof mxm_wmi kvm irqbypass k10temp snd_hda_codec_realtek
>> snd_hda_codec_hdmi snd_hda_codec_generic snd_hda_intel
>> snd_hda_codec snd_hda_core ib_mthca sp5100_tco snd_seq snd_hwdep
>> snd_seq_device i2c_piix4 snd_pcm ib_core snd_timer snd soundcore wmi
>> pcc_cpufreq acpi_cpufreq nfsd binfmt_misc nfs_acl
>> lockd grace auth_rpcgss sunrpc dm_crypt raid1 ata_generic i2c_algo_bit uas
>> drm_kms_helper pata_acpi ttm usb_storage pata_marvell drm firewire_ohci
>> firewire_core crc_itu_t r8169 ecryptfs
>>
>> ---------------------------------------------------
>>
>> # lspci | grep Mellanox
>> 02:00.0 InfiniBand: Mellanox Technologies MT25208 [InfiniHost III Ex] (rev a0)
>>
>> # ibv_devinfo
>> hca_id: mthca0
>> transport: InfiniBand (0)
>> fw_ver: 5.3.0
>> node_guid: 0002:c902:0022:1228
>> sys_image_guid: 0005:ad00:0100:d050
>> vendor_id: 0x02c9
>> vendor_part_id: 25218
>> hw_ver: 0xA0
>> board_id: MT_0150000001
>> phys_port_cnt: 2
>> port: 1
>> state: PORT_ACTIVE (4)
>> max_mtu: 2048 (4)
>> active_mtu: 2048 (4)
>> sm_lid: 2
>> port_lid: 5
>> port_lmc: 0x00
>> link_layer: InfiniBand
>>
>> port: 2
>> state: PORT_ACTIVE (4)
>> max_mtu: 2048 (4)
>> active_mtu: 2048 (4)
>> sm_lid: 2
>> port_lid: 6
>> port_lmc: 0x00
>> link_layer: InfiniBand
>>
> It seems that qp by mthca driver is not zero initialized during creation time.
> Due to which there might be garbage pointer for alt_sgid_attr.
>
> Is it possible to apply/change below code and see if it progresses?
> I will generate a proper fix if this is the likely root cause.
>
> diff --git a/drivers/infiniband/hw/mthca/mthca_provider.c b/drivers/infiniband/hw/mthca/mthca_provider.c
> index bfd741c..9f6c748 100644
> --- a/drivers/infiniband/hw/mthca/mthca_provider.c
> +++ b/drivers/infiniband/hw/mthca/mthca_provider.c
> @@ -533,7 +533,7 @@ static struct ib_qp *mthca_create_qp(struct ib_pd *pd,
> {
> struct mthca_ucontext *context;
>
> - qp = kmalloc(sizeof *qp, GFP_KERNEL);
> + qp = kzalloc(sizeof *qp, GFP_KERNEL);
> if (!qp)
> return ERR_PTR(-ENOMEM);
>
