On 2/19/2015 9:23 PM, Chris Moore wrote:
We're running RHEL 7.1 kernel plus the patches that Sagi helped port for me. On the target we're getting the stack trace below (list_del corruption). Leading up to the stack trace we get a bunch of: Feb 10 16:47:28 target-1 kernel: ib_post_send() failed for IB_WR_RDMA_READ Feb 10 16:47:28 target-1 kernel: ib_post_send failed with -12
Hey Chris, It seems that you ran out of SQ space. Does the rhel7.1 code include the completion coalescing code? Can you tell us the scenario you ran to invoke this bug? The trace below indicate some form of use-after-free where we try to delete cmd->i_conn_node.
Followed by a bunch of aborts: Feb 10 16:48:30 target-1 kernel: ABORT_TASK: Found referenced iSCSI task_tag: 26 Feb 10 16:48:30 target-1 kernel: ABORT_TASK: ref_tag: 26 already complete, skipping Feb 10 16:48:30 target-1 kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 26 I'm working on tracking down the cause of the ib_post_send failure. Has anyone seen the list_del corruption issue?
I am also seeing these kind of corruptions on rhel7.0 when working against multiple initiators (~60 sessions) doing stress logins/logouts. It doesn't look good...
Feb 10 16:50:19 target-1 kernel: ------------[ cut here ]------------ Feb 10 16:50:19 target-1 kernel: WARNING: at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0() Feb 10 16:50:19 target-1 kernel: list_del corruption. prev->next should be ffff8800c10b5040, but was ffff8802f35fb640 Feb 10 16:50:19 target-1 kernel: Modules linked in: tcp_lp target_core_pscsi target_core_file target_core_iblock 8021q garp mrp bnep bluetooth rfkill fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT iptable_filter ip_tables tun bridge stp llc xprtrdma sunrpc ib_isert(F) iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad iTCO_wdt iTCO_vendor_support ocrdma ib_core ib_addr lpc_ich serio_raw shpchp wmi mfd_core dm_mirror dm_region_hash dm_log dm_mod intel_powerclamp ipmi_devintf coretemp pcspkr kvm_intel kvm dcdbas crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel Feb 10 16:50:19 target-1 kernel: aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_si ipmi_msghandler acpi_power_meter uinput i7core_edac edac_core ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_common sr_mod cdrom ata_generic pata_acpi mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm mptsas ata_piix be2net scsi_transport_sas libata i2c_core mptscsih vxlan mptbase ip_tunnel bnx2 Feb 10 16:50:19 target-1 kernel: CPU: 12 PID: 25240 Comm: kworker/12:2 Tainted: GF I -------------- 3.10.0-224.el7.x86_64 #1 Feb 10 16:50:19 target-1 kernel: Hardware name: Dell Inc. PowerEdge R710/00NH4P, BIOS 6.3.0 07/24/2012 Feb 10 16:50:19 target-1 kernel: Workqueue: isert_comp_wq isert_do_control_comp [ib_isert] Feb 10 16:50:19 target-1 kernel: ffff8805f8f57d38 00000000f4d510fb ffff8805f8f57cf0 ffffffff81603f46 Feb 10 16:50:19 target-1 kernel: ffff8805f8f57d28 ffffffff8106e28b ffff8800c10b4e00 ffff8800c10b5040 Feb 10 16:50:19 target-1 kernel: 0000000000000000 ffff8800c10b4e40 ffff88031e04aee8 ffff8805f8f57d90 Feb 10 16:50:19 target-1 kernel: Call Trace: Feb 10 16:50:19 target-1 kernel: [<ffffffff81603f46>] dump_stack+0x19/0x1b Feb 10 16:50:19 target-1 kernel: [<ffffffff8106e28b>] warn_slowpath_common+0x6b/0xb0 Feb 10 16:50:19 target-1 kernel: [<ffffffff8106e32c>] warn_slowpath_fmt+0x5c/0x80 Feb 10 16:50:19 target-1 kernel: [<ffffffff812ed571>] __list_del_entry+0xa1/0xd0 Feb 10 16:50:19 target-1 kernel: [<ffffffffa05a757f>] isert_completion_put+0x26f/0x3f0 [ib_isert] Feb 10 16:50:19 target-1 kernel: [<ffffffffa05a77cd>] isert_do_control_comp+0xcd/0x1b0 [ib_isert] Feb 10 16:50:19 target-1 kernel: [<ffffffff8108f0ab>] process_one_work+0x17b/0x470 Feb 10 16:50:19 target-1 kernel: [<ffffffff8108fe8b>] worker_thread+0x11b/0x400 Feb 10 16:50:19 target-1 kernel: [<ffffffff8108fd70>] ? rescuer_thread+0x400/0x400 Feb 10 16:50:19 target-1 kernel: [<ffffffff8109726f>] kthread+0xcf/0xe0 Feb 10 16:50:19 target-1 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140 Feb 10 16:50:19 target-1 kernel: [<ffffffff81613d3c>] ret_from_fork+0x7c/0xb0 Feb 10 16:50:19 target-1 kernel: [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140 Feb 10 16:50:19 target-1 kernel: ---[ end trace 14489862ea1e51d4 ]--- -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
-- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
