Hi Moussa, On Tue, 2014-05-27 at 17:42 +0000, Moussa Ba (moussaba) wrote: > Nick, > > We have encountered the following issues this weekend after having had > 2 days or relative stability. Seems to be another list corruption. I > am pasting 2 separate incidents. > > So I've been trying to reproduce this bug, but haven't had any luck yet.. Can you send along the top-level targetcli output for this particular setup to make sure I'm using a similar number of network portals + endpoints..? Thanks, --nab > sending CHECK_CONDITION. > [186416.854516] ------------[ cut here ]------------ > [186416.854526] WARNING: CPU: 8 PID: 20325 at lib/list_debug.c:53 __list_del_entry+0x63/0xd0() > [186416.854528] list_del corruption, ffff88046dd40c80->next is LIST_POISON1 (dead000000100100) > [186416.854529] Modules linked in: ib_srpt tcm_qla2xxx qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt ib_isert rdma_cm iw_cm ib_addr iscsi_target_mod target_core_pscsi target_core_file target_core_iblock ebtable_nat ebtables ipt_MASQUERADE iptable_nat target_core_mod nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 configfs ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc autofs4 sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ib_ipoib ib_cm ib_uverbs ib_umad mlx4_en mlx4_ib ib_sa ib_mad ib_core mlx4_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm uinput iTCO_wdt iTCO_vendor_support microcode pcspkr serio_raw sb_edac edac_core sg i2c_i801 lpc_ich mfd_core mtip32xx(OF) igb i2c_algo_bit i2c_core ptp pps_core ioatdma dca wmi ext3(F) jbd(F) mbcache(F) sd_mod(F) crc_t10dif(F) crct10dif_common(F) ahci(F) libahci(F) isci(F) libsas(F) scsi_transport_sas(F) [last unloaded: speedstep_lib] > [186416.854589] CPU: 8 PID: 20325 Comm: iscsi_trx Tainted: GF O 3.12.19+ #2 > [186416.854590] Hardware name: Supermicro X9DRX+-F/X9DRX+-F, BIOS 3.00 07/09/2013 > [186416.854591] 0000000000000035 ffff880459121cd8 ffffffff81553ae7 0000000000000035 > [186416.854595] ffff880459121d28 ffff880459121d18 ffffffff8104d1cc ffff88047fc52d40 > [186416.854597] ffff88046dd40c80 ffff88046dd40bc0 ffff880459121e68 ffff8804687e7540 > [186416.854606] Call Trace: > [186416.854611] [<ffffffff81553ae7>] dump_stack+0x49/0x62 > [186416.854615] [<ffffffff8104d1cc>] warn_slowpath_common+0x8c/0xc0 > [186416.854618] [<ffffffff8104d2b6>] warn_slowpath_fmt+0x46/0x50 > [186416.854621] [<ffffffff81280933>] __list_del_entry+0x63/0xd0 > [186416.854623] [<ffffffff812809b1>] list_del+0x11/0x40 > [186416.854633] [<ffffffffa0703cf9>] iscsi_del_ts_from_active_list+0x29/0x50 [iscsi_target_mod] > [186416.854638] [<ffffffffa0704931>] iscsi_rx_thread_pre_handler+0xa1/0x180 [iscsi_target_mod] > [186416.854646] [<ffffffffa071ba1c>] iscsi_target_rx_thread+0x6c/0x300 [iscsi_target_mod] > [186416.854648] [<ffffffff81555d6b>] ? __schedule+0x3db/0x6e0 > [186416.854653] [<ffffffffa071b9b0>] ? iscsi_target_rx_opcode+0x280/0x280 [iscsi_target_mod] > [186416.854658] [<ffffffffa071b9b0>] ? iscsi_target_rx_opcode+0x280/0x280 [iscsi_target_mod] > [186416.854662] [<ffffffff8106f99e>] kthread+0xce/0xe0 > [186416.854664] [<ffffffff8106f8d0>] ? kthread_freezable_should_stop+0x70/0x70 > [186416.854667] [<ffffffff8156026c>] ret_from_fork+0x7c/0xb0 > [186416.854669] [<ffffffff8106f8d0>] ? kthread_freezable_should_stop+0x70/0x70 > [186416.854670] ---[ end trace 9030a82e26ae79e9 ]--- > [186416.854738] ------------[ cut here ]------------ > [186416.854744] WARNING: CPU: 11 PID: 9026 at lib/list_debug.c:33 __list_add+0xbe/0xd0() > [186416.854746] list_add corruption. prev->next should be next (ffffffffa0730f10), but was dead000000100100. (prev=ffff88046dd40c80). > [186416.854747] Modules linked in: ib_srpt tcm_qla2xxx qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt ib_isert rdma_cm iw_cm ib_addr iscsi_target_mod target_core_pscsi target_core_file target_core_iblock ebtable_nat ebtables ipt_MASQUERADE iptable_nat target_core_mod nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 configfs ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc autofs4 sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ib_ipoib ib_cm ib_uverbs ib_umad mlx4_en mlx4_ib ib_sa ib_mad ib_core mlx4_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm uinput iTCO_wdt iTCO_vendor_support microcode pcspkr serio_raw sb_edac edac_core sg i2c_i801 lpc_ich mfd_core mtip32xx(OF) igb i2c_algo_bit i2c_core ptp pps_core ioatdma dca wmi ext3(F) jbd(F) mbcache(F) sd_mod(F) crc_t10dif(F) crct10dif_common(F) ahci(F) libahci(F) isci(F) libsas(F) scsi_transport_sas(F) [last unloaded: speedstep_lib] > [186416.854790] CPU: 11 PID: 9026 Comm: iscsi_np Tainted: GF W O 3.12.19+ #2 > [186416.854791] Hardware name: Supermicro X9DRX+-F/X9DRX+-F, BIOS 3.00 07/09/2013 > [186416.854792] 0000000000000021 ffff88045defdce8 ffffffff81553ae7 0000000000000021 > [186416.854796] ffff88045defdd38 ffff88045defdd28 ffffffff8104d1cc ffff880459b04680 > [186416.854799] ffff88046dd40c80 ffffffffa0730f10 ffff88046dd40c80 ffff88045819f2f8 > [186416.854802] Call Trace: > [186416.854805] [<ffffffff81553ae7>] dump_stack+0x49/0x62 > [186416.854807] [<ffffffff8104d1cc>] warn_slowpath_common+0x8c/0xc0 > [186416.854809] [<ffffffff8104d2b6>] warn_slowpath_fmt+0x46/0x50 > [186416.854812] [<ffffffff81280a9e>] __list_add+0xbe/0xd0 > [186416.854819] [<ffffffffa0704375>] iscsi_activate_thread_set+0x45/0xb0 [iscsi_target_mod] > [186416.854826] [<ffffffffa070b6f3>] iscsi_post_login_handler+0x303/0x4f0 [iscsi_target_mod] > [186416.854832] [<ffffffffa070c5fa>] __iscsi_target_login_thread+0x2ca/0x630 [iscsi_target_mod] > [186416.854837] [<ffffffffa070c960>] ? __iscsi_target_login_thread+0x630/0x630 [iscsi_target_mod] > [186416.854845] [<ffffffffa070c991>] iscsi_target_login_thread+0x31/0x40 [iscsi_target_mod] > [186416.854847] [<ffffffff8106f99e>] kthread+0xce/0xe0 > [186416.854850] [<ffffffff8106f8d0>] ? kthread_freezable_should_stop+0x70/0x70 > [186416.854852] [<ffffffff8156026c>] ret_from_fork+0x7c/0xb0 > [186416.854854] [<ffffffff8106f8d0>] ? kthread_freezable_should_stop+0x70/0x70 > [186416.854855] ---[ end trace 9030a82e26ae79ea ]--- > [186416.854856] > > > ============================================================ > > [ 1480.692884] emulate_write_cache cannot be changed when underlying HW reports WriteCacheEnabled, ignoring request > [32967.442068] ------------[ cut here ]------------ > [32967.442077] WARNING: CPU: 9 PID: 15103 at lib/list_debug.c:53 __list_del_entry+0x63/0xd0() > [32967.442079] list_del corruption, ffff8808659a4f80->next is LIST_POISON1 (dead000000100100) > [32967.442079] Modules linked in: ib_srpt tcm_qla2xxx tcm_loop tcm_fc ib_isert iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod qla2xxx libfc scsi_transport_fc scsi_tgt rdma_cm iw_cm ib_addr ebtable_nat ebtables configfs ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc autofs4 sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ib_ipoib ib_cm ib_uverbs ib_umad mlx4_en mlx4_ib ib_sa ib_mad ib_core mlx4_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm uinput iTCO_wdt iTCO_vendor_support microcode pcspkr serio_raw sg sb_edac edac_core i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core ptp pps_core mtip32xx(OF) ioatdma dca wmi ext3(F) jbd(F) mbcache(F) sd_mod(F) crc_t10dif(F) crct10dif_common(F) ahci(F) libahci(F) isci(F) libsas(F) scsi_transport_sas(F) [last unloaded: target_core_mod] > [32967.442131] CPU: 9 PID: 15103 Comm: iscsi_ttx Tainted: GF O 3.12.19+ #2 > [32967.442132] Hardware name: Supermicro X9DRX+-F/X9DRX+-F, BIOS 3.00 07/09/2013 > [32967.442133] 0000000000000035 ffff8808438b7d08 ffffffff81553ae7 0000000000000035 > [32967.442137] ffff8808438b7d58 ffff8808438b7d48 ffffffff8104d1cc 0000000000000002 > [32967.442139] ffff8808659a4f80 ffff8808659a4ec0 ffff8808659a4ec0 ffff8808438b7e58 > [32967.442141] Call Trace: > [32967.442146] [<ffffffff81553ae7>] dump_stack+0x49/0x62 > [32967.442149] [<ffffffff8104d1cc>] warn_slowpath_common+0x8c/0xc0 > [32967.442152] [<ffffffff8104d2b6>] warn_slowpath_fmt+0x46/0x50 > [32967.442154] [<ffffffff81280933>] __list_del_entry+0x63/0xd0 > [32967.442156] [<ffffffff812809b1>] list_del+0x11/0x40 > [32967.442167] [<ffffffffa087a960>] ? iscsit_handle_snack+0x190/0x190 [iscsi_target_mod] > [32967.442172] [<ffffffffa0866cf9>] iscsi_del_ts_from_active_list+0x29/0x50 [iscsi_target_mod] > [32967.442177] [<ffffffffa08677b1>] iscsi_tx_thread_pre_handler+0xa1/0x180 [iscsi_target_mod] > [32967.442182] [<ffffffffa087a9ae>] iscsi_target_tx_thread+0x4e/0x220 [iscsi_target_mod] > [32967.442187] [<ffffffffa087a960>] ? iscsit_handle_snack+0x190/0x190 [iscsi_target_mod] > [32967.442192] [<ffffffffa087a960>] ? iscsit_handle_snack+0x190/0x190 [iscsi_target_mod] > [32967.442195] [<ffffffff8106f99e>] kthread+0xce/0xe0 > [32967.442197] [<ffffffff8106f8d0>] ? kthread_freezable_should_stop+0x70/0x70 > [32967.442199] [<ffffffff8156026c>] ret_from_fork+0x7c/0xb0 > [32967.442202] [<ffffffff8106f8d0>] ? kthread_freezable_should_stop+0x70/0x70 > [32967.442203] ---[ end trace e021a9776c155b10 ]--- > [32967.442236] ------------[ cut here ]------------ > [32967.442245] WARNING: CPU: 7 PID: 15106 at lib/list_debug.c:33 __list_add+0xbe/0xd0() > [32967.442246] list_add corruption. prev->next should be next (ffffffffa0893f00), but was ffffffffa0893f10. (prev=ffff8808659a4f80). > [32967.442248] Modules linked in: ib_srpt tcm_qla2xxx tcm_loop tcm_fc ib_isert iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod qla2xxx libfc scsi_transport_fc scsi_tgt rdma_cm iw_cm ib_addr ebtable_nat ebtables configfs ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc autofs4 sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ib_ipoib ib_cm ib_uverbs ib_umad mlx4_en mlx4_ib ib_sa ib_mad ib_core mlx4_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm uinput iTCO_wdt iTCO_vendor_support microcode pcspkr serio_raw sg sb_edac edac_core i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core ptp pps_core mtip32xx(OF) ioatdma dca wmi ext3(F) jbd(F) mbcache(F) sd_mod(F) crc_t10dif(F) crct10dif_common(F) ahci(F) libahci(F) isci(F) libsas(F) scsi_transport_sas(F) [last unloaded: target_core_mod] > [32967.442295] CPU: 7 PID: 15106 Comm: iscsi_trx Tainted: GF W O 3.12.19+ #2 > [32967.442296] Hardware name: Supermicro X9DRX+-F/X9DRX+-F, BIOS 3.00 07/09/2013 > [32967.442298] 0000000000000021 ffff8808438bdc98 ffffffff81553ae7 0000000000000021 > [32967.442301] ffff8808438bdce8 ffff8808438bdcd8 ffffffff8104d1cc ffff8808438bdcc8 > [32967.442303] ffff8808659a4180 ffffffffa0893f00 ffff8808659a4f80 ffff88086d0c6ae0 > [32967.442306] Call Trace: > [32967.442310] [<ffffffff81553ae7>] dump_stack+0x49/0x62 > [32967.442314] [<ffffffff8104d1cc>] warn_slowpath_common+0x8c/0xc0 > [32967.442316] [<ffffffff8104d2b6>] warn_slowpath_fmt+0x46/0x50 > [32967.442319] [<ffffffff81280a9e>] __list_add+0xbe/0xd0 > [32967.442326] [<ffffffffa0866dc7>] iscsi_add_ts_to_inactive_list+0x37/0x60 [iscsi_target_mod] > [32967.442331] [<ffffffffa0867621>] iscsi_allocate_thread_sets+0x1d1/0x280 [iscsi_target_mod] > [32967.442336] [<ffffffffa086770a>] iscsi_check_to_add_additional_sets+0x3a/0x40 [iscsi_target_mod] > [32967.442341] [<ffffffffa0867995>] iscsi_rx_thread_pre_handler+0x105/0x180 [iscsi_target_mod] > [32967.442347] [<ffffffffa087ea1c>] iscsi_target_rx_thread+0x6c/0x300 [iscsi_target_mod] > [32967.442349] [<ffffffff81555d6b>] ? __schedule+0x3db/0x6e0 > [32967.442354] [<ffffffffa087e9b0>] ? iscsi_target_rx_opcode+0x280/0x280 [iscsi_target_mod] > [32967.442359] [<ffffffffa087e9b0>] ? iscsi_target_rx_opcode+0x280/0x280 [iscsi_target_mod] > [32967.442362] [<ffffffff8106f99e>] kthread+0xce/0xe0 > [32967.442365] [<ffffffff8106f8d0>] ? kthread_freezable_should_stop+0x70/0x70 > [32967.442367] [<ffffffff8156026c>] ret_from_fork+0x7c/0xb0 > [32967.442369] [<ffffffff8106f8d0>] ? kthread_freezable_should_stop+0x70/0x70 > [32967.442371] ---[ end trace e021a9776c155b11 ]--- > [32967.444411] ------------[ cut here ]------------ > [32967.444415] WARNING: CPU: 7 PID: 10513 at lib/list_debug.c:33 __list_add+0xbe/0xd0() > [32967.444416] list_add corruption. prev->next should be next (ffffffffa0893f10), but was dead000000100100. (prev=ffff8808659a4f80). > [32967.444417] Modules linked in: ib_srpt tcm_qla2xxx tcm_loop tcm_fc ib_isert iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod qla2xxx libfc scsi_transport_fc scsi_tgt rdma_cm iw_cm ib_addr ebtable_nat ebtables configfs ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc autofs4 sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ib_ipoib ib_cm ib_uverbs ib_umad mlx4_en mlx4_ib ib_sa ib_mad ib_core mlx4_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm uinput iTCO_wdt iTCO_vendor_support microcode pcspkr serio_raw sg sb_edac edac_core i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core ptp pps_core mtip32xx(OF) ioatdma dca wmi ext3(F) jbd(F) mbcache(F) sd_mod(F) crc_t10dif(F) crct10dif_common(F) ahci(F) libahci(F) isci(F) libsas(F) scsi_transport_sas(F) [last unloaded: target_core_mod] > [32967.444455] CPU: 7 PID: 10513 Comm: iscsi_np Tainted: GF W O 3.12.19+ #2 > [32967.444457] Hardware name: Supermicro X9DRX+-F/X9DRX+-F, BIOS 3.00 07/09/2013 > [32967.444457] 0000000000000021 ffff8808492dbce8 ffffffff81553ae7 0000000000000021 > [32967.444460] ffff8808492dbd38 ffff8808492dbd28 ffffffff8104d1cc ffff88086e4b59c0 > [32967.444462] ffff8808659a4f80 ffffffffa0893f10 ffff8808659a4f80 ffff8808561962f8 > [32967.444464] Call Trace: > [32967.444466] [<ffffffff81553ae7>] dump_stack+0x49/0x62 > [32967.444468] [<ffffffff8104d1cc>] warn_slowpath_common+0x8c/0xc0 > [32967.444472] [<ffffffff8104d2b6>] warn_slowpath_fmt+0x46/0x50 > [32967.444474] [<ffffffff81280a9e>] __list_add+0xbe/0xd0 > [32967.444479] [<ffffffffa0867375>] iscsi_activate_thread_set+0x45/0xb0 [iscsi_target_mod] > [32967.444484] [<ffffffffa086e6f3>] iscsi_post_login_handler+0x303/0x4f0 [iscsi_target_mod] > [32967.444490] [<ffffffffa086f5fa>] __iscsi_target_login_thread+0x2ca/0x630 [iscsi_target_mod] > [32967.444495] [<ffffffffa086f960>] ? __iscsi_target_login_thread+0x630/0x630 [iscsi_target_mod] > [32967.444501] [<ffffffffa086f991>] iscsi_target_login_thread+0x31/0x40 [iscsi_target_mod] > [32967.444504] [<ffffffff8106f99e>] kthread+0xce/0xe0 > [32967.444511] [<ffffffff8106f8d0>] ? kthread_freezable_should_stop+0x70/0x70 > [32967.444522] [<ffffffff8156026c>] ret_from_fork+0x7c/0xb0 > [32967.444531] [<ffffffff8106f8d0>] ? kthread_freezable_should_stop+0x70/0x70 > [32967.444535] ---[ end trace e021a9776c155b12 ]--- > [32967.444536] ------------[ cut here ]------------ > [32967.444538] WARNING: CPU: 7 PID: 10513 at lib/list_debug.c:36 __list_add+0x9c/0xd0() > [32967.444539] list_add double add: new=ffff8808659a4f80, prev=ffff8808659a4f80, next=ffffffffa0893f10. > [32967.444539] Modules linked in: ib_srpt tcm_qla2xxx tcm_loop tcm_fc ib_isert iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod qla2xxx libfc scsi_transport_fc scsi_tgt rdma_cm iw_cm ib_addr ebtable_nat ebtables configfs ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc autofs4 sunrpc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ib_ipoib ib_cm ib_uverbs ib_umad mlx4_en mlx4_ib ib_sa ib_mad ib_core mlx4_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan vhost tun kvm_intel kvm uinput iTCO_wdt iTCO_vendor_support microcode pcspkr serio_raw sg sb_edac edac_core i2c_i801 lpc_ich mfd_core igb i2c_algo_bit i2c_core ptp pps_core mtip32xx(OF) ioatdma dca wmi ext3(F) jbd(F) mbcache(F) sd_mod(F) crc_t10dif(F) crct10dif_common(F) ahci(F) libahci(F) isci(F) libsas(F) scsi_transport_sas(F) [last unloaded: target_core_mod] > [32967.444573] CPU: 7 PID: 10513 Comm: iscsi_np Tainted: GF W O 3.12.19+ #2 > [32967.444574] Hardware name: Supermicro X9DRX+-F/X9DRX+-F, BIOS 3.00 07/09/2013 -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
