Hey Nic,
So Our QA guys recently stepped on this bug when performing stress
login-logout from a single initiator to 10 targets each exposed over
4 portals, so overall 40 sessions (needless to say we are talking on
iser...). So there are lots of logins in parallel with lots of logouts.
It seems that the connection termination causes iscsi_tx_thread to
access the connection after it is freed or something (list corruption
probably coming from iscsit_handle_immediate_queue or
iscsit_handle_response_queue, and NULL deref coming from
iscsit_take_action_for_connection_exit).
Note, isert_wait_conn waits for session commands and QP flush which is
normally pretty fast, the conn termination is done in a work that waits
for DISCONNECTED event which might take longer (which is why we do it
outside wait_conn context to avoid blocking it).
I didn't get too far with this until now, do you have any idea on what
might have happened?
Trace:
Jan 7 00:21:25 r-host83 kernel: ------------[ cut here ]------------
Jan 7 00:21:25 r-host83 kernel: isert: isert_wait_conn: Starting conn
ffff8802e595e000
Jan 7 00:21:25 r-host83 kernel: isert: isert_wait4cmds: iscsi_conn
ffff8802ebc8d000
Jan 7 00:21:25 r-host83 kernel: WARNING: CPU: 2 PID: 5280 at
lib/list_debug.c:59 __list_del_entry+0xa1/0xd0()
Jan 7 00:21:25 r-host83 kernel: list_del corruption. prev->next should
be ffff880601d6fe70, but was (null)
Jan 7 00:21:25 r-host83 kernel: Modules linked in: target_core_pscsi(O)
target_core_file(O) target_core_iblock(O) ib_isert(O)
iscsi_target_mod(O) ib_srpt(O) tcm_loop(O) tcm_fc(O) target_core_mod(O)
mst_pciconf(OE) bonding mlx5_ib(O) mlx5_core libfc scsi_transport_fc
netconsole configfs nfsv3 nfs_acl mlx4_ib(O) rdma_ucm(O) ib_ucm(O)
rdma_cm(O) iw_cm(O) ib_uverbs(O) libiscsi_tcp libiscsi
scsi_transport_iscsi mlx4_en mlx4_core ib_ipoib(O) ib_cm(O) ib_sa(O)
ib_umad(O) ib_mad(O) ib_core(O) ib_addr(O) rpcsec_gss_krb5 auth_rpcgss
nfsv4 nfs fscache lockd grace autofs4 sunrpc 8021q garp stp llc ipv6
dm_mirror dm_region_hash dm_log dm_multipath uinput ipmi_si
ipmi_msghandler acpi_pad iTCO_wdt iTCO_vendor_support dcdbas microcode
pcspkr wmi sb_edac edac_core sg lpc_ich mfd_core shpchp tg3 ptp pps_core
dm_mod ext3(E) jbd(E) mbcache(E) sr_mod(E) cdrom(E) sd_mod(E) ahci(E)
libahci(E) megaraid_sas(E) [last unloaded: target_core_mod]
Jan 7 00:21:25 r-host83 kernel: CPU: 2 PID: 5280 Comm: iscsi_ttx
Tainted: G W OE 3.18.0-rc2+ #22
Jan 7 00:21:25 r-host83 kernel: Hardware name: Dell Inc. PowerEdge
R720/0VWT90, BIOS 2.0.9 03/08/2013
Jan 7 00:21:25 r-host83 kernel: 000000000000003b ffff880601d6fd48
ffffffff8159c9e5 000000000000003b
Jan 7 00:21:25 r-host83 kernel: ffff880601d6fd98 ffff880601d6fd88
ffffffff810538ec ffff880601d6fda8
Jan 7 00:21:25 r-host83 kernel: ffff8805dc437800 ffff880601d6fe58
ffff880601d6fe70 0000000000000212
Jan 7 00:21:25 r-host83 kernel: Call Trace:
Jan 7 00:21:25 r-host83 kernel: [<ffffffff8159c9e5>] dump_stack+0x49/0x5c
Jan 7 00:21:25 r-host83 kernel: [<ffffffff810538ec>]
warn_slowpath_common+0x8c/0xc0
Jan 7 00:21:25 r-host83 kernel: [<ffffffff810539d6>]
warn_slowpath_fmt+0x46/0x50
Jan 7 00:21:25 r-host83 kernel: [<ffffffff812acea1>]
__list_del_entry+0xa1/0xd0
Jan 7 00:21:25 r-host83 kernel: [<ffffffff8108e9d3>] finish_wait+0x53/0x90
Jan 7 00:21:25 r-host83 kernel: [<ffffffffa034eccb>]
iscsi_target_tx_thread+0xeb/0x220 [iscsi_target_mod]
Jan 7 00:21:25 r-host83 kernel: [<ffffffff8108e7f0>] ?
bit_waitqueue+0xb0/0xb0
Jan 7 00:21:25 r-host83 kernel: [<ffffffffa034ebe0>] ?
iscsit_handle_snack+0x190/0x190 [iscsi_target_mod]
Jan 7 00:21:25 r-host83 kernel: [<ffffffff8107017e>] kthread+0xce/0xf0
Jan 7 00:21:25 r-host83 kernel: [<ffffffff810700b0>] ?
kthread_freezable_should_stop+0x70/0x70
Jan 7 00:21:25 r-host83 kernel: [<ffffffff815a0b6c>]
ret_from_fork+0x7c/0xb0
Jan 7 00:21:25 r-host83 kernel: [<ffffffff810700b0>] ?
kthread_freezable_should_stop+0x70/0x70
Jan 7 00:21:25 r-host83 kernel: ---[ end trace a0c33436cd0836b3 ]---
Jan 7 00:21:26 r-host83 kernel: BUG: unable to handle kernel NULL
pointer dereference at 00000000000001f0
Jan 7 00:21:26 r-host83 kernel: IP: [<ffffffffa033d992>]
iscsit_take_action_for_connection_exit+0x62/0x110 [iscsi_target_mod]
Jan 7 00:21:26 r-host83 kernel: PGD 0
Jan 7 00:21:26 r-host83 kernel: Oops: 0000 [#1] SMP
Jan 7 00:21:26 r-host83 kernel: Modules linked in: target_core_pscsi(O)
target_core_file(O) target_core_iblock(O) ib_isert(O)
iscsi_target_mod(O) ib_srpt(O) tcm_loop(O) tcm_fc(O) target_core_mod(O)
mst_pciconf(OE) bonding mlx5_ib(O) mlx5_core libfc scsi_transport_fc
netconsole configfs nfsv3 nfs_acl mlx4_ib(O) rdma_ucm(O) ib_ucm(O)
rdma_cm(O) iw_cm(O) ib_uverbs(O) libiscsi_tcp libiscsi
scsi_transport_iscsi mlx4_en mlx4_core ib_ipoib(O) ib_cm(O) ib_sa(O)
ib_umad(O) ib_mad(O) ib_core(O) ib_addr(O) rpcsec_gss_krb5 auth_rpcgss
nfsv4 nfs fscache lockd grace autofs4 sunrpc 8021q garp stp llc ipv6
dm_mirror dm_region_hash dm_log dm_multipath uinput ipmi_si
ipmi_msghandler acpi_pad iTCO_wdt iTCO_vendor_support dcdbas microcode
pcspkr wmi sb_edac edac_core sg lpc_ich mfd_core shpchp tg3 ptp pps_core
dm_mod ext3(E) jbd(E) mbcache(E) sr_mod(E) cdrom(E) sd_mod(E) ahci(E)
libahci(E) megaraid_sas(E) [last unloaded: target_core_mod]
Jan 7 00:21:26 r-host83 kernel: CPU: 2 PID: 5280 Comm: iscsi_ttx
Tainted: G W OE 3.18.0-rc2+ #22
Jan 7 00:21:26 r-host83 kernel: Hardware name: Dell Inc. PowerEdge
R720/0VWT90, BIOS 2.0.9 03/08/2013
Jan 7 00:21:26 r-host83 kernel: task: ffff8806132f9010 ti:
ffff880601d6c000 task.ti: ffff880601d6c000
Jan 7 00:21:26 r-host83 kernel: RIP: 0010:[<ffffffffa033d992>]
[<ffffffffa033d992>] iscsit_take_action_for_connection_exit+0x62/0x110
[iscsi_target_mod]
Jan 7 00:21:26 r-host83 kernel: RSP: 0018:ffff880601d6fe18 EFLAGS:
00010296
Jan 7 00:21:26 r-host83 kernel: RAX: 0000000000000000 RBX:
ffff8805dc437800 RCX: 0000000000000006
Jan 7 00:21:26 r-host83 kernel: RDX: 0000000000000000 RSI:
0000000000000200 RDI: ffffffffa033d98b
Jan 7 00:21:26 r-host83 kernel: RBP: ffff880601d6fe28 R08:
0000000000000000 R09: 000000000000dd37
Jan 7 00:21:26 r-host83 kernel: R10: 00000000ec5d4202 R11:
0000000000000001 R12: ffff8805dc437bf4
Jan 7 00:21:26 r-host83 kernel: R13: ffff88061b831600 R14:
ffff880601d6fe58 R15: ffff8806132f9010
Jan 7 00:21:26 r-host83 kernel: FS: 0000000000000000(0000)
GS:ffff88032fa20000(0000) knlGS:0000000000000000
Jan 7 00:21:26 r-host83 kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Jan 7 00:21:26 r-host83 kernel: CR2: 00000000000001f0 CR3:
0000000001a14000 CR4: 00000000000407e0
Jan 7 00:21:26 r-host83 kernel: Stack:
Jan 7 00:21:26 r-host83 kernel: ffff8805dc437800 fffffffffffffe00
ffff880601d6feb8 ffffffffa034ed40
Jan 7 00:21:26 r-host83 kernel: ffff8806132f9010 ffff880601d6fe70
0f00000000000000 ffff8805d51fbef0
Jan 7 00:21:26 r-host83 kernel: 0000000000000000 ffff8806132f9010
ffffffff8108e7f0 ffff880601d6fe70
Jan 7 00:21:26 r-host83 kernel: Call Trace:
Jan 7 00:21:26 r-host83 kernel: [<ffffffffa034ed40>]
iscsi_target_tx_thread+0x160/0x220 [iscsi_target_mod]
Jan 7 00:21:26 r-host83 kernel: [<ffffffff8108e7f0>] ?
bit_waitqueue+0xb0/0xb0
Jan 7 00:21:26 r-host83 kernel: [<ffffffffa034ebe0>] ?
iscsit_handle_snack+0x190/0x190 [iscsi_target_mod]
Jan 7 00:21:26 r-host83 kernel: [<ffffffff8107017e>] kthread+0xce/0xf0
Jan 7 00:21:26 r-host83 kernel: [<ffffffff810700b0>] ?
kthread_freezable_should_stop+0x70/0x70
Jan 7 00:21:26 r-host83 kernel: [<ffffffff815a0b6c>]
ret_from_fork+0x7c/0xb0
Jan 7 00:21:26 r-host83 kernel: [<ffffffff810700b0>] ?
kthread_freezable_should_stop+0x70/0x70
Jan 7 00:21:26 r-host83 kernel: Code: 06 0f 84 82 00 00 00 3c 08 74 4e
f6 05 39 e6 02 00 04 0f 85 9e 00 00 00 c6 43 19 08 4c 89 e7 e8 65 2a 26
e1 48 8b 83 a0 04 00 00 <48> 8b 88 f0 01 00 00 80 b9 d8 04 00 00 02 74
2e f6 05 31 e6 02
Jan 7 00:21:26 r-host83 kernel: RIP [<ffffffffa033d992>]
iscsit_take_action_for_connection_exit+0x62/0x110 [iscsi_target_mod]
Jan 7 00:21:26 r-host83 kernel: RSP <ffff880601d6fe18>
Jan 7 00:21:26 r-host83 kernel: CR2: 00000000000001f0
Jan 7 00:21:26 r-host83 kernel: ---[ end trace a0c33436cd0836b4 ]---
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
