On Fri, Feb 12, 2016 at 12:30 AM, Himanshu Madhani <himanshu.madhani@xxxxxxxxxx> wrote: > Hi Nic, > > > > On 2/11/16, 3:47 PM, "Nicholas A. Bellinger" <nab@xxxxxxxxxxxxxxx> wrote: > >>On Wed, 2016-02-10 at 22:53 -0800, Nicholas A. Bellinger wrote: >>> On Tue, 2016-02-09 at 18:03 +0000, Himanshu Madhani wrote: >>> > On 2/8/16, 9:25 PM, "Nicholas A. Bellinger" <nab@xxxxxxxxxxxxxxx> >>>wrote: >>> > >On Mon, 2016-02-08 at 23:27 +0000, Himanshu Madhani wrote: >>> > >> >>> > >> I am testing this series with with 4.5.0-rc2+ kernel and I am >>>seeing >>> > >>issue >>> > >> where trying to trigger >>> > >> sg_reset with option of host/device/bus in loop at 120second >>>interval >>> > >> causes call stack. At this point >>> > >> removing configuration hangs indefinitely. See attached dmesg >>>output >>> > >>from >>> > >> my setup. >>> > >> >>> > > >>> > >Thanks alot for testing this. >>> > > >>> > >So It looks like we're still hitting a indefinite schedule() on >>> > >se_cmd->cmd_wait_comp once tcm_qla2xxx session disconnect/reconnect >>> > >occurs, after repeated explicit active I/O remote-port sg_resets. >>> > > >>> > >Does this trigger on the first tcm_qla2xxx session reconnect after >>> > >explicit remote-port sg_reset..? Are session reconnects actively >>>being >>> > >triggered during the test..? >>> > > >>> > >To verify the latter for iscsi-target, I've been using a small patch >>>to >>> > >trigger session reset from TMR kthread context in order to simulate >>>the >>> > >I_T disconnects. Something like that would be useful for verifying >>>with >>> > >tcm_qla2xxx too. >>> > > >>> > >That said, I'll be reproducing with tcm_qla2xxx ports this week, and >>> > >will enable various debug in a WIP branch for testing. >>> >>> Following up here.. >>> >>> So far using my test setup with ISP2532 ports in P2P + RAMDISK_MCP and >>> v4.5-rc1, repeated remote-port active I/O LUN_RESET (sg_reset -d) has >>> been functioning as expected with a blocksize_range=4k-256k + iodepth=32 >>> fio write-verify style workload. >>> >>> No ->cmd_kref -1 OOPsen or qla2xxx initiator generated ABORT_TASKs from >>> outstanding target TAS responses, nor fio write-verify failures to >>> report after 800x remote-port active I/O LUN_RESETS. >>> >>> Next step will be to verify explicit tcm_qla2xxx port + module shutdown >>> after 1K test iterations, and then IBLOCK async completions <-> NVMe >>> backends with the same case. >>> >> >>After letting this test run over-night up to 7k active I/O remote-port >>LUN_RESETs, things are still functioning as expected. >> >>Also, /etc/init.d/target stop was able to successfully shutdown all >>active sessions and unload tcm_qla2xxx after the test run. >> >>So AFAICT, the active I/O remote-port LUN_RESET changes are stable with >>tcm_qla2xxx ports, separate from concurrent session disconnect hung task >>you reported earlier. >> >>That said, I'll likely push this series as-is for -rc4, given that Dan >>has also been able to verify the non conncurrent session disconnect case >>on his setup generating constant ABORT_TASKs, and it's still surviving >>both cases for iscsi-target ports. >> >>Please give the debug patch from last night a shot, and see if we can >>determine the se_cmd states when you hit the hung task. > > I¹ll give your latest debug patch try in a little while > > From the testing that I have done, what is seen is that active IO has > already been completed and qla2xxx driver is waiting for commands to be > Completed and it¹s waiting indefinitely for cmd_wait_comp. > So it looks like there is a missing complete call from target_core. I¹ve > attached our analysis from crash debug on a live system after the issues > happens. > > > I can recreate this issue at will within 5 minute of triggering sg_reset > with following steps > > 1. Export 4 RAM disk LUNs on each of 2 port adapter. Initiator will see 8 > RAM disk targets > 2. Start IO with 4K block size and 8 threads with 80% write 20% read and > 100% dandom. > (I am using vdbench for generating IO. I can provide setup/config script > if needed) > 3. Start sg_reset for each LUNs with first device, bus and host with 120s > delay. (I¹ve attached > My script that I am using for triggering sg_reset) > >> >>Thank you, >> >>-nab >> > Has there been any update to this? Thanks -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
