On 04/06/2017 08:46 AM, David Disseldorp wrote: > On Thu, 6 Apr 2017 14:27:01 +0100, Nick Fisk wrote: > ... >>> I'm not to sure what you're referring to WRT the spiral of death, but we did >>> patch some LIO issues encountered when a command was aborted while >>> outstanding at the LIO backstore layer. >>> These specific fixes are carried in the mainline kernel, and can be tested >>> using the AbortTaskSimpleAsync libiscsi test. >> >> Awesome, glad this has finally been fixed. Death spiral was referring to when using it with ESXi, both the initiator and target effectively hang forever and if you didn't catch it soon enough, sometimes you end up having to kill all vm's and reboot hosts. > > Sounds like it could be the same thing. Stale iSCSI sessions remain > around which block subsequent login attempts. > >> Do you know what kernel version these changes would have first gone into? I thought I looked back into this last summer and it was still showing the same behavior. > > The fix I was referring to is: > commit 5e2c956b8aa24d4f33ff7afef92d409eed164746 > Author: Nicholas Bellinger <nab@xxxxxxxxxxxxxxx> > Date: Wed May 25 12:25:04 2016 -0700 > > target: Fix missing complete during ABORT_TASK + CMD_T_FABRIC_STOP > > It's carried in v4.8+ and was also flagged for 3.14+ stable inclusion, > so should be present in many distro kernels by now. That said, there > have been many other changes in this area. > I think we can still hit the issue with this patch. The general problem is handling commands that are going to take longer than the initiator side's error handler. ESX will end up marking the VM/storage as failed and the user has to manually intervene. It is similar to linux where a /dev/sdX is marked offline, and the user has to then manually online it and restart layers above it. So we should root cause the reason for commands taking so long. If it is just a normal case, then to handle this issue in a more generic way for all initiators, Nick suggested to implement a target side timeout: https://www.spinics.net/lists/target-devel/msg14780.html In tcmu-runner we could then abort/kill the command based on a timer there and then fail the command before the ESX timers fire. The difficult part is of course aborting a running rbd command. Note that you can currently set the tcmu timeout: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/target/target_core_user.c?id=7d7a743543905a8297dce53b36e793e5307da5d7 discussed in that thread and you will avoid the problem, but there is no code to stop the running command in tcmu-runner, so it would not be safe in some setups. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |