On 12/07/13 00:18, Nicholas A. Bellinger wrote:
Hi Benjamin,
Hi Nicholas,
First of all, very sorry for the delay in replying. Answers are below.
On Thu, 2013-07-11 at 18:41 +0100, Benjamin ESTRABAUD wrote:
Hi!
I've come across a strange kernel panic issue on LIO on Linux 3.6.11
(b2824f4e0990716407b0c0e7acee75bb6353febf). The issue seems to be linked
with late IOs calling back to an already deleted target and loops here
until the kernel panics.
The test setup was the following:
- Exporting more than one IBLOCK (here two MD RAIDs, but seems to do the
same with any block devices) over iSCSI.
- Running intensive IOs (64 ios depth, 548k ios, 100% write, 100%
random) using IO meter from a single fast host
While the above IOs are running, when "deleting" the iSCSI targets using
rtslib, and by intermittence, the kernel panicked.
Please provide the specific rtslib calls required in order to trigger
this bug.
The issue occurs when calling the "delete" function from a valid
target.TPG instance.
The issue can also be triggered by "manually" removing the relevant
iSCSI target configFS directory (/sys/kernel/config/target/iscsi/<iqn>)
Also, a quick dump of your top-level targetcli object tree would also be
helpful for reference.
Unfortunately we don't use targetcli in this system, only "rtslib".
The issue is quite intermittent and might happen about 1 out of 20 tests
or less.
We have not yet tried to reproduce the issue on Ramdisk or file
backstores but it seems that this could be linked to the async and slow
nature of the iblock backstores. In fact, the issue usually happens
always after 100ms of tearing off the target, which could indicate that
a BIO comes back and causes the issue.
FYI, the backend type should not make any difference here.
Noted.
Attached is the kernel trace we get when running this test.
We are going to try to rollback our kernel/update but we were hoping to
stick on the 3.6.y branch for now.
Have you seen this issue before? Do you know if it has been resolved in
a later version?
Mmmmm, this smells very much like this v3.6.x specific bug:
target: Fix missing CMD_T_ACTIVE bit regression for pending WRITEs
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/drivers/target/target_core_transport.c?id=e627c615553a356f6f70215ebb3933c6e057553e
Because the bugfix was merged after v3.6.y stable support was
discontinued, v3.6.11 does *not* contain this fix.
Also, two other patches in the same bugfix series are not present in
v3.6.11 code:
target: Fix use-after-free in LUN RESET handling
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/drivers/target/target_core_transport.c?id=72b59d6ee8adaa51f70377db0a1917ed489bead8
and
target: Release se_cmd when LUN lookup fails for TMR
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/drivers/target/target_core_transport.c?id=5a3b6fc0092c5f8dee7820064ee54d2631d48573
So that said, I'd recommend applying these three patches to your local
v3.6.11 tree, or consider moving to >= v3.7.10.
Brilliant!
Thanks a million for this information. We are currently trying to move
to 3.9.y or 3.10.y (the two current -stable branches) but are having a
different issue loading the loopback and iSCSI fabric in configFS. I'll
send a separate e-mail for this issue so as not to mix threads up.
Thanks,
Thanks a lot for your help.
Regards,
Ben.
--nab
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe target-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
