On Mon, Jan 23, 2017 at 3:20 PM, Sagi Grimberg <sagi@xxxxxxxxxxx> wrote:
Hi Robert,
Seeing this, makes me realize that the entire "iscsi_trx going
into D state" thread which I did not bother to read is actually an
iser-target related bug. I'm really sorry for not addressing this
sooner (much sooner).
I was hesitant to start a new thread and fragment the discussion and
cause confusion. I thought by presenting my findings as an RFC with
some code, I might get some new ideas.
The patch looks wrong to me, but lets try and talk about
the hang you are trying to solve.
In certain circumstances the RDMA connection can be abruptly
terminated,
Does it make a difference which port is causing the abruption?
Is it the target switch port? or the initiator switch port?
It seems more tied to initiator port, but previously with our target
export scripts it also seems that a target crash and re-export
(generally out of order) could also cause the issue. My theory is that
the session didn't mach enough and because of that would not get torn
down completely. It is more of speculation from observations and
thought than anything concrete.
but something is getting stuck preventing the iSCSI clean
up commands from being completed.
I think this means, that at least one command was missing
a final kref_put and causing target_wait_for_sess_cmds() to
block forever.
Just removing the isert_wait4*
commands isn't enough.
Yes, because all the inflight IO needs to be properly cleaned up
for the session to terminate gracefully.
Just resetting the queue pair isn't enough
either.
This is true as well. Before tearing down the RDMA queue pair we
need to make sure we will never see a completion for it on its
corresponding completion queue. This is why ib_drain_qp exists.
some help getting this patch fixed right as resetting the queue pair
is probably not the right approach and overkill to solving the
problem. I think it at least shows where the problem is occurring and
how I can get around it.
As said, I have a feeling that we have a flow we are missing the last
kref_put on (at least) one of the session commands. The fact that this
involves port toggling, probably boils down to error completions.
Bart, I recall you had a patch at some point to periodically print
out the hanging session commands in target_wait_for_sess_cmds(), do we
want to get it in? I think we can all benefit from it.
Do you have a link to the path and I can add it in?
Would it be possible to turn on isert debug_level=4 and send us the log?
$ echo 4 > /sys/module/ib_isert/parameters/debug_level
I'll get that to you tomorrow.
The problem easily shows up with two ConnectX-4-LX card connected to a
10 Gb switch. The target is a RAM disk and the initiator just mounts
it as ext4 and runs fio.
Can you please share the fio workload? Does this happen when for example
you run 100% read workload? or 100% write workload?
echo "3" > /proc/sys/vm/drop_caches; fio --rw=read --bs=4K --size=1G
--numjobs=40 --name=worker.matt --group_reporting
With the ConnectX-4-LX cards, it will usually cause the issue while
laying out the first four files, usually on the first one or two. When
I was able to replicate it on IB, I believe I was able to replicate by
pulling the cable on lay out or during the read portion of the test.
So I don't think it matters what the workload is, but I think it needs
a workload of some sort.
And, can you try and disable the unsolicited-data-out in the target
(IIRC its InitialR2T=Yes)? unsol dataout has been known to trigger
cause similar hangs before (which were supposed to be solved).
I'll figure out how to do this and test it tomorrow and let you know
the results.
Also, can you please summarize what kernel versions do you see this
with? The previous thread is a bit hard to follow at once.
4.9, 4.4.x, 4.1.x (we believe we saw it here, but it has been a long
time since we have run this version.)
Thanks for the response Sagi, I'm happy to have new things to try, I
was really lost at where to go next.
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
