Ian,
--On 24 January 2013 10:42:11 +0000 Ian Campbell <Ian.Campbell@xxxxxxxxxx>
wrote:
This is exactly what can happen:
1. send request (A)
2. timeout waiting for ACK to (A)
3. queue TCP retransmit of (A) as (B)
4. receive ACK to original (A), sent at #1, and rpc reply to that
request.
5. return success to userspace
6. userspace reuses (or unmaps under Xen) the buffer
7. (B), queued at #3, reaches the head of the queue
8. Try to transmit (B), bug has now happened.
You can also s/TCP/RPC/ and construct a similar issue at the next layer
of the stack, which only happens on NFSv3 AIUI.
Got it - finally! Thanks for your patience in explaining.
I am guessing a simpler fix for the tcp retransmit problem would be
to copy (or optionally copy) the page(s) for B at step 3. Given tcp
retransmit is infrequent and performance is not going to be good
with tcp retransmissions going on anyway, that might be acceptable.
However in practice *anything* that causes a multiple references
to a page in the networking stack is going to have this problem,
and multiple skbuff's can refer to the same page, which is I presume
why you were fixing this by skbuff reference counting, presumably
so you know you can do (5) only when the skbuff is entirely unreferenced
(i.e. after (8)).
My understanding (which may well be completely wrong) is that the problem
was that xen was unmapping the page even though it still had kernel
references to it. This is why the problem does not happen in kvm (which
does not as I understand it do a similar map/unmap operation). From Ian
C I understand that just looking at the number of kernel references is
not sufficient.
Under any userspace process (which includes KVM) you get retransmission
of data which may have changed, because userspace believes the kernel
when it has said it is done with it, and has reused the buffer. All that
is different under Xen is that "changed" can mean "unmapped" which makes
the symptom much worse.
Indeed. And the fact that kvm by default does not use O_DIRECT whereas
xen_disk.c does, so kvm will hide the problem as a copy is performed in (1).
--
Alex Bligh
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
