Hi folks,
I saw this old thread and wondered if the problem I am seeing is related:
Re: 2.6.15.1: UDP fragments >27208 bytes lost with ne2k-pci on DP83815
For some reason, even with this box specifying rsize=8192, I am seeing a
UDP NFS mount request a read of 32k and the other end is trying to send a
32k fragmented UDP response. Here is a tcpdump from the receiving side:
00:50:42.872174 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], length: 136) CLIENT.1585774366 > SERVER.2049: 108 read fh Unknown/0100000100090003020000000A8002009C6E3C2C000000000000000000008000 32768 bytes @ 0x0
00:50:42.872548 IP (tos 0x0, ttl 64, id 1633, offset 0, flags [+], length: 1500) SERVER.2049 > CLIENT.1585774366: reply ok 1472 read REG 100644 ids 0/0 sz 0x1228c 32768 bytes
00:50:42.872669 IP (tos 0x0, ttl 64, id 1633, offset 1480, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.872794 IP (tos 0x0, ttl 64, id 1633, offset 2960, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.872916 IP (tos 0x0, ttl 64, id 1633, offset 4440, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.873042 IP (tos 0x0, ttl 64, id 1633, offset 5920, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.873160 IP (tos 0x0, ttl 64, id 1633, offset 7400, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.873283 IP (tos 0x0, ttl 64, id 1633, offset 8880, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.873406 IP (tos 0x0, ttl 64, id 1633, offset 10360, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.873529 IP (tos 0x0, ttl 64, id 1633, offset 11840, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.873655 IP (tos 0x0, ttl 64, id 1633, offset 13320, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.873775 IP (tos 0x0, ttl 64, id 1633, offset 14800, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.873899 IP (tos 0x0, ttl 64, id 1633, offset 16280, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.874022 IP (tos 0x0, ttl 64, id 1633, offset 17760, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.874145 IP (tos 0x0, ttl 64, id 1633, offset 19240, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.874268 IP (tos 0x0, ttl 64, id 1633, offset 20720, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.874390 IP (tos 0x0, ttl 64, id 1633, offset 22200, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.874515 IP (tos 0x0, ttl 64, id 1633, offset 23680, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.874636 IP (tos 0x0, ttl 64, id 1633, offset 25160, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.874761 IP (tos 0x0, ttl 64, id 1633, offset 26640, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.874880 IP (tos 0x0, ttl 64, id 1633, offset 28120, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.875006 IP (tos 0x0, ttl 64, id 1633, offset 29600, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.875133 IP (tos 0x0, ttl 64, id 1633, offset 31080, flags [+], length: 1500) SERVER > CLIENT: udp
00:50:42.875157 IP (tos 0x0, ttl 64, id 1633, offset 32560, flags [none], length: 364) SERVER > CLIENT: udp
As you can see, all fragments are received. The issue here appears to
be that the kernel is having problems reassembling the packet, and the
NFS/RPC layer just keeps retransmitting the request forever.
"nstat" shows this interesting part:
UdpInErrors 1 0.0
UdpOutDatagrams 1 0.0
UdpRcvbufErrors 1 0.0
Tracing this down in the source (2.6.21-rc3, btw), I see the only way
this counter can get incremented is if sock_queue_rcv_skb returns -ENOMEM.
In core/sock.c, I see:
if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
(unsigned)sk->sk_rcvbuf) {
err = -ENOMEM;
goto out;
}
I'm guessing the rcvbuf must be too small in this case. I figure at this
point it can't be changed and I'm going to have to reboot to unclog this
thing, but I wonder why it's hitting this case at all ...and why it's
replying with a 32 kB UDP blob to begin with.
Seen anything like this before?
Simon-
-
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
