On Thu, Feb 9, 2012 at 15:15, Jim Schutt <jaschut@xxxxxxxxxx> wrote: > I suspect a bug in the stack, as at an application level I get > the same sort of stalls whether I use IP over ethernet or IPoIB. > I need to get traces for both cases to prove that it is the same > stall... Hi. I just wanted to confirm that what your tcpdump shows is a packet loss between the client and the osd. If this wasn't packet loss, you'd expect the TCP window size to drop to 0 -- not remain at 64k as it is in your dump -- and you'd expect to see ACKs that don't advance the sequence number. Something like this: 16:25:16.914407 IP 127.0.0.1.60336 > 127.0.0.1.9999: Flags [P.], seq 90150:93094, ack 1, win 257, options [nop,nop,TS val 3732293 ecr 3732270], length 2944 16:25:16.914416 IP 127.0.0.1.9999 > 127.0.0.1.60336: Flags [.], ack 93094, win 0, options [nop,nop,TS val 3732293 ecr 3732293], length 0 16:25:17.144409 IP 127.0.0.1.60336 > 127.0.0.1.9999: Flags [.], ack 1, win 257, options [nop,nop,TS val 3732316 ecr 3732293], length 0 16:25:17.144421 IP 127.0.0.1.9999 > 127.0.0.1.60336: Flags [.], ack 93094, win 0, options [nop,nop,TS val 3732316 ecr 3732293], length 0 16:25:17.604409 IP 127.0.0.1.60336 > 127.0.0.1.9999: Flags [.], ack 1, win 257, options [nop,nop,TS val 3732362 ecr 3732316], length 0 16:25:17.604419 IP 127.0.0.1.9999 > 127.0.0.1.60336: Flags [.], ack 93094, win 0, options [nop,nop,TS val 3732362 ecr 3732293], length 0 As pointed out by Sridhar, various TCP offload mechanisms (and firewalling!) may make your tcpdump not see the underlying reality. You might also be just actually losing packets, and the osd settings might, perhaps, influence the performance of the machine enough to make it lose packets -- though that sounds a bit far fetched. You might also be suffering from a Path MTU Discovery black hole, and need the osd size cap to get full-frame packets out. I see your tcpdump indicated jumbo frames (at least until the TSO engine!), that might be its own source of pain. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
