On Tue, Mar 6, 2012 at 1:01 PM, Tom Evans <tevans.uk@xxxxxxxxxxxxxx> wrote: > So, we've been trying to track disappearing requests. We see lots of > requests that go via the CDN to reach our data centre failing with > error code 503. This error message is produced by the CDN, and the > request is not logged in either of the FEPs. > > We've been trying to track what happens with tcpdump running at SQUID > and at FW. At SQUID, we see a POST request for a resource, followed by > a long wait, and then a 503 generated by the CDN. Interestingly, 95% > of the failing requests are POST requests. > > Tracking that at FW, we see the request coming in, and no reply from > the FEP. The connection is a keep-alive connection, and had just > completed a similar request 4 seconds previously, to which we returned > a 200 and data. This (failing) request is made on the same connection, > we reply with an ACK, then no data for 47 seconds (same wait as seen > by squid), and finally the connection is closed with a FIN. > Sorry, one final thing - we can see these hanging connections on the FEP: netstat -an | head -n 2 ; netstat -an | fgrep EST | fgrep -v "tcp4 0" This shows the established sockets with unread recv-q. Obviously not every socket shown is hanging; but by observing it over an extended (10s) period, you can quickly see connections whose recv-q is not drained. Cheers Tom --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|