More odd tcp_miss Only had a small portion of the site which would work. Works fine from the primary, but fails on the secondary squid. 1269906612.412 5464 10.xxx.xxxx.xxx TCP_MISS/000 0 GET http://www.internode.on.net/products/ - DIRECT/203.16.214.27 - 1269906612.930 17 10.xxx.xxxx.xxx TCP_MISS/200 851 GET http://advisories.internode.on.net/images/menu2-on.gif - DIRECT/192.231.203.146 image/gif 1269906613.075 9 10.xxx.xxxx.xxx TCP_REFRESH_MODIFIED/200 782 GET http://advisories.internode.on.net/images/menu2.gif - DIRECT/192.231.203.146 image/gif 1269906613.331 221 10.xxx.xxxx.xxx TCP_MISS/200 819 GET http://advisories.internode.on.net/images/menu1-on.gif - DIRECT/192.231.203.146 image/gif 1269906614.487 1865 10.xxx.xxxx.xxx TCP_MISS/000 0 GET http://www.internode.on.net/ - DIRECT/203.16.214.27 - 1269906696.702 60903 10.xxx.xxxx.xxx TCP_MISS/000 0 GET http://www.internode.on.net/products/ - DIRECT/203.16.214.27 - 1269906767.709 61004 10.xxx.xxxx.xxx TCP_MISS/000 0 GET http://www.internode.on.net/products/ - DIRECT/203.16.214.27 - 1269906840.719 60299 10.xxx.xxxx.xxx TCP_MISS/000 0 GET http://www.internode.on.net/products/broadband/plan_changes/ - DIRECT/203.16.214.27 - 1269906911.707 60981 10.xxx.xxxx.xxx TCP_MISS/000 0 GET http://www.internode.on.net/products/broadband/plan_changes/ - DIRECT/203.16.214.27 - On Mon, Mar 29, 2010 at 5:56 PM, Ivan . <ivanhec@xxxxxxxxx> wrote: > That is so odd, as I have two identical boxes, now running the same > Squid version, going through the same infrastructure, one works, the > other one doesn't? > > The only difference are the public addresses configured on each of the > squid proxy systems..... > > The TCP stats on the interface on the squid box that won't access that > site, don't look to bad at all > > [root@pcr-proxy ~]# netstat -s > Ip: > 1593488410 total packets received > 17991 with invalid addresses > 0 forwarded > 0 incoming packets discarded > 1593318874 incoming packets delivered > 1413863445 requests sent out > 193 reassemblies required > 95 packets reassembled ok > Icmp: > 22106 ICMP messages received > 0 input ICMP message failed. > ICMP input histogram: > destination unreachable: 16 > echo requests: 22090 > 155761 ICMP messages sent > 0 ICMP messages failed > ICMP output histogram: > destination unreachable: 133671 > echo replies: 22090 > IcmpMsg: > InType3: 16 > InType8: 22090 > OutType0: 22090 > OutType3: 133671 > Tcp: > 27785486 active connections openings > 78777077 passive connection openings > 68247 failed connection attempts > 560600 connection resets received > 569 connections established > 1589479495 segments received > 1403833081 segments send out > 6034370 segments retransmited > 0 bad segments received. > 626711 resets sent > Udp: > 3817253 packets received > 20 packets to unknown port received. > 0 packet receive errors > 3840233 packets sent > TcpExt: > 217 invalid SYN cookies received > 15888 resets received for embryonic SYN_RECV sockets > 42765 packets pruned from receive queue because of socket buffer overrun > 7282834 TCP sockets finished time wait in fast timer > 3 active connections rejected because of time stamp > 11427 packets rejects in established connections because of timestamp > 8682907 delayed acks sent > 1268 delayed acks further delayed because of locked socket > Quick ack mode was activated 1227980 times > 36 packets directly queued to recvmsg prequeue. > 14 packets directly received from prequeue > 538829561 packets header predicted > 492906318 acknowledgments not containing data received > 190275750 predicted acknowledgments > 372 times recovered from packet loss due to fast retransmit > 348117 times recovered from packet loss due to SACK data > 174 bad SACKs received > Detected reordering 71 times using FACK > Detected reordering 963 times using SACK > Detected reordering 25 times using reno fast retransmit > Detected reordering 998 times using time stamp > 560 congestion windows fully recovered > 10689 congestion windows partially recovered using Hoe heuristic > TCPDSACKUndo: 921 > 197231 congestion windows recovered after partial ack > 1316789 TCP data loss events > TCPLostRetransmit: 22 > 3020 timeouts after reno fast retransmit > 78970 timeouts after SACK recovery > 10665 timeouts in loss state > 743644 fast retransmits > 1003156 forward retransmits > 1884003 retransmits in slow start > 1604549 other TCP timeouts > TCPRenoRecoveryFail: 150 > 31151 sack retransmits failed > 4198383 packets collapsed in receive queue due to low socket buffer > 814608 DSACKs sent for old packets > 33462 DSACKs sent for out of order packets > 65506 DSACKs received > 266 DSACKs for out of order packets received > 215231 connections reset due to unexpected data > 10630 connections reset due to early user close > 76801 connections aborted due to timeout > IpExt: > InBcastPkts: 9199 > > > On Mon, Mar 29, 2010 at 5:47 PM, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote: >> Ivan . wrote: >>> >>> Some even more strange access.log entries? >>> >>> This is odd? Does that mean no DNS record? strange as both squid's use >>> the same DNS setup, with a primary, secondary and tertiary setup. >>> 1269833940.167 0 127.0.0.1 NONE/400 1868 GET >>> www.environment.gov.au - NONE/- text/html >>> >>> 1269833960.464 60997 10.132.17.30 TCP_MISS/000 0 GET >>> http://www.environment.gov.au/ - DIRECT/155.187.3.81 - >>> >>> 1269834108.182 120002 127.0.0.1 TCP_MISS/000 0 GET >>> http://www.environment.gov.au - DIRECT/155.187.3.81 - >>> >>> This one is new? >>> 1269842635.028 295660 10.143.254.22 TCP_MISS/502 2514 GET >>> http://www.environment.gov.au/ - DIRECT/155.187.3.81 text/html >>> >> >> The TCP_MISS/000 are another version of the READ_ERROR you are receiving as >> TCP_MISS/502. The 000 ones are on the client facing side though, the TCP >> link read failing before the request headers are finished being received >> from the client. >> The first line is received (to get the URL) but not the rest of the request >> headers. >> >> The NONE/400 might be yet another version of the read failing at some point >> of processing. It's hard to say. >> >> Something is definitely very screwed at the TCP protocol level for those >> requests. >> >> Amos >> >>> >>> >>> On Mon, Mar 29, 2010 at 4:56 PM, Ivan . <ivanhec@xxxxxxxxx> wrote: >>>> >>>> Hi Amos >>>> >>>> You can see the tcp_miss in the access.log here:- >>>> >>>> 1269834108.182 120002 127.0.0.1 TCP_MISS/000 0 GET >>>> http://www.environment.gov.au - DIRECT/155.187.3.81 - >>>> >>>> Here is a tcpdump output from the connection. You can see the TCP >>>> handshake setup and then the http session just hangs? I have confirmed >>>> with the website admin these are no ddos type protection, which would >>>> block multiple requests in quick succession. >>>> >>>> The tcp connection times out and then resets. >>>> >>>> [root@squid-proxy ~]# tcpdump net 155.187.3 >>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol >>>> decode >>>> listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes >>>> 16:58:59.369482 IP xxx.xxxx.xxx.xxx.41338 > 155.187.3.81.http: S >>>> 1781942738:1781942738(0) win 5840 <mss 1460,sackOK,timestamp >>>> 1321171542 0,nop,wscale 7> >>>> 16:58:59.418150 IP 155.187.3.81.http > xxx.xxxx.xxx.xxx.41338: S >>>> 2343505326:2343505326(0) ack 1781942739 win 32768 <mss 1460,nop,wscale >>>> 0,nop,nop,timestamp 234270252 1321171542,sackOK,eol> >>>> 16:58:59.418167 IP xxx.xxxx.xxx.xxx.41338 > 155.187.3.81.http: . ack 1 >>>> win 46 <nop,nop,timestamp 1321171591 234270252> >>>> 16:58:59.418213 IP xxx.xxxx.xxx.xxx.41338 > 155.187.3.81.http: P >>>> 1:696(695) ack 1 win 46 <nop,nop,timestamp 1321171591 234270252> >>>> 16:58:59.477692 IP 155.187.3.81.http > xxx.xxxx.xxx.xxx.41338: P >>>> 2897:4081(1184) ack 696 win 33304 <nop,nop,timestamp 234270307 >>>> 1321171591> >>>> 16:58:59.477700 IP xxx.xxxx.xxx.xxx.41338 > 155.187.3.81.http: . ack 1 >>>> win 46 <nop,nop,timestamp 1321171650 234270252,nop,nop,sack 1 >>>> {2897:4081}> >>>> >>>> >>>> cheers >>>> Ivan >>>> >>>> On Mon, Mar 29, 2010 at 3:59 PM, Amos Jeffries <squid3@xxxxxxxxxxxxx> >>>> wrote: >>>>> >>>>> Ivan . wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> What would cause a TCP MISS 502, which would prevent a site from >>>>>> loading? The site works on squidv3.0 but not on v2.6? >>>>>> >>>>> Any one of quite a few things. The ERR_READ_ERROR result means the >>>>> remote >>>>> server or network is closing the TCP link on you for some unknown >>>>> reason. >>>>> >>>>> Why it works in 3.0 is as much a mystery as why it does not in 2.6 until >>>>> details of the traffic on Squid->Server TCP link are known. >>>>> >>>>> >>>>> Amos >>>>> -- >>>>> Please be using >>>>> Current Stable Squid 2.7.STABLE8 or 3.0.STABLE25 >>>>> Current Beta Squid 3.1.0.18 >>>>> >> >> >> -- >> Please be using >> Current Stable Squid 2.7.STABLE8 or 3.0.STABLE25 >> Current Beta Squid 3.1.0.18 >> >
