Hello,lately I noticed that our web servers don't answer to a request from time to time. I created a dummy http checking script every few seconds to debug a problem. It seems that server does not even respond with SYNACK packet. Here is the excerpt from wireshark/tcpdump log:
# failed request10629 12:56:23.642861 10.9.0.14 10.9.1.33 TCP 57400 > http [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=663117245 TSER=0 WS=7 10632 12:56:26.642216 10.9.0.14 10.9.1.33 TCP 57400 > http [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=663117545 TSER=0 WS=7
# OK request10767 12:57:39.643092 10.9.0.14 10.9.1.33 TCP 57209 > http [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=663124844 TSER=0 WS=7 10768 12:57:39.643109 10.9.1.33 10.9.0.14 TCP http > 57209 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=134534397 TSER=663124844 WS=7 10769 12:57:39.643280 10.9.0.14 10.9.1.33 TCP 57209 > http [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=663124844 TSER=134534397
10770 12:57:39.710840 10.9.0.14 10.9.1.33 HTTP GET / HTTP/1.110771 12:57:39.710862 10.9.1.33 10.9.0.14 TCP http > 57209 [ACK] Seq=1 Ack=114 Win=5888 Len=0 TSV=134534404 TSER=663124851
10772 12:57:39.711134 10.9.1.33 10.9.0.14 HTTP HTTP/1.1 200 OK (text/html)10773 12:57:39.711189 10.9.1.33 10.9.0.14 TCP http > 57209 [FIN, ACK] Seq=308 Ack=114 Win=5888 Len=0 TSV=134534404 TSER=663124851 10774 12:57:39.711297 10.9.0.14 10.9.1.33 TCP 57209 > http [ACK] Seq=114 Ack=308 Win=6912 Len=0 TSV=663124851 TSER=134534404 10775 12:57:39.721484 10.9.0.14 10.9.1.33 TCP 57209 > http [FIN, ACK] Seq=114 Ack=309 Win=6912 Len=0 TSV=663124852 TSER=134534404 10776 12:57:39.721495 10.9.1.33 10.9.0.14 TCP http > 57209 [ACK] Seq=309 Ack=115 Win=5888 Len=0 TSV=134534405 TSER=663124852
OS is 64-bit slackware derivate with glibc 2.7, kernel 2.6.28.7 and e1000e network driver. Hardware is Intel S5000PAL board, 2x Quad-Xeon E5345 with 16gb RAM.
We're running Apache 2.2.11 with worker mpm.
So far I've tried tweaking the following (with no luck):
- changing accept mutex - tried sysvsem, flock, fcntl, posixsem
- tweaked ListenBackLog + /proc/sys/net/core/somaxconn
- tweaked various parts of mpm settings
- tweaking at least the following settings:
/proc/sys/net/ipv4/tcp_max_syn_backlog
/proc/sys/net/ipv4/tcp_rmem
/proc/sys/net/ipv4/tcp_wmem
/proc/sys/net/ipv4/tcp_synack_retries
/proc/sys/net/ipv4/ip_local_port_range
set txqueuelen on interfaces to 2000
There is no firewall on any machine and both machines on local network,
no routers in the middle. Servers are quite loaded though - pushing
about ~15mbit/s on each of the two interfaces. However the check also
fails during the night, when the volume of traffic is quite low.
Any ideas? -- Miha --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscribe@xxxxxxxxxxxxxxxx " from the digest: users-digest-unsubscribe@xxxxxxxxxxxxxxxx For additional commands, e-mail: users-help@xxxxxxxxxxxxxxxx
|