On 17/11/2011 16:11, Nataniel Klug wrote:
Hello all,
I am facing a very difficult problem in my network. I am
using a layout like this:
(internet) ==<router> ==<squid> == [clients]
I am running CentOS v5.1 with Squid-2.6 STABLE22 and Tproxy
(cttproxy-2.6.18-2.0.6). My kernel is kernel-2.6.18-92. This is the most
reliable setup I ever made running Squid. My problem is that I am having
serious connections troubles when running squid over 155000 conntrack
connections.
From my clients I start losing packets to router when the
connections go over 155000. My kernel is prepared to run over 260k
connections. I am sending a screenshot about the problem where I have 156k
connections and I started connections on port 80 to go through squid (bellow
I will post every rule I am using for my firewall and transparent
connections, also I will send my squid.conf).
http://imageshack.us/photo/my-images/12/problemsg.png/
The configuration I am using:
/etc/firewall/firewall
#!/bin/bash
IPT="/sbin/iptables"
RT="/sbin/route"
SYS="/sbin/sysctl -w"
$IPT -F
$IPT -t nat -F
$IPT -t nat -X
$IPT -t mangle -F
$IPT -t mangle -X
$IPT -t filter -F
$IPT -t filter -X
$IPT -X
$IPT -F INPUT
$IPT -F FORWARD
$IPT -F OUTPUT
$SYS net.ipv4.ip_forward=1
$SYS net.ipv4.ip_nonlocal_bind=1
$SYS net.ipv4.netfilter.ip_conntrack_max=262144
/etc/firewall/squid-start
#!/bin/bash
IP="/sbin/ip"
IPT="/sbin/iptables"
FWDIR="/etc/firewall"
/etc/firewall/firewall
$IPT -t tproxy -F
for i in `cat $FWDIR/squid-no-dst`
do
$IPT -t tproxy -A PREROUTING -d $i -j ACCEPT
done
for i in `cat $FWDIR/squid-no-src`
do
$IPT -t tproxy -A PREROUTING -s $i -j ACCEPT
done
$IPT -t tproxy -A PREROUTING -p tcp --dport 80 -j TPROXY --on-port 3128
/etc/squid/squid.conf
http_port 3128 tproxy transparent
tcp_outgoing_address XXX.XXX.144.67
icp_port 0
cache_mem 128 MB
cache_swap_low 92
cache_swap_high 96
maximum_object_size 1000000 KB
cache_replacement_policy heap LFUDA
memory_replacement_policy heap LFUDA
cache_dir aufs /cache/01/01 47000 64 256
cache_dir aufs /cache/01/02 47000 64 256
cache_dir aufs /cache/02/01 47000 64 256
cache_dir aufs /cache/02/02 47000 64 256
cache_dir aufs /cache/03/01 47000 64 256
cache_dir aufs /cache/03/02 47000 64 256
#--[ Max Usage : by Drive ]--#
# sdb1 ( max = 228352 / usg = 95400 (41,77%) ]
# sdb1 ( max = 228352 / usg = 95400 (41,77%) ]
# sdb3 [ max = 234496 / usg = 95400 (40,68%) ]
#-- [ Max HDD sdb Usage ]--#
# sdb [ max = 923994 / aloc = 691200 (74,81%) ]
cache_store_log none
access_log /usr/local/squid/var/logs/access.log squid
client_netmask 255.255.255.255
ftp_user squid@xxxxxxxxxxxx
diskd_program /usr/local/squid/libexec/diskd
unlinkd_program /usr/local/squid/libexec/unlinkd
error_directory /usr/local/squid/share/errors/Portuguese
dns_nameservers XXX.XXX.144.14 XXX.XXX.144.6
acl all src 0.0.0.0/0
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl QUERY urlpath_regex cgi-bin \?
acl SSL_ports port 443
acl Safe_ports port 80 21 443 70 210 280 488 591 777 1025-65535
acl CONNECT method CONNECT
acl ASN53226_001 src XXX.XXX.144.0/22
acl ASN53226_002 src XXX.XXX.148.0/22
http_access allow ASN53226_001
http_access allow ASN53226_002
http_access allow localhost
http_access allow to_localhost
cache deny QUERY
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny all
icp_access deny all
cache_mgr suporte@xxxxxxxxxxxx
cache_effective_user squid
cache_effective_group squid
visible_hostname cache
unique_hostname 02.cache
When I first start linux and there is just a few connections
going through the squid box it works just fine. When the connections go over
155k the problems began. Is there anything I can do to solve the problem?
well this is one of the big problems of the conntrack thingy..
what you can try is to also to change the tcp to:
sysctl net.ipv4.netfilter.ip_conntrack_tcp_timeout_established=3600
cause it might causing the problem of such a huge ammount of connection
tracking size.
the basic size is 120 minutes which can cause a lot of troubles in many
cases of open connections.
and by the way.. do you really have 155K connections? it seems like too
much.
hope to hear more about the situation.
Regards Eliezer
--
Att,
Nataniel Klug
