On 07/11/10 13:21, Leonardo Rodrigues wrote:
Hi,
i'll try to describe with the most details i can what i think is
something like a forwarding-loop-detection bug on 2.7S9
i have squid 2.7S9 running on a CentOS 5.5 x64 box whici has 4 NICs. 3
NICs are for internal networks (192.168.x) and 1 NIC is for internet
(189.73.x.x). It was built with:
[root@firewall squid]# squid -v
Squid Cache: Version 2.7.STABLE9
configure options: '--prefix=/usr' '--exec-prefix=/usr/bin'
'--bindir=/usr/bin' '--sbindir=/usr/sbin' '--libexecdir=/usr/bin'
'--sysconfdir=/etc/squid' '--datadir=/var/squid' '--localstatedir=/var'
'--enable-removal-policies=heap,lru' '--enable-storeio=ufs,aufs,null'
'--enable-delay-pools' '--enable-http-violations' '--with-maxfd=8192'
'--enable-async-io=8' '--enable-err-languages=Portuguese English'
'--enable-default-err-language=Portuguese' '--enable-snmp'
'--disable-ident-lookups' '--enable-linux-netfilter'
'--enable-auth=basic digest ntlm negotiate'
'--enable-basic-auth-helpers=DB LDAP NCSA SMB'
'--enable-digest-auth-helpers=password ldap'
'--enable-external-acl-helpers=ip_user ldap_group session wbinfo_group'
'--enable-negotiate-auth-helpers=squid_kerb_auth'
'--enable-ntlm-auth-helpers=fakeauth no_check' '--enable-useragent-log'
'--enable-referer-log' '--disable-wccp' '--disable-wccpv2'
'--enable-arp-acl' '--with-large-files' '--enable-large-cache-files'
'--enable-ssl' '--enable-icmp'
i've setup squid with something like:
acl localhost src 127.0.0.1/255.255.255.255
acl localhost_to dst 127.0.0.1/255.255.255.255
acl network1 src 192.168.1.0/255.255.255.0
acl network1_to dst 192.168.1.0/255.255.255.0
acl network2 src 192.168.2.0/255.255.255.0
acl network2_to dst 192.168.2.0/255.255.255.0
acl network3 src 192.168.3.0/255.255.255.0
acl network3_to dst 192.168.3.0/255.255.255.0
http_port 8080 transparent
http_port 3128 transparent
tcp_outgoing_address 127.0.0.1 localhost_to
tcp_outgoing_address 192.168.1.1 network1_to
tcp_outgoing_address 192.168.2.1 network2_to
tcp_outgoing_address 192.168.3.1 network3_to
tcp_outgoing_address 189.73.x.x all
config is OK, it runs just fine.
Obviously not. Or you would not be reporting this problem.
problem is, on a given day, squid stop responding new connections and i
have to stop it (service squid stop). After searching logs, i have found
some interesting requests:
1288136326.944 48437 192.168.2.15 TCP_MISS/000 0 GET
http://localhost:8080/sync/sis/index.php - DIRECT/127.0.0.1 -
DoS attack by 192.168.2.15.
1288136326.944 48426 127.0.0.1 TCP_MISS/000 0 GET
http://localhost:8080/sync/sis/index.php - DIRECT/127.0.0.1 -
(and this second line repeated about 13000 times)
This is why the default Squid ruleset includes:
"http_access deny to_localhost"
and during these, i got also on cache.log:
2010/10/26 21:37:59| WARNING! Your cache is running out of filedescriptors
2010/10/26 21:38:15| WARNING! Your cache is running out of filedescriptors
2010/10/26 21:38:31| WARNING! Your cache is running out of filedescriptors
2010/10/26 21:38:48| WARNING! Your cache is running out of filedescriptors
2010/10/26 21:39:04| WARNING! Your cache is running out of filedescriptors
2010/10/26 21:39:20| WARNING! Your cache is running out of filedescriptors
i'm running with 8192 filedescriptors on a 150 clients network, that's
more than enough filedescriptors for normal usage.
(from cache.log)
2010/10/31 12:27:50| Starting Squid Cache version 2.7.STABLE9 for
x86_64-unknown-linux-gnu...
2010/10/31 12:27:50| Process ID 16093
2010/10/31 12:27:50| With 8192 file descriptors available
Well ..... after found that, i tried to reproduce it doing some request
to localhost:8080 on 8080 squid port and i could successfully reproduce
it, all the times, with the above squid.conf configuration.
after some tryings, i have found that:
1) removing the:
tcp_outgoing_address 127.0.0.1 localhost_to
would avoid the problem and make the forwarding-loop-detection works fine
Indicating that your NAT rules are incorrect.
The above line is simply forcing Squid to send from 127.0.0.1. It would
only have any effect if your NAT intercept rules were forcing all
localhost traffic back into Squid.
Removing the above line may mean that you are simply shifting the
problem from your Squid to some web server elsewhere. Your Squid will be
passing it requests for "http://localhost:8080/...";. The upside is that
at least it will not be a DoS flood when it arrives there.
2) removing the transparent from
http_port 8080 transparent
would avoid the problem too, even with the tcp_outgoing_address
127.0.0.1 active
Yes. Not doing NAT is good protection against this whole class of IP
address problems.
question is ..... squid NOT detecting this forwarding-loop should be
expected with this transparent and tcp_outgoing_address combination ?
Do you have both "via on" and "forwarded_for on" set in your squid.conf?
They are both needed.
Are we talking of a bug or are we talking of some expected behavior ? Is
there any other information that i could provide to help tracking this ?
The forwarding detection is not perfect. The to_localhost DoS protection
exists to prevent this.
In other news ... why do you have 2 interception ports? they are only
relevant between Squid and the NAT logic on the local box.
Also using two well-known ports such as 8080 and 3128 is just asking to
be kicked by CVE-2009-0801 and this particular DoS attack you have seen.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE9 or 3.1.9
Beta testers wanted for 3.2.0.2
