Hi Amos,
regrets, I am late. On 21 April 2015 at 09:15, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote:
On 20/04/2015 7:31 p.m., Jagannath Naidu wrote:
> Hi,
>
> I am having this issue very frequently. Please help on this.
>
> I get these errors randomly, mostly when usage is at very peak. (800 users)
>
>
> /var/log/squid/cache.log
>
> 2015/04/20 12:37:40| externalAclLookup: 'wbinfo_group_helper' queue
> overload (ch=0x7fc99e2ce518)
What do you think "overload" means?
The helper is unable to cope with the traffic load being passed to it.
Here is the biggest hint:
>
> in /var/log/messages, I get the following errors
>
> pr 20 12:59:15 GGNPROXY01 winbindd[1910]: winbindd: Exceeding 200 client
> connections, no idle connection found
> Then squid stops working. For squid to start work again, I have to dlete
> the cache and restart the squid "squid -k reconfigure", and then squid
> restart.
What Squid version are you using?
my squid version squid-3.1.10-19.el6_4.x86_64
>
> squid.conf
>
> max_filedesc 17192
> acl manager proto cache_object
> acl localhost src 172.16.50.61/24
changed to "acl localhost src 172.16.50.61" already
You have an entire /24 (256 IPs) assigned to this machine?
I think you need to remove that "/24" part if the *.61 is the local
machines *public* IP.
> http_access allow manager localhost
> dns_nameservers 172.16.3.34 10.1.2.91
> acl allowips src 172.16.58.187 172.16.16.192 172.16.58.113 172.16.58.63
> 172.16.58.98 172.16.60.244 172.16.58.165 172.16.58.157
> http_access allow allowips
> auth_param basic realm Squid proxy-caching web server
> auth_param basic credentialsttl 2 hours external_acl_type nt_group ttl=0
> children=60 %LOGIN /usr/lib64/squid/wbinfo_group.pl
The above two very mangled config lines are useless. Remove them.
> acl localnet src 172.16.0.0/24
changed
Its a bit strange that none of the localhost machine IPs
(172.16.50.0-172.16.50.255) are part of the LAN its plugged into
172.16.0.0-172.16.0.255.
> acl localnet src fc00::/7 # RFC 4193 local private network range
> acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
> auth_param ntlm program /usr/bin/ntlm_auth --diagnostics
> --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET
Okay you have configured NTLM...
> auth_param ntlm program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET
... but twice. With different settings. Only these last ones will have
any effect.
> auth_param ntlm children 600
> auth_param ntlm keep_alive off
> auth_param negotiate children 150
> auth_param negotiate keep_alive off
> visible_hostname GGNPROXY01.HTMEDIA.NET
> external_acl_type wbinfo_group_helper ttl=0 children=40 %LOGIN
> /usr/lib64/squid/wbinfo_group.pl -d
> auth_param negotiate keep_alive off
You have several useless configuration lines for Negotiate auth which is
not being used in any way. Remove those.
You already have an ACL named "auth" which performs authentication.
> acl Safe_ports port 8080 #https
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl CONNECT method CONNECT
> acl auth proxy_auth REQUIRED
> acl google dstdomain -i "/etc/squid/google_site.com"
> http_access allow google
> acl sq1 external wbinfo_group_helper "/etc/squid/HT/sq1"
> acl sq2 external wbinfo_group_helper "/etc/squid/HT/sq2"
> acl sq3 external wbinfo_group_helper "/etc/squid/HT/sq3"
> acl sq4 external wbinfo_group_helper "/etc/squid/HT/sq4"
> acl sq5 external wbinfo_group_helper "/etc/squid/HT/sq5"
> acl pro1 external wbinfo_group_helper "/etc/squid/HT/pro1"
> acl pro2 external wbinfo_group_helper "/etc/squid/HT/pro2"
> acl pro3 external wbinfo_group_helper "/etc/squid/HT/pro3"
> acl pro4 external wbinfo_group_helper "/etc/squid/HT/pro4"
> acl pro5 external wbinfo_group_helper "/etc/squid/HT/pro5"
> acl pro6 external wbinfo_group_helper "/etc/squid/HT/pro6"
> acl webvip external wbinfo_group_helper "/etc/squid/HT/webvip"
> acl allgroup external wbinfo_group_helper "/etc/squid/HT/allgreop"
> acl restricted external wbinfo_group_helper "/etc/squid/HT/restricted"
> acl ad_auth proxy_auth REQUIRE
The above line is not useful. Remove it and replace all uses of
"ad_auth" ACL with "auth" ACL.
> acl allowwebsites dstdomain -i "/blacklists/allowedwebsite/domains"
> acl allowwebsites_url url_regex -i "/blacklists/allowedwebsite/url"
> http_access allow allowwebsites
> http_access allow allowwebsites_url
> acl shopping dstdomain -i "/etc/squid/shopping.txt"
> acl social_networking dstdomain -i "/blacklists/social/social.networking"
> acl youtube dstdomain -i .youtube.com
> http_access allow Safe_ports pro1 pro2 pro3 pro4 pro5 pro6 webvip
Incorrect use of "Safe_ports" security check. Correct usage is to deny
access to all *unsafe* ports. They are unsafe because HTTP can be
smuggled within the ports native protocol to attack your proxy.
Once the correct security protections for Safe_port and CONNECT tunnels
have been moved up the top remove the "Safe_ports" check from this line.
This line is also very odd in another way. ACL tests in a single line
are AND'ed together - so this means the request must be from a user who is:
authenticated AND a member of group pro1 AND pro2 AND pro3 AND pro4
AND pro5 AND pro6 AND webvip
This hints at what your main helper problem is. The above line requires
7 group helper lookups *per request*. The winbind helper has a maximum
of 200 simultaneous connections. This line alone will limit your proxy
just under 30 new visitors per second (that becomes 60 lookups/sec
before queue overload).
The helper result caching will help a lot, but you also have a LOT of
other group checks being made and 800 users.
> http_access allow youtube pro5
> http_access allow youtube pro6
> http_access allow youtube webvip
> http_access deny youtube
> http_access allow shopping pro5
> http_access allow shopping pro6
> http_access allow shopping webvip
> http_access deny shopping
Optimization hint:
"youtube" and "shopping" have the same allow/deny criteria. It would be
worth combining them into one ACL.
> http_access allow social_networking pro2
> http_access allow social_networking pro4
> http_access allow social_networking pro6
> http_access allow social_networking webvip
> http_access deny social_networking
> acl porn_site1 dstdomain "/etc/squid/blacklists/porn/domains.txt"
> acl porn_site2 dstdom_regex -i "/etc/squid/blacklists/porn/expressions"
> acl porn_site3 dstdom_regex -i "/etc/squid/blacklists/porn/urls.txt"
> acl audio_video1 dstdomain "/etc/squid/blacklists/audio-video/urls.txt"
> ###################### THERE ARE TOO MANY acls and http_access , so not
> bothering with vast linux
I will bet a lot of those ACLs are also calling the group helper too yes?
> http_access allow liquorinfo webvip
> http_access deny liquorinfo
> http_access allow ad_auth
> http_access allow auth
Once you have removed ad_auth ACL, this becomes:
http_access allow auth
http_access allow auth
I hope you can see how redundant that is.
Also, its very likely that the "allow auth" is a useless operation after
a great many group checks have also performed authentication. That "TOO
MANY acls and https_access" list you omitted will be needed to determine
that.
> http_access allow sq1 sq2
> acl NTLMUsers proxy_auth REQUIRED
You already have an ACL named "auth" which performs authentication.
The above line is not being used in any way. Remove it.
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
These are basic security protection against Denial of Service and other
types of protocol smuggling attacks. They only work when they are used
*above* your custom "allow" rules.
Move these two lines above your "http_access allow google" line.
> http_port 8080
> hierarchy_stoplist cgi-bin ?
The above line is not useful these days. Remove it.
> cache_effective_user squid
> cache_dir aufs /var/spool/squid 20384 32 512
> cache_mem 50 MB
> cache_replacement_policy heap LFUDA
> cache_swap_low 85
> cache_swap_high 95
> maximum_object_size 5 MB
> maximum_object_size_in_memory 50 KB
> ipcache_size 5240
> ipcache_low 90
> ipcache_high 95
> cache_mgr amit
> acl SSL_ports port 443
The above is a duplicate config line. Remove it.
> http_access allow CONNECT SSL_ports
> coredump_dir /var/spool/squid
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> refresh_pattern . 0 20% 4320
> url_rewrite_program /usr/local/bin/squidGuard -c
> /usr/local/squidGuard/squidGuard.conf
>
Now, as to solving your problem:
1) Clean up your config. Reduce the amount of redundant or unused
things. I've mentioned a few above.
2) Run "squid -k parse" and fix any other problems it highlights.
3) optimize your ACls and http_access rules. I've mentioned a few, such
as moving the main security checks to the top so DoS traffic does not
put load on the helpers and other ACLs.
I believe though that you will probably find Squid works much better
having the following access controls pattern:
"
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
# if they are not authenticated, they will not be in a group
http_access deny !auth
# assuming that webvip are the group with full access?
http_access allow webvip
# your long list of per-site group check ACLs go here
...
# this is where defining the LAN ranges correctly comes in.
# note that users have authenticated simply to get near here
http_access allow localnet
http_access deny all
"
4) consider an upgrade to Squid 3.4+. The "notes" ACL type offers much
more efficient ACL testing with a custom group lookup helper. The all-of
and any-of ACL types can also much reduce your http_access lines.
HTH
Amos
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
Thank you Amos, I will check and will update the list.
--
Thanks & Regards
B Jagannath
B Jagannath
Keen & Able Computers Pvt. Ltd.
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
