NTLM AUTH: All redirector processes are busy

[Jagannath Naidu <jagannath.naidu@xxxxxxxxxxxxxxxxxx>]

Hi List/Amos,

I am facing an using squid in production.

I get these messages in cache.log, and service stop for a period of time (like 14 seconds). During this period, users get panic as they get "proxy server resfusing connections". And automatically the service starts functioning again. But this happens very frequently whole day.

2015/04/29 10:34:10| WARNING: All redirector processes are busy.
2015/04/29 10:34:10| WARNING: 15 pending requests queued
2015/04/29 10:34:10| storeDirWriteCleanLogs: Starting...
2015/04/29 10:34:10| WARNING: Closing open FD 3327
2015/04/29 10:34:10|     65536 entries written so far.
2015/04/29 10:34:10|    131072 entries written so far.
2015/04/29 10:34:10|    196608 entries written so far.
2015/04/29 10:34:10|    262144 entries written so far.
2015/04/29 10:34:10|    327680 entries written so far.
2015/04/29 10:34:10|    393216 entries written so far.
2015/04/29 10:34:10|    458752 entries written so far.
2015/04/29 10:34:10|    524288 entries written so far.
2015/04/29 10:34:10|    589824 entries written so far.
2015/04/29 10:34:10|    655360 entries written so far.
2015/04/29 10:34:10|   Finished.  Wrote 716101 entries.
2015/04/29 10:34:10|   Took 0.22 seconds (3266168.90 entries/sec).
FATAL: Too many queued redirector requests
Squid Cache (Version 3.1.10): Terminated abnormally.
CPU Usage: 4206.393 seconds = 3778.049 user + 428.344 sys
Maximum Resident Size: 2599760 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
        total space in arena:  750272 KB
        Ordinary blocks:       717419 KB   6620 blks
        Small blocks:               0 KB      1 blks
        Holding blocks:         23020 KB     11 blks
        Free Small blocks:          0 KB
        Free Ordinary blocks:   32852 KB
        Total in use:          740439 KB 99%
        Total free:             32852 KB 4%
fgets() failed! dying..... errno=1 (Operation not permitted)

2015/04/29 10:34:19| Starting Squid Cache version 3.1.10 for x86_64-redhat-linux-gnu...
2015/04/29 10:34:19| Process ID 4326
2015/04/29 10:34:19| With 100000 file descriptors available
2015/04/29 10:34:19| Initializing IP Cache...
2015/04/29 10:34:19| DNS Socket created at [::], FD 8
2015/04/29 10:34:19| DNS Socket created at 0.0.0.0, FD 9
2015/04/29 10:34:19| Adding nameserver 172.16.3.34 from squid.conf
2015/04/29 10:34:19| Adding nameserver 10.1.2.91 from squid.conf
2015/04/29 10:34:19| helperOpenServers: Starting 5/5 'squidGuard' processes
2015/04/29 10:34:19| helperOpenServers: Starting 1500/1500 'ntlm_auth' processes
2015/04/29 10:34:24| helperOpenServers: Starting 150/150 'wbinfo_group.pl' processes

ntlm helpers count is 1500 and external "wbinfo_group.pl" helpers are 150.

squid.conf
###################################################

max_filedesc 100000
acl manager proto cache_object
acl localhost src 172.16.50.61
http_access allow manager localhost
dns_nameservers 172.16.3.34 10.1.2.91
acl allowips src 172.16.58.187 172.16.16.192 172.16.58.113 172.16.58.63 172.16.58.98 172.16.60.244 172.16.58.165 172.16.58.157
http_access allow allowips
#acl haproxy src 172.16.50.61
#follow_x_forwarded_for allow haproxy
#follow_x_forwarded_for deny all
#acl manager proto cache_object
acl localnet src 172.16.0.0/16
acl manager proto cache_object
acl localhost src 127.0.0.1
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl office dstdomain "/etc/squid/officesites"
http_access allow office
log_ip_on_direct off
#debug_options ALL,3
#logformat squid %9d.%03d %6d %s %s/%03d %d %s %s %s %s%s/%s %s
logformat squid %ts.%03tu %tl %3tr %3dt %3un %>a %Ss/%>Hs %<st %rm %ru  %Sh/%<A %mt
access_log /var/log/squid/access1.log squid
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours external_acl_type nt_group ttl=0 children=60 %LOGIN /usr/lib64/squid/wbinfo_group.pl
#auth_param ntlm program /etc/squid/helper-mux.pl /usr/bin/ntlm_auth --diagnostics --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET
auth_param ntlm program /usr/bin/ntlm_auth --diagnostics --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET
auth_param ntlm children 1500
#auth_param ntlm children 500
auth_param ntlm keep_alive off
auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET
external_acl_type wbinfo_group_helper ttl=600 children=150 %LOGIN /usr/lib64/squid/wbinfo_group.pl -d
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
cl Safe_ports port 8080 #https
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
acl auth proxy_auth REQUIRED

and rest of acls and http_access rules configured ...............

It seems the helper programs are not closing automatically after serving and causes this issue. Could anyone help resolving this issue.

[root@GGNPROXY01 squid]# rpm -qa | grep squid
squid-3.1.10-19.el6_4.x86_64

[root@GGNPROXY01 squid]# rpm -qa | grep winbind
samba-winbind-clients-3.6.9-164.el6.x86_64
samba-winbind-3.6.9-164.el6.x86_64

[root@GGNPROXY01 squid]# lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID:    RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 6.5 (Santiago)
Release:    6.5
Codename:    Santiago

--

Thanks & Regards

B Jagannath

Keen & Able Computers Pvt. Ltd.

+919871324006

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users