2013/2/15 Amos Jeffries <squid3@xxxxxxxxxxxxx>: > On 15/02/2013 11:53 p.m., dahanhsi wrote: >> >> Hi Amos, >> >> >> 2013/2/15 Amos Jeffries <squid3@xxxxxxxxxxxxx>: >>> >>> On 15/02/2013 10:43 p.m., dahanhsi wrote: >>>> >>>> Thanks for your reply, >>>> provide more information below: >>>> >>>> 2013/2/15 Amos Jeffries <squid3@xxxxxxxxxxxxx>: >>>>> >>>>> On 15/02/2013 10:12 p.m., dahanhsi wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I use squid as a reverse proxy, and make thousands of connection to >>>>>> the >>>>>> it. >>>>> >>>>> Which version of Squid? >>>> >>>> I use Squid 2.7 >>> >>> >>> >>> Output of "squid -v" please. >> >> # squid -v >> Squid Cache: Version 2.7.STABLE9 >> configure options: '--prefix=/usr' '--exec_prefix=/usr' >> '--bindir=/usr/sbin' '--sbindir=/usr/sbin' >> '--libexecdir=/usr/lib/squid' '--sysconfdir=/etc/squid' >> '--localstatedir=/var/spool/squid' '--datadir=/usr/share/squid' >> '--enable-async-io' '--with-pthreads' >> '--enable-storeio=ufs,aufs,coss,diskd,null' '--enable-linux-netfilter' >> '--enable-arp-acl' '--enable-epoll' >> '--enable-removal-policies=lru,heap' '--enable-snmp' >> '--enable-delay-pools' '--enable-htcp' '--enable-cache-digests' >> '--enable-underscores' '--enable-referer-log' '--enable-useragent-log' >> '--enable-auth=basic,digest,ntlm,negotiate' >> '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-carp' >> '--enable-follow-x-forwarded-for' '--with-large-files' >> '--with-maxfd=65536' 'amd64-debian-linux' >> 'build_alias=amd64-debian-linux' 'host_alias=amd64-debian-linux' >> 'target_alias=amd64-debian-linux' 'CFLAGS=-Wall -g -O2' >> 'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS=' >> >>> >>> >>>>> What do you mean by "thousands of connections". 1's of tousands? 10's >>>>> of >>>>> thousands? 100's of thousands? >>>> >>>> # netstat -nat|grep -i "80"|wc -l >>>> the result vary from 4651 to 9404 >>>> >>>>>> There are one ten of all connections can not establish in TCP layer, >>>>>> because squid does not respond SYN-ACK to client's SYN packet. How can >>>>>> I solve it? >>>>>> Thanks >>>>> >>>>> >>>>> Check ulimit settings for Squid? >>>>> >>>> # ulimit -a >>>> core file size (blocks, -c) 0 >>>> data seg size (kbytes, -d) unlimited >>>> scheduling priority (-e) 20 >>>> file size (blocks, -f) unlimited >>>> pending signals (-i) 16382 >>>> max locked memory (kbytes, -l) 64 >>>> max memory size (kbytes, -m) unlimited >>>> open files (-n) 655360 >>>> pipe size (512 bytes, -p) 8 >>>> POSIX message queues (bytes, -q) 819200 >>>> real-time priority (-r) 0 >>>> stack size (kbytes, -s) 8192 >>>> cpu time (seconds, -t) unlimited >>>> max user processes (-u) unlimited >>>> virtual memory (kbytes, -v) unlimited >>>> file locks (-x) unlimited >>>> >>>>> Check your cache.log for messages about running out of filedescriptors? >>>> >>>> I set my limit.conf to: >>>> root soft nofile 655360 >>>> root hard nofile 655360 >>> >>> >>> That does not answer the question. Squid may have been built or >>> configured >>> with a limit of less than 655360 filedescriptors. >>> cache.log should tell you if Squid is reaching some limit like this. >> >> my cache.log: >> 2013/02/15 8:30:10| Starting Squid Cache version 2.7.STABLE9 for >> x86_64-debian-linux-gnu... >> 2013/02/15 8:30:10| Process ID 8136 >> 2013/02/15 8:30:10| With 2048 file descriptors available >> 2013/02/15 8:30:10| Using epoll for the IO loop >> 2013/02/15 8:30:10| DNS Socket created at 0.0.0.0, port 6450, FD 6 >> 2013/02/15 8:30:10| Adding nameserver 8.8.8.8 from /etc/resolv.conf >> 2013/02/15 8:30:10| User-Agent logging is disabled. >> 2013/02/15 8:30:10| Referer logging is disabled. >> 2013/02/15 8:30:10| logfileOpen: opening log /var/log/squid/access.log >> 2013/02/15 8:30:10| Unlinkd pipe opened on FD 12 >> 2013/02/15 8:30:10| Swap maxSize 8192 + 8388608 KB, estimated 645907 >> objects >> 2013/02/15 8:30:10| Target number of buckets: 32295 >> 2013/02/15 8:30:10| Using 32768 Store buckets >> 2013/02/15 8:30:10| Max Mem size: 8388608 KB >> 2013/02/15 8:30:10| Max Swap size: 8192 KB >> 2013/02/15 8:30:10| Local cache digest enabled; rebuild/rewrite every >> 3600/3600 sec >> 2013/02/15 8:30:10| logfileOpen: opening log /var/log/squid/store.log >> 2013/02/15 8:30:10| Rebuilding storage in /var/spool/squid (CLEAN) >> 2013/02/15 8:30:10| Using Least Load store dir selection >> 2013/02/15 8:30:10| Set Current Directory to /var/spool/squid >> 2013/02/15 8:30:10| Loaded Icons. >> 2013/02/15 8:30:10| Accepting accelerated HTTP connections at 0.0.0.0, >> port 80, FD 14. >> 2013/02/15 8:30:10| Accepting ICP messages at 0.0.0.0, port 3130, FD 15. >> 2013/02/15 8:30:10| HTCP Disabled. >> 2013/02/15 8:30:10| WCCP Disabled. >> 2013/02/15 8:30:10| Configuring localhost Parent localhost/12080/0 >> 2013/02/15 8:30:10| Ready to serve requests. >> 2013/02/15 8:30:10| Done reading /var/spool/squid swaplog (11 entries) >> 2013/02/15 8:30:10| Finished rebuilding storage from disk. >> 2013/02/15 8:30:10| 11 Entries scanned >> 2013/02/15 8:30:10| 0 Invalid entries. >> 2013/02/15 8:30:10| 0 With invalid flags. >> 2013/02/15 8:30:10| 11 Objects loaded. >> 2013/02/15 8:30:10| 0 Objects expired. >> 2013/02/15 8:30:10| 0 Objects cancelled. >> 2013/02/15 8:30:10| 0 Duplicate URLs purged. >> 2013/02/15 8:30:10| 0 Swapfile clashes avoided. >> 2013/02/15 8:30:10| Took 0.3 seconds ( 41.8 objects/sec). >> 2013/02/15 8:30:10| Beginning Validation Procedure >> 2013/02/15 8:30:10| Completed Validation Procedure >> 2013/02/15 8:30:10| Validated 11 Entries >> 2013/02/15 8:30:10| store_swap_size = 44k >> 2013/02/15 8:30:11| storeLateRelease: released 0 objects >> 2013/02/15 8:30:35| CACHEMGR: <unknown>@127.0.0.1 requesting 'info' >> 2013/02/15 8:30:39| CACHEMGR: <unknown>@127.0.0.1 requesting 'info' >> 2013/02/15 8:30:40| CACHEMGR: <unknown>@127.0.0.1 requesting 'info' >> 2013/02/15 8:30:42| CACHEMGR: <unknown>@127.0.0.1 requesting 'info' >> 2013/02/15 8:30:44| CACHEMGR: <unknown>@127.0.0.1 requesting 'info' >> 2013/02/15 8:33:10| CACHEMGR: <unknown>@127.0.0.1 requesting 'info' >> >> when connection timeout error occurred, I do not see error about file >> descriptor. >> >> my /etc/sysclt.conf >> net.ipv4.tcp_syncookies = 1 >> net.ipv4.tcp_tw_reuse = 1 >> net.ipv4.tcp_tw_recycle = 1 >> net.ipv4.tcp_fin_timeout = 30 >> fs.file-max = 65536 >> >> my squid.conf has: >> max_filedescriptors 2048 >> >> and my squidclient says: >> squidclient -p 80 mgr:info | grep "file desc" >> Maximum number of file descriptors: 2048 >> Available number of file descriptors: 1651 >> Reserved number of file descriptors: 100 > > > There you go then. Squid is not permitted to _use_ more than 1651 FD. Every > client TCP connection uses at least 1, sometimes 2 FD. > When all the FD are used up Squid waits until some are free'd before > accepting more client connections. > > With "from 4651 to 9404" I would set your max_filedescriptors to at least > 18000. It can be anything up to the ulimit max. I set max_filedescriptors to 655360, and confirm that ulimit -n is also 655360. After restart Squid, I observe that rate of connection timeout in client is still about 10%, and no additional error such as filedescriptors error found in cache.log or dmesg. any ideas? thanks > > Amos
