Hello folks, Since a long time I'm experiencing a problem with Squid and after a long work, I didn't find the point. I believe this must be recurring discussion, but seems to be a kind of particular tunning on my enviroments to get my problem solved, so here we go with my history: Around two months ago, I started a patterned configuration of Squid on 15 servers over 10 different ISP's which I'm network manager. The number of users served by each server varies around 150 to 500 users behind them. The bandwidth on each ISP varies around 4 to 30 Mbps. I know that this can be a big range of variation, but I'm telling you that, only to give an overview about the dimension of enviroments we are talking about. On the first 10 or 15 days, all of then were running perfectly... no matter, no issues! After that till now, 4 of these(other 11 are still working perfectly) servers starts with the same symptom: - The browsing becomes lazy till it almost stop's(in some cases get back working with no intervention - intermittently) The proxies are running in transparent mode, so if I just disable the redirect on netfilter, making the traffic goes directly to the net(masquerading), the browsing get fast again! I'm using almost the same config in all the servers, basically only changing the cache_dir based on the size of the disk. The main intention of those servers are to save bandwidht usage, but keeping the balance with the response time to the users, in a way that using proxy must still be faster than directly connect. All the servers are running Squid v.2.6.stable22. I've already checked CPU Load, Memory usage, disk I/O, and in my humble analyse, I didn't find the point. I'll put here some info's about the system of one of those problematic servers, so if anyone could find some clue, I'll be very gratefull. ================================================================== => Hardware Configs -------------------------------- CPU: Intel Core 2 Quad - 2.66GHz RAM: 8 Gb => OS -------------------------------- Linux Slackware v.13 - Kernel 2.6.29.6-smp => SQUID CONF (Squid v.2.6.stable22) -------------------------------- acl all src 0.0.0.0/0.0.0.0 acl manager proto cache_object acl localhost src 127.0.0.1/255.255.255.255 acl to_localhost dst 127.0.0.0/8 acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access deny !Safe_ports http_access deny CONNECT !SSL_ports acl our_networks src xxx.xxx.xxx.xxx/16 acl this_server src yyy.yyy.yyy.yyy/32 http_access allow our_networks http_access allow localhost http_access deny all icp_access allow all http_port 3128 transparent hierarchy_stoplist cgi-bin ? cache_mem 512 MB maximum_object_size_in_memory 384 KB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA cache_dir aufs /cache 500000 64 256 maximum_object_size 400 MB access_log /usr/local/squid/var/logs/access.log squid cache_log /usr/local/squid/var/logs/cache.log cache_store_log none logfile_rotate 3 acl QUERY urlpath_regex cgi-bin \? cache deny QUERY refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern . 0 20% 4320 acl apache rep_header Server ^Apache broken_vary_encoding allow apache cache_mgr support@xxxxxxxxxxxxxxx visible_hostname yyy.yyy.yyy.yyy snmp_port 3401 acl snmppublic snmp_community public snmp_access allow snmppublic this_server snmp_access deny all snmp_incoming_address 0.0.0.0 snmp_outgoing_address 255.255.255.255 coredump_dir /cache => Squid CACHE MANAGER INFO: -------------------------------- Squid Object Cache: Version 2.6.STABLE22 Start Time: Sat, 02 Jan 2010 18:08:57 GMT Current Time: Sat, 02 Jan 2010 21:10:19 GMT Connection information for squid: Number of clients accessing cache: 9 Number of HTTP requests received: 277856 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 1532.1 Average ICP messages per minute since start: 0.0 Select loop called: 3581679 times, 3.038 ms avg Cache information for squid: Request Hit Ratios: 5min: 35.5%, 60min: 39.7% Byte Hit Ratios: 5min: 17.0%, 60min: 17.0% Request Memory Hit Ratios: 5min: 9.0%, 60min: 5.7% Request Disk Hit Ratios: 5min: 52.4%, 60min: 48.6% Storage Swap size: 56684352 KB Storage Mem size: 513584 KB Mean Object Size: 20.48 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.18699 0.15888 Cache Misses: 0.30459 0.30459 Cache Hits: 0.01309 0.00562 Near Hits: 0.32154 0.08265 Not-Modified Replies: 0.00286 0.00286 DNS Lookups: 0.13042 0.12472 ICP Queries: 0.00000 0.00000 Resource usage for squid: UP Time: 10881.453 seconds CPU Time: 127.668 seconds CPU Usage: 1.17% CPU Usage, 5 minute avg: 1.30% CPU Usage, 60 minute avg: 0.91% Process Data Segment Size via sbrk(): 851000 KB Maximum Resident Size: 0 KB Page faults with physical i/o: 0 Memory usage for squid via mallinfo(): Total space in arena: 851000 KB Ordinary blocks: 850398 KB 31794 blks Small blocks: 0 KB 0 blks Holding blocks: 27348 KB 5 blks Free Small blocks: 0 KB Free Ordinary blocks: 601 KB Total in use: 877746 KB 100% Total free: 601 KB 0% Total size: 878348 KB Memory accounted for: Total accounted: 726322 KB memPoolAlloc calls: 51472567 memPoolFree calls: 44989921 File descriptor usage for squid: Maximum number of file descriptors: 16384 Largest file desc currently in use: 1153 Number of file desc currently in use: 972 Files queued for open: 0 Available number of file descriptors: 15412 Reserved number of file descriptors: 100 Store Disk files open: 69 IO loop method: epoll Internal Data Structures: 2767942 StoreEntries 36519 StoreEntries with MemObjects 36391 Hot Object Cache Items 2767222 on-disk objects => 'top' COMMAND OUTPUT: -------------------------------- top - 18:14:18 up 2 days, 23:34, 1 user, load average: 0.26, 0.17, 0.08 Tasks: 113 total, 1 running, 112 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.3%sy, 0.0%ni, 98.7%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 0.3%sy, 0.0%ni, 97.7%id, 2.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.3%us, 0.0%sy, 0.0%ni, 99.3%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.3%us, 0.0%sy, 0.0%ni, 99.0%id, 0.3%wa, 0.0%hi, 0.3%si, 0.0%st Mem: 8305904k total, 3632340k used, 4673564k free, 525752k buffers Swap: 6297440k total, 0k used, 6297440k free, 1980300k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12561 nobody 20 0 874m 863m 2372 S 0 10.6 2:09.52 squid => 'free' COMMAND OUTPUT: -------------------------------- total used free shared buffers cached Mem: 8305904 3648668 4657236 0 527276 1991908 -/+ buffers/cache: 1129484 7176420 Swap: 6297440 0 6297440 => 'vmstat' COMMAND OUTPUT: -------------------------------- root@proxy:/usr/local/squid/etc# vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 4655236 527384 1993244 0 0 12 37 8 24 0 0 98 1 0 0 0 4654980 527384 1993292 0 0 0 504 4612 368 0 0 99 0 0 0 0 4654848 527424 1993524 0 0 40 0 4816 671 0 0 98 2 0 0 0 4654724 527436 1993712 0 0 168 0 4740 522 0 0 99 1 0 0 0 4654352 527448 1993876 0 0 0 76 4818 487 0 0 100 0 0 0 0 4654344 527460 1993976 0 0 4 0 4718 500 0 0 100 0 0 0 0 4654220 527468 1994120 0 0 24 588 4579 398 0 0 99 1 0 0 0 4653848 527480 1994332 0 0 104 0 4853 627 0 0 99 1 0 0 0 4653360 527500 1994532 0 0 0 0 4909 683 0 0 100 0 0 0 0 4653212 527516 1994716 0 0 0 40 4877 447 0 0 100 0 0 0 0 4652964 527536 1994732 0 0 12 0 4721 518 0 0 100 0 0 0 0 4652964 527548 1994800 0 0 16 820 4810 708 0 1 99 1 0 0 0 4652716 527572 1994936 0 0 20 0 4696 628 0 0 99 1 ================================================================== So, with that info's, anyone would suggest something wrong? Sorry about the long mail, but I'm just trying to be detailed enough so you could help me. Best regards, -- Igor Luiz Oliveira de Souza Solic Tecnologia e Sistemas +55 (77) 8829-7678 igorluiz@xxxxxxxxxxxx
