Hi, Any news on this topic? Unfortunately, RAM is running full within days and at the moment, our workaround is to do a reboot ... We would appreciate any other solution! thanks, peter On Wed, May 11, 2011 at 3:47 PM, guest01 <guest01@xxxxxxxxx> wrote: > On Wed, May 11, 2011 at 10:47 AM, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote: >> On 11/05/11 19:19, guest01 wrote: >>> >>> Hi, >>> >>> I am currently using squid 3.1.12 as forward-proxy without >>> harddisk-caching (only RAM is used for caching). Each server is >>> running on RHEL5.5 and is pretty strong (16 CPUs, 28GB RAM), but each >>> servers starts swapping a few days after start. The workaround at the >>> moment is to reboot the server once a week, which I don't really like. >>> But swapping leads to serious side effects, e.g. performance troubles, >>> ... >>> >>> way too much swapping: >>> http://imageshack.us/m/52/6149/memoryday.png >>> >>> I already read a lot of posts and mails for similar problems, but >>> unfortunately, I was not able to solve this problem. I added following >>> infos to my squid.conf-file: >>> # cache specific settings >>> cache_replacement_policy heap LFUDA >>> cache_mem 1600 MB >>> memory_replacement_policy heap LFUDA >>> maximum_object_size_in_memory 2048 KB >>> memory_pools off >>> cache_swap_low 85 >>> cache_swap_high 90 >>> >>> (There are four squid instances per server, which means that 1600*4 = >>> 6400MB RAM used for caching, which is not even 1/4 of the total >>> available amount of RAM. Plenty enough, don't you think?) >> >> Not that is "for HTTP object caching", emphasis on *caching* and "HTTP >> object". In-transit objects and non-HTTP caches (Ip cache, domain name >> cache, persistent connections cache, client database, via/fwd database, >> network performance cache, auth caches, external ACL caches) and the indexes >> for all those caches use other memory. >> >> Then again they should all be using no more than a few GB combined. So you >> may have hit a new leak (all the known ones are resolved before 3.1.12). > > Ok, very strange. But at least it is reproducible, it takes about a > week until squid is starting to swap ... > http://img191.imageshack.us/img191/9615/memorymonth.png > >>> >>> Very strange are the negative values (Memory usage for squid via >>> mallinfo():) from the output below. Maybe that is a reason for running >>> out of RAM? >> >> mallinfo() sucks badly when going above 2GB of RAM. It can be ignored. >> >> The section underneath it "Memory accounted for:" is Squids own accounting >> and more of a worry. It should not have had negatives since before 3.1.10. >> >>> >>> HTTP/1.0 200 OK >>> Server: squid/3.1.12 >>> Mime-Version: 1.0 >>> Date: Wed, 11 May 2011 07:06:10 GMT >>> Content-Type: text/plain >>> Expires: Wed, 11 May 2011 07:06:10 GMT >>> Last-Modified: Wed, 11 May 2011 07:06:10 GMT >>> X-Cache: MISS from xlsqip03_1 >>> Via: 1.0 xlsqip03_1 (squid/3.1.12) >>> Connection: close >>> >>> Squid Object Cache: Version 3.1.12 >>> Start Time: Wed, 27 Apr 2011 11:01:13 GMT >>> Current Time: Wed, 11 May 2011 07:06:10 GMT >>> Connection information for squid: >>> Number of clients accessing cache: 1671 >>> Number of HTTP requests received: 16144359 >>> Number of ICP messages received: 0 >>> Number of ICP messages sent: 0 >>> Number of queued ICP replies: 0 >>> Number of HTCP messages received: 0 >>> Number of HTCP messages sent: 0 >>> Request failure ratio: 0.00 >>> Average HTTP requests per minute since start: 810.3 >>> Average ICP messages per minute since start: 0.0 >>> Select loop called: 656944758 times, 1.820 ms avg >>> Cache information for squid: >>> Hits as % of all requests: 5min: 17.4%, 60min: 18.2% >>> Hits as % of bytes sent: 5min: 45.6%, 60min: 39.9% >>> Memory hits as % of hit requests: 5min: 86.1%, 60min: 88.9% >>> Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0% >>> Storage Swap size: 0 KB >>> Storage Swap capacity: 0.0% used, 0.0% free >>> Storage Mem size: 1622584 KB >>> Storage Mem capacity: 100.0% used, 0.0% free >> >> Okay 1.6 GB of RAM used for caching HTTP objects. Fully used. >> >>> Mean Object Size: 0.00 KB >> >> Problem #1. It *may* be Squid not accounting for the memory objects in the >> mean. >> >>> Requests given to unlinkd: 0 >>> Median Service Times (seconds) 5 min 60 min: >>> HTTP Requests (All): 0.01648 0.01235 >>> Cache Misses: 0.05046 0.04277 >>> Cache Hits: 0.00091 0.00091 >>> Near Hits: 0.01469 0.01745 >>> Not-Modified Replies: 0.00000 0.00091 >>> DNS Lookups: 0.00190 0.00190 >>> ICP Queries: 0.00000 0.00000 >>> Resource usage for squid: >>> UP Time: 1195497.286 seconds >>> CPU Time: 22472.507 seconds >>> CPU Usage: 1.88% >>> CPU Usage, 5 minute avg: 5.38% >>> CPU Usage, 60 minute avg: 5.44% >>> Process Data Segment Size via sbrk(): 3145032 KB >>> Maximum Resident Size: 0 KB >>> Page faults with physical i/o: 8634 >>> Memory usage for squid via mallinfo(): >>> Total space in arena: -1049140 KB >>> Ordinary blocks: -1277813 KB 87831 blks >>> Small blocks: 0 KB 0 blks >>> Holding blocks: 2240 KB 5 blks >>> Free Small blocks: 0 KB >>> Free Ordinary blocks: 228673 KB >>> Total in use: -1275574 KB 122% >>> Total free: 228674 KB -22% >>> Total size: -1046900 KB >>> Memory accounted for: >>> Total accounted: -1375357 KB 131% >>> memPool accounted: 2818947 KB -269% >>> memPool unaccounted: -3865847 KB 0% >>> memPoolAlloc calls: 111 >>> memPoolFree calls: 8322084644 >>> File descriptor usage for squid: >>> Maximum number of file descriptors: 1024 >>> Largest file desc currently in use: 563 >>> Number of file desc currently in use: 472 >>> Files queued for open: 0 >>> Available number of file descriptors: 552 >>> Reserved number of file descriptors: 100 >>> Store Disk files open: 0 >>> Internal Data Structures: >>> 96996 StoreEntries >>> 96996 StoreEntries with MemObjects >>> 96980 Hot Object Cache Items >>> 0 on-disk objects >>> >>> Has anyone experienced similar things or does even know a solution? >> >> Not since we fixed Squid's capacity to calculate storage used by >2GB >> objects. >> >> Can you find the actual maximum and average object size for the cached >> objects in that Squid? (the mgr:vm_objects report should have all their >> details, "inmem_hi:" being the object size) > > Unfortunately, we had to reboot this server. At the moment there is no > swapping. But the mean object size is still 0.0kb: > xlsqip03:/root>/opt/squid/bin/squidclient -h 10.122.125.4 > cache_object://10.122.125.4/info > HTTP/1.0 200 OK > Server: squid/3.1.12 > Mime-Version: 1.0 > Date: Wed, 11 May 2011 13:33:58 GMT > Content-Type: text/plain > Expires: Wed, 11 May 2011 13:33:58 GMT > Last-Modified: Wed, 11 May 2011 13:33:58 GMT > X-Cache: MISS from xlsqip03_1 > Via: 1.0 xlsqip03_1 (squid/3.1.12) > Connection: close > > Squid Object Cache: Version 3.1.12 > Start Time: Wed, 11 May 2011 09:35:49 GMT > Current Time: Wed, 11 May 2011 13:33:58 GMT > Connection information for squid: > Number of clients accessing cache: 814 > Number of HTTP requests received: 504290 > Number of ICP messages received: 0 > Number of ICP messages sent: 0 > Number of queued ICP replies: 0 > Number of HTCP messages received: 0 > Number of HTCP messages sent: 0 > Request failure ratio: 0.00 > Average HTTP requests per minute since start: 2117.6 > Average ICP messages per minute since start: 0.0 > Select loop called: 15248273 times, 0.937 ms avg > Cache information for squid: > Hits as % of all requests: 5min: 17.6%, 60min: 15.8% > Hits as % of bytes sent: 5min: 38.0%, 60min: 34.3% > Memory hits as % of hit requests: 5min: 88.2%, 60min: 87.0% > Disk hits as % of hit requests: 5min: 0.1%, 60min: 0.1% > Storage Swap size: 0 KB > Storage Swap capacity: 0.0% used, 0.0% free > Storage Mem size: 1622020 KB > Storage Mem capacity: 100.0% used, 0.0% free > Mean Object Size: 0.00 KB > Requests given to unlinkd: 0 > Median Service Times (seconds) 5 min 60 min: > HTTP Requests (All): 0.02190 0.01847 > Cache Misses: 0.04277 0.04776 > Cache Hits: 0.00091 0.00091 > Near Hits: 0.02069 0.02069 > Not-Modified Replies: 0.00091 0.00091 > DNS Lookups: 0.00190 0.00094 > ICP Queries: 0.00000 0.00000 > Resource usage for squid: > UP Time: 14288.578 seconds > CPU Time: 642.922 seconds > CPU Usage: 4.50% > CPU Usage, 5 minute avg: 5.70% > CPU Usage, 60 minute avg: 5.27% > Process Data Segment Size via sbrk(): 3035580 KB > Maximum Resident Size: 0 KB > Page faults with physical i/o: 0 > Memory usage for squid via mallinfo(): > Total space in arena: -1158592 KB > Ordinary blocks: -1187512 KB 8338 blks > Small blocks: 0 KB 0 blks > Holding blocks: 2240 KB 5 blks > Free Small blocks: 0 KB > Free Ordinary blocks: 28920 KB > Total in use: -1185273 KB 103% > Total free: 28921 KB -3% > Total size: -1156352 KB > Memory accounted for: > Total accounted: -1290919 KB 112% > memPool accounted: 2903385 KB -251% > memPool unaccounted: -4059737 KB 0% > memPoolAlloc calls: 111 > memPoolFree calls: 263492993 > File descriptor usage for squid: > Maximum number of file descriptors: 1024 > Largest file desc currently in use: 372 > Number of file desc currently in use: 298 > Files queued for open: 0 > Available number of file descriptors: 726 > Reserved number of file descriptors: 100 > Store Disk files open: 0 > Internal Data Structures: > 105790 StoreEntries > 105790 StoreEntries with MemObjects > 105780 Hot Object Cache Items > 0 on-disk objects > > xlsqip03:/root>/opt/squid/bin/squidclient -h 10.122.125.4 > cache_object://10.122.125.4/vm_objects | head -n 50 > HTTP/1.0 200 OK > Server: squid/3.1.12 > Mime-Version: 1.0 > Date: Wed, 11 May 2011 13:34:39 GMT > Content-Type: text/plain > Expires: Wed, 11 May 2011 13:34:39 GMT > Last-Modified: Wed, 11 May 2011 13:34:39 GMT > X-Cache: MISS from xlsqip03_1 > Via: 1.0 xlsqip03_1 (squid/3.1.12) > Connection: close > > KEY 00C06887AD5B919595711834A5195511 > STORE_OK IN_MEMORY SWAPOUT_NONE PING_DONE > CACHABLE,DISPATCHED,VALIDATED > LV:1305110370 LU:1305110371 LM:1157000708 EX:-1 > 0 locks, 0 clients, 1 refs > Swap Dir -1, File 0XFFFFFFFF > GET http://www.hardwaremuseum.at/mitarbeiter/keske.jpg > inmem_lo: 0 > inmem_hi: 37138 > swapout: 0 bytes queued > > KEY 00A02D3964813C2CC4A7D6553D2CF0A7 > STORE_OK IN_MEMORY SWAPOUT_NONE PING_DONE > CACHABLE,DISPATCHED,VALIDATED > LV:1305110512 LU:1305111016 LM:1275291526 EX:-1 > 0 locks, 0 clients, 2 refs > Swap Dir -1, File 0XFFFFFFFF > GET http://nyx.at/bikeboard/images/design/icon_hotels.png > inmem_lo: 0 > inmem_hi: 1507 > swapout: 0 bytes queued > > KEY 0020B363AC66E4EAF4B812DCCB561B99 > STORE_OK IN_MEMORY SWAPOUT_NONE PING_DONE > CACHABLE,DISPATCHED,VALIDATED > LV:1305110827 LU:1305110833 LM:966956701 EX:-1 > 0 locks, 0 clients, 1 refs > Swap Dir -1, File 0XFFFFFFFF > GET http://windfind.ivwbox.de/blank.gif > inmem_lo: 0 > inmem_hi: 296 > swapout: 0 bytes queued > > KEY 00C09B8188CC913834CF292CC2A4D72E > STORE_OK IN_MEMORY SWAPOUT_NONE PING_DONE > CACHABLE,DISPATCHED,VALIDATED > LV:1305110909 LU:1305110991 LM:1153932790 EX:-1 > 0 locks, 0 clients, 3 refs > Swap Dir -1, File 0XFFFFFFFF > > xlsqip03:/root>/opt/squid/bin/squidclient -h 10.122.125.4 > cache_object://10.122.125.4/vm_objects | grep inmem_hi > inmem_hi: 37138 > inmem_hi: 1507 > inmem_hi: 296 > inmem_hi: 747 > inmem_hi: 5149 > inmem_hi: 1449 > inmem_hi: 5231 > inmem_hi: 3594 > inmem_hi: 7911 > inmem_hi: 699 > inmem_hi: 141554 > inmem_hi: 8997 > inmem_hi: 977 > inmem_hi: 37114 > inmem_hi: 6229 > inmem_hi: 79923 > inmem_hi: 1016 > inmem_hi: 11075 > inmem_hi: 1475 > inmem_hi: 6113 > inmem_hi: 3118 > inmem_hi: 26894 > inmem_hi: 1433 > inmem_hi: 2763 > inmem_hi: 2388 > inmem_hi: 287 > inmem_hi: 2481 > inmem_hi: 297 > inmem_hi: 3399 > inmem_hi: 217 > inmem_hi: 1549 > inmem_hi: 41865 > inmem_hi: 4920 > inmem_hi: 7857 > inmem_hi: 1608 > inmem_hi: 5666 > inmem_hi: 9507 > inmem_hi: 8653 > inmem_hi: 11728 > inmem_hi: 6106 > inmem_hi: 452 > inmem_hi: 2740 > inmem_hi: 1122 > inmem_hi: 2183 > inmem_hi: 3861 > inmem_hi: 14803 > inmem_hi: 861 > inmem_hi: 26080 > inmem_hi: 28724 > inmem_hi: 2094 > inmem_hi: 428 > inmem_hi: 1445 > inmem_hi: 468 > inmem_hi: 21754 > > I also checked our Test-Squid with Squid 3.2.0.7: > squid@xlsqit01 /etc/squid $ /opt/squid/bin/squidclient -h 10.122.125.2 > cache_object://10.122.125.2/info > HTTP/1.1 200 OK > Server: squid/3.2.0.7 > Mime-Version: 1.0 > Date: Wed, 11 May 2011 13:40:19 GMT > Content-Type: text/plain > Expires: Wed, 11 May 2011 13:40:19 GMT > Last-Modified: Wed, 11 May 2011 13:40:19 GMT > Connection: close > > Squid Object Cache: Version 3.2.0.7 > Start Time: Fri, 29 Apr 2011 07:56:13 GMT > Current Time: Wed, 11 May 2011 13:40:19 GMT > Connection information for squid: > Number of clients accessing cache: 28 > Number of HTTP requests received: 44628 > Number of ICP messages received: 0 > Number of ICP messages sent: 0 > Number of queued ICP replies: 0 > Number of HTCP messages received: 0 > Number of HTCP messages sent: 0 > Request failure ratio: 0.00 > Average HTTP requests per minute since start: 2.5 > Average ICP messages per minute since start: 0.0 > Select loop called: 1549800905 times, 1.365 ms avg > Cache information for squid: > Hits as % of all requests: 5min: 0.0%, 60min: 0.0% > Hits as % of bytes sent: 5min: 100.0%, 60min: 57.0% > Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0% > Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0% > Storage Swap size: 0 KB > Storage Swap capacity: 0.0% used, 0.0% free > Storage Mem size: 30452 KB > Storage Mem capacity: 5.9% used, 94.1% free > Mean Object Size: 0.00 KB > Requests given to unlinkd: 0 > Median Service Times (seconds) 5 min 60 min: > HTTP Requests (All): 0.00189 0.00000 > Cache Misses: 0.00000 0.12262 > Cache Hits: 0.00000 0.00000 > Near Hits: 0.00000 0.00000 > Not-Modified Replies: 0.00000 0.00000 > DNS Lookups: 0.00047 0.00047 > ICP Queries: 0.00000 0.00000 > Resource usage for squid: > UP Time: 1057446.006 seconds > CPU Time: 366.647 seconds > CPU Usage: 0.03% > CPU Usage, 5 minute avg: 0.08% > CPU Usage, 60 minute avg: 0.04% > Process Data Segment Size via sbrk(): 63720 KB > Maximum Resident Size: 0 KB > Page faults with physical i/o: 0 > Memory usage for squid via mallinfo(): > Total space in arena: 63984 KB > Ordinary blocks: 63645 KB 22 blks > Small blocks: 0 KB 0 blks > Holding blocks: 18480 KB 12 blks > Free Small blocks: 0 KB > Free Ordinary blocks: 339 KB > Total in use: 339 KB 0% > Total free: 339 KB 0% > Total size: 82464 KB > Memory accounted for: > Total accounted: 54669 KB 66% > memPool accounted: 54669 KB 66% > memPool unaccounted: 27795 KB 34% > memPoolAlloc calls: 9115016 > memPoolFree calls: 13310573 > File descriptor usage for squid: > Maximum number of file descriptors: 32768 > Largest file desc currently in use: 60 > Number of file desc currently in use: 41 > Files queued for open: 0 > Available number of file descriptors: 32727 > Reserved number of file descriptors: 200 > Store Disk files open: 0 > Internal Data Structures: > 1461 StoreEntries > 1461 StoreEntries with MemObjects > 1459 Hot Object Cache Items > 0 on-disk objects > squid@xlsqit01 /etc/squid $ /opt/squid/bin/squidclient -h 10.122.125.2 > cache_object://10.122.125.2/info | grep -i mean > Mean Object Size: 0.00 KB > > Seems to be the same problem: > squid@xlsqit01 /etc/squid $ /opt/squid/bin/squidclient -h 10.122.125.2 > cache_object://10.122.125.2/vm_objects | grep inmem_hi > inmem_hi: 410 > inmem_hi: 4859 > inmem_hi: 5927 > inmem_hi: 1494 > inmem_hi: 766 > inmem_hi: 681 > inmem_hi: 2215 > inmem_hi: 423 > inmem_hi: 236 > inmem_hi: 386 > inmem_hi: 3899 > inmem_hi: 1060 > inmem_hi: 1093 > inmem_hi: 39037 > inmem_hi: 2321 > inmem_hi: 53163 > inmem_hi: 822 > inmem_hi: 272 > inmem_hi: 420 > inmem_hi: 421 > inmem_hi: 381 > inmem_hi: 2186 > inmem_hi: 307 > inmem_hi: 394 > > Very strange indeed ... Maybe it is related to our RHEL5.5 environment > or compile options? Otherwise my guess is that many people would have > the same problem? > > xlsqip03:/root>/opt/squid/sbin/squid -v > Squid Cache: Version 3.1.12 > configure options: '--enable-ssl' '--enable-icap-client' > '--sysconfdir=/etc/squid' '--enable-async-io' '--enable-snmp' > '--enable-poll' '--with-maxfd=32768' '--enable-storeio=aufs' > '--enable-removal-policies=heap,lru' '--enable-epoll' > '--disable-ident-lookups' '--enable-truncate' > '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' > '--with-default-user=squid' '--prefix=/opt/squid' '--enable-auth=basic > digest ntlm negotiate' > '-enable-negotiate-auth-helpers=squid_kerb_auth' > --with-squid=/home/squid/squid-3.1.12 --enable-ltdl-convenience > > squid@xlsqit01 /etc/squid $ /opt/squid/sbin/squid -v > Squid Cache: Version 3.2.0.7 > configure options: '--enable-ssl' '--enable-icap-client' > '--sysconfdir=/etc/squid' '--enable-async-io' '--enable-snmp' > '--enable-poll' '--with-maxfd=32768' '--enable-storeio=aufs' > '--enable-removal-policies=heap,lru' '--enable-epoll' > '--disable-ident-lookups' '--enable-truncate' > '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' > '--with-default-user=squid' '--prefix=/opt/squid' > '-enable-negotiate-auth-helpers=squid_kerb_auth' > --enable-ltdl-convenience > > We are basically only doing authentication (Kerberos, LDAP, NTLM) and > ICAP authorization. > Is there anything else I can do? > > thanks and regards! > Peter >
