Search squid archive

Re: Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

> On 26/11/2014 8:59 a.m., Doug Sampson wrote:
> >
> > Thanks, Amos, for your pointers.
> >
> > I've commented out all the fresh_patterns lines appearing above
> > the last two lines.
> >
> > I also have dropped diskd in favor of using aufs exclusively,
> > taking out the min-size parameter. I've commented out the
> > diskd_program support option. In the previous version of squid
> > (2.7) I had split the cache_dir into two types with great success
> > using coss and aufs. Previously I had only aufs and performance
> > wasn't where I wanted it. Apparently coss is no longer supported in
> > the 3.x version of squid atop FreeBSD.
> 
> COSS has been replaced with Rock storage type in Squid-3. They should
> be used in roughly similar ways in terms of traffic optimization.
> 
> >
> > The pathname for the cache swap logs have been fixed. Apparently
> > this came from a squid.conf example that I copied in parts. Would
> > this be the reason why we are seeing the error messages in
> > /var/log/messages regarding swapping mentioned in my original
> > post?
> 
> No. I think that is coming out of the OS kernel memory management. It
> uses the term "swap" as well in regards to disk backed virtual memory.
> 
> If your system is "swapping" (using that disk backed "swap memory")
> while running Squid then you will get terrible performance as a matter
> of course since the Squid cache index and cache_meme is often very
> large in RAM and accessed often.
> 
> >
> > The hierarchy_stoplist line has been stripped out as you say it is
> > deprecated.
> >
> > The mem .TSV file is attached herewith.
> >
> > Currently I have the cache_dir located on the OS disk and all of
> > the cache logging files on a second drive. Is this the optimal
> > setup of cache-dir and logs?
> 
> I would do it the other way around. Logs are appended with a small
> amount of data each transaction, whereas the main cache_dir has a
> relativey large % of the bandwidth throughput being written out to it
> constantly (less % in recent Squid, but still a lot). The dik most
> likely to die early is the one holding cache_dir.
> 
I'm still running into the issue of being out of available space in the swapfile on my system. I've attached another TSV file indicating the various type of memory being in use and whatnot. Is there anything in there that screams out?

Amos, you said earlier that it was the OS system that needed to be tuned. Are there any references to where I can fine-tune it for Squid usage? I looked here http://oss.org.cn/man/newsoft/squid/Squid_FAQ/FAQ-8.html#how-much-ram and I'm unable to figure out a way to decrease the amount of memory that I could use. I tried limiting cache_mem to 1344 MB from some higher value but that didn't work. What are some of the methods that FreeBSD 10.0 users using to limit the use of memory that Squid uses?

Sort of fixing the memory leaks, it looks like I need to consider the possibility of restarting the Squid service on a regular basis (i.e. at least once a week) in order to enable Squid to perform at an acceptable level and to avoid clogging /var/log/messages with such messages as follows:

<...>
+swap_pager_getswapspace(15): failed
+swap_pager_getswapspace(2): failed
+swap_pager_getswapspace(16): failed
+swap_pager_getswapspace(16): failed
<...>

Is this a common practice among squid admins to restart squid periodically?

Thank you!

~Doug
Current memory usage:
Pool Obj SizeChunksAllocatedIn UseIdleAllocations SavedRate
(bytes)KB/ch obj/ch(#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot(#) (KB) high (KB) high (hrs) %alloc(#) (KB) high (KB)(#) %cnt %vol(#)/sec
mem_node 4136 292390 1180982 1336651 4.76 86.464 292191 1180178 1336651 4.76 99.932 199 804 16173 1641718 1.322 21.088 0.001
Short Strings 40 1701206 66454 73289 4.76 4.865 1699725 66396 73289 4.76 99.913 1481 58 250 37674459 30.347 4.680 0.029
HttpHeaderEntry 56 796862 43579 47925 4.76 3.191 796197 43543 47925 4.76 99.917 665 37 163 10727306 8.641 1.866 0.008
HttpReply 280 74302 20317 22720 4.76 1.487 74275 20310 22720 4.76 99.964 27 8 135 754702 0.608 0.656 0.001
MemObject 240 74278 17409 19473 4.76 1.275 74273 17408 19473 4.76 99.993 5 2 121 187591 0.151 0.140 0.000
StoreEntry 104 79949 8120 9020 4.76 0.594 79944 8120 9020 4.76 99.994 5 1 53 198382 0.160 0.064 0.000
HttpHdrCc 96 50739 4757 5279 4.76 0.348 50725 4756 5279 4.76 99.972 14 2 41 602296 0.485 0.180 0.000
Medium Strings 128 37543 4693 4889 3.29 0.344 37426 4679 4889 3.29 99.688 117 15 48 2579153 2.078 1.025 0.002
cbdata MemBuf (9) 64 74339 4647 5197 4.76 0.340 74310 4645 5197 4.76 99.961 29 2 31 1332078 1.073 0.265 0.001
Long Strings 512 4170 2085 2085 0.04 0.153 4105 2053 2085 0.04 98.441 65 33 130 389656 0.314 0.620 0.000
LRU policy node 24 74220 1740 1946 4.76 0.127 74217 1740 1946 4.76 99.996 3 1 12 29622 0.024 0.002 0.000
4KB Strings 4096 394 1576 1576 4.94 0.115 310 1240 1576 4.94 78.680 84 336 340 123478 0.099 1.571 0.000
16K Buffer 16384 81 1296 1296 51.80 0.095 3 48 1296 51.80 3.704 78 1248 1296 262855 0.212 13.375 0.000
MD5 digest 16 79949 1250 1388 4.76 0.091 79944 1250 1388 4.76 99.994 5 1 9 327376 0.264 0.016 0.000
4K Buffer 4096 279 1116 1284 78.52 0.082 128 512 1284 78.52 45.878 151 604 1104 209870 0.169 2.670 0.000
cbdata IdleConnList (32) 4160 243 988 988 27.81 0.072 53 216 988 27.81 21.811 190 772 988 126976 0.102 1.640 0.000
cbdata clientReplyContext (18) 4320 204 861 861 51.45 0.063 59 249 861 51.45 28.922 145 612 861 368268 0.297 4.941 0.000
cbdata ClientSocketContext (17) 4256 204 848 848 51.45 0.062 59 246 848 51.45 28.922 145 603 848 368268 0.297 4.868 0.000
16KB Strings 16384 29 464 464 50.22 0.034 0 0 464 50.22 0.000 29 464 464 32235 0.026 1.640 0.000
1KB Strings 1024 441 441 441 6.60 0.032 401 401 441 6.60 90.930 40 40 62 106565 0.086 0.339 0.000
8K Buffer 8192 49 392 392 53.90 0.029 1 8 392 53.90 2.041 48 384 392 43308 0.035 1.102 0.000
HttpRequest 1744 204 348 348 51.45 0.025 59 101 348 51.45 28.922 145 247 348 368420 0.297 1.995 0.000
64K Buffer 65536 5 320 320 28.56 0.023 0 0 320 28.56 0.000 5 320 320 503 0.000 0.102 0.000
32K Buffer 32768 7 224 224 55.14 0.016 1 32 224 55.14 14.286 6 192 192 1304 0.001 0.133 0.000
Comm::Connection 192 1074 202 354 51.45 0.015 869 163 354 51.45 80.912 205 39 117 1720507 1.386 1.026 0.001
ipcache_entry 128 1006 126 126 30.49 0.009 920 115 126 30.49 91.451 86 11 11 37928 0.031 0.015 0.000
squidaio_small_bufs 4096 29 116 116 74.11 0.008 0 0 116 74.11 0.000 29 116 116 66345 0.053 0.844 0.000
cbdata ConnStateData (15) 440 253 109 124 78.52 0.008 97 42 124 78.52 38.340 156 68 118 165645 0.133 0.226 0.000
2K Buffer 2048 45 90 90 52.43 0.007 6 12 90 52.43 13.333 39 78 82 1684921 1.357 10.717 0.001
cbdata ClientHttpRequest (16) 312 204 63 63 51.45 0.005 59 18 63 51.45 28.922 145 45 63 368268 0.297 0.357 0.000
cbdata clientStreamNode (19) 128 292 37 51 51.45 0.003 118 15 51 51.45 40.411 174 22 34 734584 0.592 0.292 0.001
cbdata TunnelStateData (22) 224 163 36 36 51.45 0.003 55 13 36 51.45 33.742 108 24 36 82049 0.066 0.057 0.000
ClientInfo 432 41 18 18 5.66 0.001 41 18 18 5.66 100.000 0 0 0 0 0.000 0.000 0.000
MimeEntry 96 177 17 17 95.64 0.001 177 17 17 95.64 100.000 0 0 0 0 0.000 0.000 0.000
cbdata HttpStateData (29) 256 55 14 14 75.44 0.001 3 1 14 75.44 5.455 52 13 14 253803 0.204 0.202 0.000
cbdata ps_state (23) 248 48 12 12 77.86 0.001 0 0 12 77.86 0.000 48 12 12 336826 0.271 0.259 0.000
cbdata UFSStoreState (33) 176 63 11 11 6.67 0.001 0 0 11 6.67 0.000 63 11 11 121460 0.098 0.066 0.000
cbdata FwdState (27) 168 65 11 11 77.87 0.001 3 1 11 77.87 4.615 62 11 11 254597 0.205 0.133 0.000
cbdata store_client (26) 160 67 11 11 77.87 0.001 4 1 11 77.87 5.970 63 10 11 312925 0.252 0.155 0.000
squidaio_micro_bufs 128 64 8 8 6.67 0.001 0 0 8 6.67 0.000 64 8 8 212552 0.171 0.084 0.000
UFSStoreState::_queued_write 32 254 8 14 5.30 0.001 0 0 14 5.30 0.000 254 8 14 1381963 1.113 0.137 0.001
cbdata helper_server (8) 240 32 8 8 78.52 0.001 32 8 8 78.52 100.000 0 0 0 0 0.000 0.000 0.000
aio_ctrl 112 64 7 7 6.67 0.001 0 0 7 6.67 0.000 64 7 7 278926 0.225 0.097 0.000
aio_request 104 64 7 7 6.67 0.000 0 0 7 6.67 0.000 64 7 7 278926 0.225 0.090 0.000
cbdata DiskThreadsDiskFile (34) 88 63 6 6 6.67 0.000 0 0 6 6.67 0.000 63 6 6 121460 0.098 0.033 0.000
cbdata ConnOpener (25) 136 40 6 6 6.17 0.000 0 0 6 6.17 0.000 40 6 6 189679 0.153 0.080 0.000
cbdata BodyPipe (39) 152 32 5 5 54.43 0.000 1 1 5 54.43 3.125 31 5 5 8884 0.007 0.004 0.000
link_list 16 254 4 7 5.30 0.000 0 0 7 5.30 0.000 254 4 7 1385055 1.116 0.069 0.001
Acl::AndNode 152 24 4 4 95.64 0.000 24 4 4 95.64 100.000 0 0 0 0 0.000 0.000 0.000
cbdata CbDataList (30) 96 32 3 3 75.44 0.000 0 0 3 75.44 0.000 32 3 3 42060667 33.880 12.540 0.030
cbdata Logfile (11) 1120 2 3 3 95.64 0.000 2 3 3 95.64 100.000 0 0 0 0 0.000 0.000 0.000
cbdata ReadRequest (35) 72 29 3 3 74.11 0.000 0 0 3 74.11 0.000 29 3 3 66345 0.053 0.015 0.000
cbdata generic_cbdata (24) 32 64 2 2 6.67 0.000 0 0 2 6.67 0.000 64 2 2 156953 0.126 0.016 0.000
helper_request 40 48 2 2 77.86 0.000 0 0 2 77.86 0.000 48 2 2 38879 0.031 0.005 0.000
cbdata Tree (4) 208 9 2 2 95.64 0.000 9 2 2 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ev_entry 48 34 2 2 28.53 0.000 13 1 2 28.53 38.235 21 1 2 1480349 1.192 0.221 0.001
HttpHdrScTarget 88 17 2 2 54.25 0.000 13 2 2 54.25 76.471 4 1 1 111 0.000 0.000 0.000
acl_ip_data 96 14 2 2 95.64 0.000 13 2 2 95.64 92.857 1 1 1 7 0.000 0.000 0.000
cbdata IoResult (36) 40 29 2 2 74.11 0.000 0 0 2 74.11 0.000 29 2 2 66345 0.053 0.008 0.000
FwdServer 24 48 2 2 77.86 0.000 0 0 2 77.86 0.000 48 2 2 336826 0.271 0.025 0.000
UFSStoreState::_queued_read 40 26 2 2 79.20 0.000 0 0 2 79.20 0.000 26 2 2 3036 0.002 0.000 0.000
ACLStrategised 152 5 1 1 95.64 0.000 5 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ACLStrategised 152 5 1 1 95.64 0.000 5 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
cbdata RebuildState (12) 688 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 0 0.000 0.000 0.000
ACLSourceIP 136 5 1 1 95.64 0.000 5 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
fqdncache_entry 160 4 1 1 95.64 0.000 3 1 1 95.64 75.000 1 1 1 0 0.000 0.000 0.000
aio_thread 40 16 1 1 95.64 0.000 16 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
cbdata ErrorState (40) 240 2 1 1 74.81 0.000 0 0 1 74.81 0.000 2 1 1 969 0.001 0.001 0.000
cbdata CbDataList (2) 40 12 1 1 95.64 0.000 12 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
wordlist 16 30 1 1 95.64 0.000 12 1 1 95.64 40.000 18 1 1 48 0.000 0.000 0.000
cbdata ACLFilledChecklist (21) 408 1 1 1 95.64 0.000 1 1 1 95.64 100.000 0 0 1 368428 0.297 0.467 0.000
cbdata CbDataList (1) 40 10 1 1 95.64 0.000 10 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
dlink_node 24 16 1 1 77.86 0.000 0 0 1 77.86 0.000 16 1 1 40 0.000 0.000 0.000
cbdata PortCfg (5) 344 1 1 1 95.64 0.000 1 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
HttpHdrContRange 24 13 1 1 70.12 0.000 1 1 1 70.12 7.692 12 1 1 2500 0.002 0.000 0.000
ACLStrategised 152 2 1 1 95.64 0.000 2 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
Acl::NotNode 152 2 1 1 95.64 0.000 2 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ACLStrategised 152 2 1 1 95.64 0.000 2 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ACLRegexData 16 18 1 1 95.64 0.000 18 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
RegexList 56 5 1 1 95.64 0.000 5 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
HttpHdrSc 16 17 1 1 54.25 0.000 13 1 1 54.25 76.471 4 1 1 111 0.000 0.000 0.000
ACLCertificateData 80 3 1 1 95.64 0.000 3 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
cbdata StoreSearchHashIndex (28) 104 2 1 1 95.64 0.000 1 1 1 95.64 50.000 1 1 1 95 0.000 0.000 0.000
cbdata ClientRequestContext (20) 104 2 1 1 79.20 0.000 0 0 1 79.20 0.000 2 1 1 368427 0.297 0.119 0.000
cbdata RemovalPolicy (6) 104 2 1 1 95.64 0.000 2 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
cbdata CbDataList (3) 64 3 1 1 95.64 0.000 3 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
cbdata helper (7) 168 1 1 1 95.64 0.000 1 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ACLDestinationIP 136 1 1 1 95.64 0.000 1 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
HttpHdrRange 32 4 1 1 75.80 0.000 0 0 1 75.80 0.000 4 1 1 1277 0.001 0.000 0.000
ACLHTTPHeaderData 48 2 1 1 95.64 0.000 2 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ACLSslErrorData 16 6 1 1 95.64 0.000 6 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
cbdata TcpAcceptor (14) 96 1 1 1 95.64 0.000 1 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
cbdata WriteRequest (37) 80 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 1384769 1.115 0.344 0.001
ACLStringData 16 5 1 1 95.64 0.000 5 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ACLUserData 24 3 1 1 95.64 0.000 3 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
StoreSwapLogData 72 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 261703 0.211 0.059 0.000
cbdata RemovalPurgeWalker (31) 72 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 337261 0.272 0.075 0.000
HttpHdrRangeSpec 16 4 1 1 75.80 0.000 0 0 1 75.80 0.000 4 1 1 1278 0.001 0.000 0.000
ACLDomainData 16 4 1 1 95.64 0.000 4 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ACLMethodData 16 3 1 1 95.64 0.000 3 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
dwrite_q 48 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 1646474 1.326 0.245 0.001
cbdata IoResult (38) 40 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 1384769 1.115 0.172 0.001
ACLNoteData 40 1 1 1 95.64 0.000 1 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
StoreMetaObjSize 32 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 121139 0.098 0.012 0.000
StoreMetaURL 32 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 121139 0.098 0.012 0.000
StoreMetaSTDLFS 32 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 121139 0.098 0.012 0.000
StoreMetaVary 32 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 17105 0.014 0.002 0.000
ACLTimeData 32 1 1 1 95.64 0.000 1 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ACLASN 16 2 1 1 95.64 0.000 2 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
StoreMetaMD5 32 1 1 1 95.64 0.000 0 0 1 95.64 0.000 1 1 1 121139 0.098 0.012 0.000
CacheDigest 32 1 1 1 95.64 0.000 1 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ACLHierCodeData 32 1 1 1 95.64 0.000 1 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
ACLProtocolData 16 1 1 1 95.64 0.000 1 1 1 95.64 100.000 0 0 0 0 0.000 0.000 0.000
Total 1 3347142 1365861 1539630 4.76 100.000 3341020 1358561 1532156 4.76 99.466 6122 7300 22854 118923020 95.794 94.284 0.088
Cumulative allocated volume: 32.199 GB
Current overhead: 31649 bytes (0.002%)
Idle pool limit: 2097152.00 MB
Total Pools created: 123
Pools ever used: 113 (shown above)
Currently in use: 78
String Pool Impact
(%strings) (%volume)
Short Strings 98 89
Medium Strings 2 6
Long Strings 0 3
1KB Strings 0 1
4KB Strings 0 2
16KB Strings 0 0
Other Strings 0 0
Large buffers: 0 (0 KB)
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux