Hey Emmanuel, Something is not clear to me. Are you using url_rewrite or store_id helpers in any form? Also what DNS lookups squid does exactly? - Reverse - Forward Also: - internal clients - external domains Thanks, Eliezer ---- Eliezer Croitoru Linux System Administrator Mobile: +972-5-28704261 Email: eliezer@xxxxxxxxxxxx -----Original Message----- From: squid-users [mailto:squid-users-bounces@xxxxxxxxxxxxxxxxxxxxx] On Behalf Of FUSTE Emmanuel Sent: Friday, August 18, 2017 14:53 To: Amos Jeffries <squid3@xxxxxxxxxxxxx>; squid-users@xxxxxxxxxxxxxxxxxxxxx Subject: Memory leak (was: Squid 3.x never_direct and DNS requests problem.) Le 24/01/2017 à 10:55, FUSTE Emmanuel a écrit : > Le 23/01/2017 à 23:41, Amos Jeffries a écrit : >> On 24/01/2017 3:58 a.m., FUSTE Emmanuel wrote: >>> All was carefully checked and nothing in my configuration (acl etc ...) >>> explain why Squid insist to do DNS requests for requests forwarded to >>> the peer(s). >>> >> <snip> >>> #bug #4575 >>> url_rewrite_extras XXX >>> store_id_extras XXX >> I dont think that workaround is working. >> >>> ------------------------------------ >>> >>> Since the switch from 3.5.12 to 3.5.19/23, I am able to use a simpler >>> work around (I switched directly from 3.5.12 to 3.5.19 so I don't know >>> when the behavior changed): >>> Instead of installing a fake local DNS server and using >>> dns_nameservers 127.0.0.1 >>> I could use >>> dns_nameservers none >>> Squid warn about non usable DNS and proceed normally. Before (tested >>> with 3.5.12 and lower) Squid hang. >>> >> :-) nice. >> >> I'm prety sure this is still bug 4575. I've added a comment there to >> mention how the workaround is broken, and your improved one. >> > Thank you ! > If there's anything I can help with to solve this bug, I'd be happy to. > > Emmanuel. > It seems that using this this workaround is a bad idea. It expose or induce HUGE memory leak: Current memory usage: Pool Obj Size Chunks Allocated In Use Idle Allocations Saved Rate (bytes) KB/ch obj/ch (#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB) high (KB) high (hrs) %alloc (#) (KB) high (KB) (#) %cnt %vol (#)/sec cbdata idns_query (6) 8696 689173 5852587 5852587 0.00 87.949 689173 5852587 5852587 0.00 100.000 0 0 0 0 0.000 0.000 0.000 mem_node 4136 129945 524856 534368 3.85 7.887 129801 524275 534368 3.85 99.889 144 582 19715 4096894 0.873 9.316 0.003 fqdncache_entry 160 340128 53145 53145 0.00 0.799 340128 53145 53145 0.00 100.000 0 0 0 0 0.000 0.000 0.000 ipcache_entry 128 343083 42886 42886 0.00 0.644 343083 42886 42886 0.00 100.000 0 0 1 1 0.000 0.000 0.000 Short Strings 40 822953 32147 34373 0.28 0.483 819903 32028 34373 0.28 99.629 3050 120 605 247675706 52.776 5.447 0.166 cbdata generic_cbdata (14) 32 1026262 32071 32071 0.00 0.482 1026262 32071 32071 0.00 100.000 0 0 1 60 0.000 0.000 0.001 16KB Strings 16384 1306 20896 42624 3.15 0.314 1215 19440 42624 3.15 93.032 91 1456 13488 1976589 0.421 17.805 0.001 HttpHeaderEntry 56 372842 20390 21773 0.26 0.306 371478 20316 21773 0.26 99.634 1364 75 385 51897666 11.059 1.598 0.035 MemObject 328 35177 11268 12032 0.27 0.169 34892 11177 12032 0.27 99.190 285 92 233 1599153 0.341 0.288 0.001 HttpReply 280 35179 9620 10274 0.26 0.145 34893 9542 10274 0.26 99.187 286 79 199 4877844 1.039 0.751 0.003 Long Strings 512 18426 9213 9967 0.30 0.138 18176 9088 9967 0.30 98.643 250 125 203 5984528 1.275 1.685 0.004 Digest Scheme nonce's 72 109609 7707 14004 3.17 0.116 109609 7707 14004 3.17 100.000 0 0 7186 29051 0.006 0.001 0.001 Medium Strings 128 53288 6661 7158 0.26 0.100 53025 6629 7158 0.26 99.506 263 33 157 11608569 2.474 0.817 0.008 4KB Strings 4096 1104 4416 4916 0.28 0.066 1077 4308 4916 0.28 97.554 27 108 304 435808 0.093 0.981 0.000 StoreEntry 120 35177 4123 4402 0.27 0.062 34892 4089 4402 0.27 99.190 285 34 85 1599153 0.341 0.106 0.001 cbdata clientReplyContext (17) 4320 913 3852 5932 1.00 0.058 840 3544 5932 1.00 92.004 73 308 1047 1805429 0.385 4.288 0.001 cbdata ClientSocketContext (16) 4256 913 3795 5844 1.00 0.057 840 3492 5844 1.00 92.004 73 304 1031 1805429 0.385 4.225 0.001 1KB Strings 1024 3793 3793 4299 0.29 0.057 3766 3766 4299 0.29 99.288 27 27 155 421553 0.090 0.237 0.000 cbdata MemBuf (11) 64 35210 2201 2354 0.27 0.033 34920 2183 2354 0.27 99.176 290 19 48 9673363 2.061 0.340 0.006 HttpHdrCc 96 18070 1695 1802 0.27 0.025 17940 1682 1802 0.27 99.281 130 13 43 1112930 0.237 0.059 0.001 HttpRequest 1784 913 1591 2450 1.00 0.024 840 1464 2450 1.00 92.004 73 128 433 1805487 0.385 1.771 0.001 LRU policy node 24 35104 823 878 0.26 0.012 34823 817 878 0.26 99.200 281 7 17 45784 0.010 0.001 0.000 Comm::Connection 200 4116 804 1334 3.15 0.012 3786 740 1334 3.15 91.983 330 65 201 2095512 0.447 0.230 0.001 MD5 digest 16 35177 550 587 0.27 0.008 34892 546 587 0.27 99.190 285 5 12 1804763 0.385 0.016 0.001 cbdata Server (13) 416 1318 536 1140 3.15 0.008 1230 500 1140 3.15 93.323 88 36 346 305920 0.065 0.070 0.000 16K Buffer 16384 33 528 1488 1.23 0.008 17 272 1488 1.23 51.515 16 256 1040 395413 0.084 3.562 0.000 64K Buffer 65536 8 512 1344 3.17 0.008 1 64 1344 3.17 12.500 7 448 1344 199003 0.042 7.170 0.000 MemBlob 48 8078 379 418 0.30 0.006 7945 373 418 0.30 98.354 133 7 40 63235417 13.475 1.669 0.042 cbdata ClientHttpRequest (15) 384 913 343 528 1.00 0.005 840 315 528 1.00 92.004 73 28 93 1805429 0.385 0.381 0.001 cbdata TunnelStateData (25) 304 894 266 405 1.84 0.004 822 245 405 1.84 91.946 72 22 74 177202 0.038 0.030 0.000 cbdata clientStreamNode (18) 128 1826 229 352 1.00 0.003 1680 210 352 1.00 92.004 146 19 62 3610290 0.769 0.254 0.002 ClientInfo 448 417 183 183 0.04 0.003 417 183 183 0.04 100.000 0 0 0 0 0.000 0.000 0.000 Auth::Digest::UserRequest 152 820 122 195 1.84 0.002 761 113 195 1.84 92.805 59 9 36 1745806 0.372 0.146 0.001 8K Buffer 8192 10 80 568 0.60 0.001 1 8 568 0.60 10.000 9 72 560 2164806 0.461 9.750 0.001 Auth::Digest::User 208 350 72 79 0.18 0.001 345 71 79 0.18 98.571 5 2 13 125838 0.027 0.014 0.000 4K Buffer 4096 13 52 124 2.55 0.001 10 40 124 2.55 76.923 3 12 84 1392001 0.297 3.135 0.001 MimeEntry 128 177 23 23 4.42 0.000 177 23 23 4.42 100.000 0 0 0 0 0.000 0.000 0.000 2K Buffer 2048 10 20 68 2.25 0.000 5 10 68 2.25 50.000 5 10 62 15326460 3.266 17.257 0.010 AuthUserIP 64 306 20 21 0.20 0.000 306 20 21 0.20 100.000 0 0 2 2 0.000 0.000 0.000 NotePairs::Entry 48 399 19 35 0.31 0.000 338 16 35 0.31 84.712 61 3 9 4729264 1.008 0.125 0.003 acl_proxy_auth_match_cache 40 306 12 13 0.20 0.000 306 12 13 0.20 100.000 0 0 2 2 0.000 0.000 0.000 cbdata HttpStateData (29) 320 33 11 30 1.23 0.000 17 6 30 1.23 51.515 16 5 21 337305 0.072 0.059 0.000 AuthUserHashPointer 24 344 9 9 0.20 0.000 344 9 9 0.20 100.000 0 0 1 0 0.000 0.000 0.000 cbdata store_client (22) 160 37 6 15 1.23 0.000 19 3 15 1.23 51.351 18 3 11 1665410 0.355 0.146 0.001 cbdata FwdState (28) 176 33 6 17 1.23 0.000 17 3 17 1.23 51.515 16 3 12 337300 0.072 0.033 0.000 cbdata IdleConnList (30) 4160 1 5 5 4.41 0.000 1 5 5 4.41 100.000 0 0 5 9196 0.002 0.021 0.000 netdbEntry 168 17 3 3 0.27 0.000 17 3 3 0.27 100.000 0 0 0 0 0.000 0.000 0.000 cbdata helper_server (24) 240 10 3 3 2.63 0.000 10 3 3 2.63 100.000 0 0 0 0 0.000 0.000 0.000 Acl::AndNode 160 13 3 3 4.42 0.000 13 3 3 4.42 100.000 0 0 0 0 0.000 0.000 0.000 cbdata ACLFilledChecklist (20) 456 4 2 6 2.63 0.000 1 1 6 2.63 25.000 3 2 6 2917323 0.622 0.731 0.002 cbdata Tree (3) 216 6 2 2 4.42 0.000 6 2 2 4.42 100.000 0 0 0 0 0.000 0.000 0.000 cbdata CachePeer (4) 872 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ev_entry 48 16 1 2 0.49 0.000 11 1 2 0.49 68.750 5 1 2 43525 0.009 0.001 0.000 net_db_name 32 21 1 1 0.27 0.000 21 1 1 0.27 100.000 0 0 0 0 0.000 0.000 0.000 acl_ip_data 96 7 1 1 4.42 0.000 7 1 1 4.42 100.000 0 0 1 4 0.000 0.000 0.000 HttpHdrScTarget 88 7 1 1 0.19 0.000 7 1 1 0.19 100.000 0 0 1 73 0.000 0.000 0.000 cbdata BodyPipe (32) 152 4 1 4 0.76 0.000 1 1 4 0.76 25.000 3 1 4 33979 0.007 0.003 0.000 cbdata ConnOpener (27) 136 4 1 4 0.49 0.000 0 0 4 0.49 0.000 4 1 4 250616 0.053 0.019 0.000 cbdata ClientRequestContext (19) 104 5 1 4 0.60 0.000 0 0 4 0.60 0.000 5 1 4 1827236 0.389 0.104 0.001 ACLStrategised 160 3 1 1 4.42 0.000 3 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 cbdata CbDataList (1) 40 12 1 1 4.42 0.000 12 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLSourceIP 144 3 1 1 4.42 0.000 3 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 cbdata Logfile (9) 352 1 1 1 3.75 0.000 1 1 1 3.75 100.000 0 0 1 0 0.000 0.000 0.000 ACLStringData 56 6 1 1 4.42 0.000 6 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 wordlist 16 21 1 4 4.42 0.000 21 1 4 4.42 100.000 0 0 4 18 0.000 0.000 0.000 ACLStrategised 160 2 1 1 4.42 0.000 2 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 Acl::NotNode 160 2 1 1 4.42 0.000 2 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLRegexData 16 18 1 1 4.42 0.000 18 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 RegexList 88 3 1 1 4.42 0.000 3 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLUserData 64 4 1 1 4.42 0.000 4 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 cbdata ps_state (26) 256 1 1 1 4.41 0.000 0 0 1 4.41 0.000 1 1 1 543789 0.116 0.077 0.000 cbdata ErrorState (21) 232 1 1 1 1.91 0.000 0 0 1 1.91 0.000 1 1 1 1271537 0.271 0.162 0.001 cbdata TcpAcceptor (12) 104 2 1 1 4.42 0.000 2 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 cbdata CbDataList (33) 96 2 1 3 1.34 0.000 0 0 3 1.34 0.000 2 1 3 738165 0.157 0.039 0.000 cbdata helper (7) 168 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLStrategised 160 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLProxyAuth 152 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLDestinationIP 144 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 cbdata StateData (23) 48 3 1 1 2.63 0.000 0 0 1 2.63 0.000 3 1 1 1155036 0.246 0.030 0.001 Helper::Request 48 3 1 1 2.63 0.000 0 0 1 2.63 0.000 3 1 1 1155036 0.246 0.030 0.001 HttpHdrSc 16 7 1 1 0.19 0.000 7 1 1 0.19 100.000 0 0 1 73 0.000 0.000 0.000 cbdata RemovalPolicy (10) 104 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLHTTPHeaderData 48 2 1 1 4.42 0.000 2 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 FwdServer 24 3 1 1 4.41 0.000 0 0 1 4.41 0.000 3 1 1 1631367 0.348 0.022 0.001 cbdata RemovalPurgeWalker (34) 72 1 1 1 4.17 0.000 0 0 1 4.17 0.000 1 1 1 6250 0.001 0.000 0.000 cbdata CbDataList (2) 64 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLNoteData 40 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 CacheDigest 40 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLDomainData 16 2 1 1 4.42 0.000 2 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLHierCodeData 32 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLMethodData 16 2 1 1 4.42 0.000 2 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLTimeData 32 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 ACLASN 16 2 1 1 4.42 0.000 2 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 HttpHdrRange 32 1 1 1 2.85 0.000 0 0 1 2.85 0.000 1 1 1 7768 0.002 0.000 0.000 ACLProtocolData 16 1 1 1 4.42 0.000 1 1 1 4.42 100.000 0 0 0 0 0.000 0.000 0.000 HttpHdrRangeSpec 16 1 1 1 2.85 0.000 0 0 1 2.85 0.000 1 1 1 7768 0.002 0.000 0.000 32K Buffer 32768 0 0 96 4.28 0.000 0 0 96 4.28 -1.000 0 0 96 534 0.000 0.010 0.000 dlink_node 24 0 0 1 4.41 0.000 0 0 1 4.41 -1.000 0 0 1 7 0.000 0.000 0.000 HttpHdrContRange 24 0 0 1 3.76 0.000 0 0 1 3.76 -1.000 0 0 1 657 0.000 0.000 0.000 cbdata netdbExchangeState (31) 4176 0 0 5 4.41 0.000 0 0 5 4.41 -1.000 0 0 5 0 0.000 0.000 0.000 cbdata StatObjectsState (35) 48 0 0 1 0.14 0.000 0 0 1 0.14 -1.000 0 0 1 1 0.000 0.000 0.000 cbdata StoreSearchHashIndex (36) 104 0 0 1 0.14 0.000 0 0 1 0.14 -1.000 0 0 1 1 0.000 0.000 0.000 Total 1 4164421 6654528 6654528 0.00 100.000 4156051 6650022 6650022 0.00 99.932 8370 4506 27047 461504099 98.340 95.012 0.312 Cumulative allocated volume: 181.888 GB Current overhead: 34740 bytes (0.001%) Idle pool limit: 5.00 MB Total Pools created: 111 Pools ever used: 102 (shown above) Currently in use: 85 String Pool Impact (%strings) (%volume) Short Strings 91 42 Medium Strings 6 9 Long Strings 2 12 1KB Strings 0 5 4KB Strings 0 6 16KB Strings 0 26 Other Strings 0 1 Large buffers: 0 (0 KB) A special/real "dns_nameservers none" handling need to be implemented. So I'm back with fake local DNS server. We will see if my Squid instances will survive more than 8h with that. Emmanuel. _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
