run "free -m" in shell and paste the output here On Mon, Jul 7, 2014 at 5:46 AM, Martin Sperl <Martin.Sperl@xxxxxxxxxx> wrote: > Hi! > > We have found out that since we moved from squid 3.3 to 3.4.3 (and the corresponding Vary) the memory utilization of squid has increased. > It is 10GB right now 9.6G in memory, but we have only configured 4GB for in Memory caching. > > Here the ps line: > squid 17695 10.9 61.2 10629864 10001520 ? Rl Apr21 12131:59 (squid-1) -f /opt/squid/squid.conf > > Any Ideas what that could be? See below the fine details from mgr:... > > BTW: I know it is not the latest version, but I am unsure if there have been any fixes for this behavior in a later version... > > Thanks, > Martin > > P.s: the relevant config-section for memory settings: > > # define the memory cache as 4GB > cache_mem 4096 MB > > # and in memory sizes for single objects: > maximum_object_size 32 KB > maximum_object_size_in_memory 32 KB > > # define a disk cache (preferably on ram-disk to avoid IOs) > cache_dir aufs /var/spool/squid/ 8192 16 256 > > here the infos from: > > Mgr:info: > Squid Object Cache: Version 3.4.3 > Build Info: > Start Time: Mon, 21 Apr 2014 08:31:27 GMT > Current Time: Mon, 07 Jul 2014 08:21:22 GMT > Connection information for squid: > Number of clients accessing cache: 0 > Number of HTTP requests received: 998061101 > Number of ICP messages received: 0 > Number of ICP messages sent: 0 > Number of queued ICP replies: 0 > Number of HTCP messages received: 0 > Number of HTCP messages sent: 0 > Request failure ratio: 0.00 > Average HTTP requests per minute since start: 9002.1 > Average ICP messages per minute since start: 0.0 > Select loop called: 7233613675 times, 0.920 ms avg > Cache information for squid: > Hits as % of all requests: 5min: 40.8%, 60min: 45.5% > Hits as % of bytes sent: 5min: 50.7%, 60min: 51.6% > Memory hits as % of hit requests: 5min: 72.6%, 60min: 74.9% > Disk hits as % of hit requests: 5min: 1.8%, 60min: 2.0% > Storage Swap size: 7549744 KB > Storage Swap capacity: 90.0% used, 10.0% free > Storage Mem size: 4153612 KB > Storage Mem capacity: 99.0% used, 1.0% free > Mean Object Size: 5.14 KB > Requests given to unlinkd: 0 > Median Service Times (seconds) 5 min 60 min: > HTTP Requests (All): 0.00179 0.00091 > Cache Misses: 0.27332 0.23230 > Cache Hits: 0.00000 0.00000 > Near Hits: 0.00286 0.00286 > Not-Modified Replies: 0.00000 0.00000 > DNS Lookups: 0.00094 0.00094 > ICP Queries: 0.00000 0.00000 > Resource usage for squid: > UP Time: 6652194.459 seconds > CPU Time: 727888.282 seconds > CPU Usage: 10.94% > CPU Usage, 5 minute avg: 2.76% > CPU Usage, 60 minute avg: 2.93% > Process Data Segment Size via sbrk(): 10517816 KB > Maximum Resident Size: 40036704 KB > Page faults with physical i/o: 2100678 > Memory usage for squid via mallinfo(): > Total space in arena: -2064964 KB > Ordinary blocks: 2047465 KB 161900 blks > Small blocks: 0 KB 0 blks > Holding blocks: 38432 KB 9 blks > Free Small blocks: 0 KB > Free Ordinary blocks: 81875 KB > Total in use: 81875 KB -4% > Total free: 81875 KB -4% > Total size: -2026532 KB > Memory accounted for: > Total accounted: 1269447 KB -63% > memPool accounted: 9658055 KB -477% > memPool unaccounted: -11684587 KB -0% > memPoolAlloc calls: 507567155336 > memPoolFree calls: 508713005568 > File descriptor usage for squid: > Maximum number of file descriptors: 65536 > Largest file desc currently in use: 1351 > Number of file desc currently in use: 249 > Files queued for open: 0 > Available number of file descriptors: 65287 > Reserved number of file descriptors: 100 > Store Disk files open: 0 > Internal Data Structures: > 1477933 StoreEntries > 851676 StoreEntries with MemObjects > 851675 Hot Object Cache Items > 1470120 on-disk objects > > Mgr:mem: > Current memory usage: > Pool Obj Size Chunks Allocated In UseIdle Allocations Saved Rate > (bytes) KB/ch obj/ch (#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot (#) (KB) high (KB) high (hrs) %alloc (#) (KB) high (KB) (#) %cnt %vol (#)/sec > mem_node 4136 1038636 4195116 4195359538.01 43.434 1038434 4194300 4195359 538.01 99.981 202 816 1564 695940913 0.137 1.2200.499 > 4K Buffer 4096 471762 1887048 1916096 61.2219.538 471750 1887000 1916096 61.22 99.997 12 48 3120 986312276 0.194 1.713 0.707 > Short Strings 40 20241644 790690 794615 61.01 8.18620241331 790677 794615 61.01 99.998 313 13 841 -2147483648 66.374 5.726 240.914 > HttpHeaderEntry 56 9903143 541579 544664 61.01 5.6079902997 541571 544664 61.01 99.999 146 8 551 -2147483648 16.522 1.996 60.070 > 2K Buffer 2048 238817 477634 724132 967.29 4.945 238800 477600 724132 967.29 99.993 17 34 546 -2147483648 1.566 6.917 5.673 > 16K Buffer 16384 25372 405952 1106704 1357.164.203 25355 405680 1106704 1357.16 99.933 17 272 4048 268150503 0.053 1.863 0.193 > HttpReply 280 851758 232903 233312 67.89 2.411 851732 232896 233312 67.89 99.997 26 8 138 -2147483648 0.688 0.415 2.495 > 8K Buffer 8192 27681 221448 275280 1357.54 2.29327679 221432 275280 1357.54 99.993 2 16 312 235405729 0.046 0.818 0.168 > MemObject 240 851740 199627 199973 67.89 2.067 851731 199625 199973 67.89 99.999 9 3 93 492884384 0.097 0.050 0.354 > StoreEntry 120 1477969 173200 173405 69.33 1.7931477949 173198 173405 69.33 99.999 20 3 32 448740915 0.088 0.023 0.322 > Medium Strings 128 1262162 157771 160057 61.01 1.6331262100 157763 160057 61.01 99.995 62 8 279 -2147483648 4.731 1.306 17.148 > cbdata BodyPipe (43) 152 765008 113556 114424 68.08 1.176 765004 113556 114424 68.08 99.999 4 1 34 358799947 0.071 0.023 0.257 > HttpHdrCc 96 769579 72149 72667 59.52 0.747 769566 72147 72667 59.52 99.998 13 2 34 1594117469 0.313 0.065 1.138 > LRU policy node 24 2321816 54418 54496 68.09 0.5632321811 54418 54496 68.09 100.000 5 1 12 159292188 0.031 0.002 0.117 > cbdata MemBuf (13) 64 851759 53235 53330 67.89 0.551 851734 53234 53330 67.89 99.997 25 2 33 -2147483648 0.923 0.127 3.346 > 32K Buffer 32768 1603 51296 111520 979.75 0.531 160051200 111520 979.75 99.813 3 96 736 20665690 0.004 0.287 0.015 > MD5 digest 16 1477969 23094 23121 69.33 0.2391477949 23093 23121 69.33 99.999 20 1 5 595307879 0.117 0.004 0.426 > Long Strings 512 6736 3368 7858 1063.01 0.0356699 3350 7858 1063.01 99.451 37 19 234 -2147483648 1.034 1.142 3.750 > 64K Buffer 65536 46 2944 33792 1711.70 0.0300 0 33792 1711.70 0.000 46 2944 32768 -2147483648 0.466 65.894 1.691 > acl_ip_data 96 5725 537 537 1848.01 0.0065725 537 537 1848.01 100.000 0 0 1 276 0.000 0.000 0.000 > cbdata IdleConnList (16) 4160 51 208 281 1757.200.002 45 183 281 1757.20 88.235 6 25 86 36686446 0.007 0.065 0.026 > cbdata clientReplyContext (23) 4320 31 131 1730 1293.490.001 4 17 1730 1293.49 12.903 27 114 1321 995516255 0.196 1.823 0.711 > cbdata ClientSocketContext (22) 4256 31 129 1705 1293.490.001 4 17 1705 1293.49 12.903 27 113 1301 995516208 0.196 1.796 0.711 > HttpRequest 1768 63 109 1428 1293.49 0.0017 13 1428 1293.49 11.111 56 97 992 -2147483648 0.827 3.153 2.998 > cbdata ConnStateData (20) 440 201 87 1259 390.81 0.001171 74 1259 390.81 85.075 30 13 343 528103233 0.104 0.099 0.381 > Comm::Connection 192 247 47 591 390.81 0.000 21741 591 390.81 87.854 30 6 171 829565512 0.163 0.068 0.596 > ipcache_entry 128 289 37 82 1252.79 0.000288 36 82 1252.79 99.654 1 1 49 821666 0.000 0.000 0.001 > cbdata idns_query (9) 8696 3 26 3661 1252.79 0.0000 0 3661 1252.79 0.000 3 26 3346 1134547 0.000 0.004 0.001 > Acl::AndNode 136 184 25 25 1848.01 0.000184 25 25 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > MimeEntry 128 177 23 23 1848.01 0.000177 23 23 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > fqdncache_entry 160 99 16 16 11.53 0.000 9916 16 11.53 100.000 0 0 1 607 0.000 0.000 0.000 > cbdata ClientHttpRequest (21) 408 31 13 164 1293.490.000 4 2 164 1293.49 12.903 27 11 125 995516208 0.196 0.172 0.711 > 4KB Strings 4096 3 12 100 136.37 0.000 00 100 136.37 0.000 3 12 100 4568442 0.001 0.008 0.003 > cbdata ServiceRep (15) 344 34 12 12 1848.01 0.00034 12 12 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata Tree (6) 192 52 10 10 1848.01 0.00052 10 10 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata ModXact (30) 496 17 9 127 1711.70 0.0000 0 127 1711.70 0.000 17 9 126 1177041017 0.231 0.247 0.839 > squidaio_small_bufs 4096 2 8 120 1065.11 0.0000 0 120 1065.11 0.000 2 8 120 60491040 0.012 0.105 0.043 > cbdata clientStreamNode (24) 128 62 8 103 1293.490.000 8 1 103 1293.49 12.903 54 7 79 1991032510 0.391 0.108 1.421 > cbdata OptXact (32) 312 19 6 11 885.88 0.000 00 11 885.88 0.000 19 6 11 7038032 0.001 0.001 0.005 > RegexList 88 59 6 6 1848.01 0.00059 6 6 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > ACLStrategised 136 38 6 6 1848.01 0.00038 6 6 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > ACLStrategised 136 37 5 5 1848.01 0.00037 5 5 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > HttpHdrScTarget 88 55 5 6 343.75 0.000 555 6 343.75 100.000 0 0 2 149 0.000 0.000 0.000 > cbdata Logfile (11) 4192 1 5 5 1848.01 0.0001 5 5 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata store_client (34) 160 23 4 117 1293.490.000 4 1 117 1293.49 17.391 19 3 76 1290240791 0.254 0.088 0.920 > 1KB Strings 1024 3 3 55 412.45 0.000 00 55 412.45 0.000 3 3 38 50912098 0.010 0.022 0.037 > cbdata HttpStateData (38) 304 9 3 115 1293.490.000 2 1 115 1293.49 22.222 7 3 76 190925179 0.038 0.025 0.137 > cbdata Iterator (28) 160 17 3 40 1711.70 0.0000 0 40 1711.70 0.000 17 3 40 1177188448 0.231 0.080 0.839 > ACLSourceIP 120 22 3 3 1848.01 0.00022 3 3 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata ModXactLauncher (29) 152 17 3 38 1711.700.000 0 0 38 1711.70 0.000 17 3 38 1177189213 0.231 0.076 0.839 > Acl::NotNode 136 18 3 3 1848.01 0.00018 3 3 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata AccessCheck (27) 144 17 3 41 1711.69 0.0000 0 41 1711.69 0.000 17 3 41 1177394228 0.231 0.072 0.839 > cbdata OptXactLauncher (31) 128 19 3 5 885.88 0.0000 0 5 885.88 0.000 19 3 5 7037587 0.001 0.000 0.005 > ev_entry 48 47 3 3 1831.15 0.00045 3 3 1831.15 95.745 2 1 2 33129146 0.007 0.001 0.024 > ACLHTTPHeaderData 48 40 2 2 1848.01 0.00040 2 2 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata ClientRequestContext (25) 104 18 2 30 1711.70 0.000 0 0 30 1711.70 0.000 18 2 30 988402315 0.194 0.044 0.705 > ACLRegexData 16 110 2 2 1848.01 0.000110 2 2 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata FwdState (36) 168 9 2 65 1293.49 0.0002 1 65 1293.49 22.222 7 2 42 191697095 0.038 0.014 0.137 > cbdata ACLFilledChecklist (26) 440 3 2 93 1252.790.000 1 1 93 1252.79 33.333 2 1 85 -2147483648 2.155 2.045 7.805 > cbdata ConnOpener (33) 136 6 1 27 1711.70 0.0000 0 27 1711.70 0.000 6 1 27 66249937 0.013 0.004 0.048 > cbdata PortCfg (2) 344 2 1 1 1848.01 0.0002 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > ACLStrategised 136 5 1 1 1848.01 0.0005 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > aio_thread 40 16 1 1 1847.93 0.00016 1 1 1847.93 100.000 0 0 0 0 0.000 0.000 0.000 > squidaio_micro_bufs 128 5 1 14 1065.11 0.0000 0 14 1065.11 0.000 5 1 14 176721246 0.035 0.010 0.126 > aio_ctrl 112 5 1 13 1065.11 0.0000 0 13 1065.11 0.000 5 1 13 237358432 0.047 0.011 0.169 > ACLStrategised 136 4 1 1 1848.01 0.0004 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata CbDataList (3) 40 13 1 1 1848.01 0.00013 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > aio_request 104 5 1 12 1065.11 0.0000 0 12 1065.11 0.000 5 1 12 237365801 0.047 0.010 0.169 > cbdata ErrorState (46) 240 2 1 3 1497.43 0.0000 0 3 1497.43 0.000 2 1 3 10391751 0.002 0.001 0.007 > ACLStrategised 136 3 1 1 1848.01 0.0003 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata CbDataList (1) 40 10 1 1 1848.01 0.00010 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata CbDataList (47) 96 4 1 4 1664.29 0.0000 0 4 1664.29 0.000 4 1 4 27953666 0.005 0.001 0.020 > acl_name_list 72 5 1 1 1848.01 0.0005 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > ACLDestinationIP 120 3 1 1 1848.01 0.0003 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata UFSStoreState (39) 176 2 1 13 1065.110.000 0 0 13 1065.11 0.000 2 1 13 112646684 0.022 0.008 0.080 > ACLDomainData 16 19 1 1 1848.01 0.00019 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata CbDataList (4) 64 4 1 1 1848.01 0.0004 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata ps_state (37) 248 1 1 1 1835.63 0.0000 0 1 1835.63 0.000 1 1 1 192773573 0.038 0.020 0.137 > ACLCertificateData 72 3 1 1 1848.01 0.0003 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata RemovalPolicy (7) 104 2 1 1 1848.010.000 2 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata StoreSearchHashIndex (18) 104 2 1 1 1848.01 0.000 1 1 1 1848.01 50.000 1 1 1 20 0.000 0.000 0.000 > cbdata TcpAcceptor (17) 88 2 1 1 1848.01 0.0002 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata DiskThreadsDiskFile (40) 88 2 1 7 1065.110.000 0 0 7 1065.11 0.000 2 1 7 112646684 0.022 0.004 0.080 > AclDenyInfoList 32 5 1 1 1848.01 0.0005 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > HttpHdrRange 32 5 1 3 339.96 0.000 21 3 339.96 40.000 3 1 1 1563142 0.000 0.000 0.001 > cbdata ReadRequest (41) 72 2 1 3 1065.11 0.0000 0 3 1065.11 0.000 2 1 3 60491040 0.012 0.002 0.043 > ACLStrategised 136 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > UFSStoreState::_queued_write 32 4 1 3 1428.470.000 0 0 3 1428.47 0.000 4 1 3 134435143 0.026 0.002 0.096 > HttpHdrContRange 24 5 1 2 1664.29 0.0000 0 2 1664.29 0.000 5 1 2 792627 0.000 0.000 0.001 > ACLStringData 16 6 1 1 1848.01 0.0006 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > ACLSslErrorData 16 6 1 1 1848.01 0.0006 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata generic_cbdata (35) 32 3 1 14 1252.790.000 0 0 14 1252.79 0.000 3 1 13 65234012 0.013 0.001 0.047 > UFSStoreState::_queued_read 40 2 1 2 1065.110.000 0 0 2 1065.11 0.000 2 1 2 46057286 0.009 0.001 0.033 > cbdata IoResult (42) 40 2 1 2 1065.11 0.0000 0 2 1065.11 0.000 2 1 2 60491040 0.012 0.001 0.043 > cbdata WriteRequest (44) 80 1 1 1 1847.930.000 0 0 1 1847.93 0.000 1 1 1 134753572 0.026 0.005 0.096 > HttpHdrRangeSpec 16 5 1 2 339.96 0.000 21 2 339.96 40.000 3 1 1 1563133 0.000 0.000 0.001 > ACLUserData 24 3 1 1 1848.01 0.0003 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > StoreSwapLogData 72 1 1 1 1848.01 0.0000 0 1 1848.01 0.000 1 1 1 129809816 0.026 0.004 0.092 > cbdata RemovalPurgeWalker (19) 72 1 1 1 1848.010.000 0 0 1 1848.01 0.000 1 1 1 7712019 0.002 0.000 0.006 > ACLMethodData 16 4 1 1 1848.01 0.0004 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > link_list 16 4 1 2 1065.11 0.0000 0 2 1065.11 0.000 4 1 2 180627075 0.036 0.001 0.129 > dwrite_q 48 1 1 1 1848.01 0.0000 0 1 1848.01 0.000 1 1 1 264625573 0.052 0.005 0.188 > ACLNoteData 40 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata CbDataList (5) 40 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > cbdata IoResult (45) 40 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 134753572 0.026 0.002 0.096 > ACLASN 16 2 1 1 1848.01 0.0002 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > ACLHierCodeData 32 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > ACLProtocolData 16 2 1 1 1848.01 0.0002 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > ACLTimeData 32 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > CacheDigest 32 1 1 1 1848.01 0.0001 1 1 1848.01 100.000 0 0 0 0 0.000 0.000 0.000 > wordlist 16 2 1 1 1848.01 0.0002 1 1 1848.01 100.000 0 0 1 169 0.000 0.000 0.000 > StoreMetaMD5 32 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 110536338 0.022 0.001 0.079 > StoreMetaSTDLFS 32 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 110536338 0.022 0.001 0.079 > StoreMetaURL 32 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 110536338 0.022 0.001 0.079 > StoreMetaObjSize 32 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 110535723 0.022 0.001 0.079 > StoreMetaVary 32 1 1 1 1847.93 0.0000 0 1 1847.93 0.000 1 1 1 6413952 0.001 0.000 0.005 > FwdServer 24 1 1 1 1835.63 0.0000 0 1 1835.63 0.000 1 1 1 192773573 0.038 0.002 0.137 > 16KB Strings 16384 0 0 32 853.17 0.000 00 32 853.17 -1.000 0 0 32 0 0.000 0.000 0.000 > cbdata nsvc (8) 72 0 0 1 1819.31 0.0000 0 1 1819.31 -1.000 0 0 1 0 0.000 0.000 0.000 > cbdata RebuildState (12) 688 0 0 1 1848.010.000 0 0 1 1848.01 -1.000 0 0 1 0 0.000 0.000 0.000 > cbdata RemovalPolicyWalker (48) 56 0 0 1 1846.340.000 0 0 1 1846.34 -1.000 0 0 1 0 0.000 0.000 0.000 > cbdata BodySink (49) 80 0 0 1 1844.65 0.0000 0 1 1844.65 -1.000 0 0 1 549 0.000 0.000 0.000 > HttpHdrSc 16 0 0 1 343.78 0.000 00 1 343.78 -1.000 0 0 1 82 0.000 0.000 0.000 > Total 1 42593287 9658537 971909061.01 100.000 42591806 9653779 9713153 61.04 99.951 1481 4758 37935 -2147483648 99.766 99.807 362.183 > Cumulative allocated volume: 235.8852 TB > Current overhead: 34522 bytes (0.000%) > Idle pool limit: 5.00 MB > Total Pools created: 134 > Pools ever used: 123 (shown above) > Currently in use: 78 > String Pool Impact > (%strings) (%volume) > Short Strings 94 83 > Medium Strings 6 17 > Long Strings 0 0 > 1KB Strings 0 0 > 4KB Strings 0 0 > 16KB Strings 0 0 > Other Strings 0 0 > > Large buffers: 0 (0 KB) > > > > This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, > you may review at http://www.amdocs.com/email_disclaimer.asp
