Search squid archive

Re: How to analyse squid memory usage

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Thanks Amos -

Not sure how self-explanatory the output is, though.

I’ve attached the output from a site with a 12GB server where top was showing 2.9GB allocated to squid (this is normal e.g. “the control"). But the mem output shows the allocated total as ~1GB, apparently?

Maybe things will become clearer once I have a “leaky” server’s output to compare with it.
Pool Obj SizeChunksAllocatedIn UseIdleAllocations SavedRate
(bytes)KB/ch obj/ch(#) used free part %Frag (#) (KB) high (KB) high (hrs) %Tot(#) (KB) high (KB) high (hrs) %alloc(#) (KB) high (KB)(#) %cnt %vol(#)/sec
mem_node 4136 64944 262313 265205 2.87 25.807 64854 261950 265205 2.87 99.861 90 364 4036 4445407 2.478 20.090 0.003
ExternalACLEntry 256 998772 249693 249693 0.00 24.566 998771 249693 249693 0.00 100.000 1 1 1 708911 0.395 0.198 0.001
Long Strings 512 437504 218752 224054 0.71 21.522 437498 218749 224054 0.71 99.999 6 3 839 4201687 2.342 2.351 0.003
Medium Strings 128 543962 67996 68087 0.05 6.690 543714 67965 68087 0.05 99.954 248 31 170 12529486 6.983 1.752 0.009
StoreEntry 120 442960 51910 56978 3.15 5.107 442949 51909 56978 3.15 99.998 11 2 102 332274 0.185 0.044 0.001
Short Strings 40 1323856 51714 53226 0.72 5.088 1323436 51697 53226 0.72 99.968 420 17 520 63712274 35.509 2.785 0.045
NotePairs::Entry 48 669665 31391 31391 0.00 3.088 669636 31390 31391 0.00 99.996 29 2 30 9098556 5.071 0.477 0.007
16KB Strings 16384 1232 19712 52272 2.52 1.939 1231 19696 52272 2.52 99.919 1 16 4576 782012 0.436 14.000 0.001
LRU policy node 24 460379 10791 11683 2.88 1.062 460370 10790 11683 2.88 99.998 9 1 27 69551 0.039 0.002 0.000
HttpHeaderEntry 56 179215 9801 10985 1.21 0.964 178979 9788 10985 1.21 99.868 236 13 350 13955943 7.778 0.854 0.010
MD5 digest 16 442960 6922 7597 3.15 0.681 442949 6922 7597 3.15 99.998 11 1 14 398517 0.222 0.007 0.001
MemObject 328 17563 5626 6040 0.98 0.553 17539 5618 6040 0.98 99.863 24 8 223 378802 0.211 0.136 0.000
4KB Strings 4096 1259 5036 5036 0.01 0.495 1247 4988 5036 0.01 99.047 12 48 748 127748 0.071 0.572 0.000
HttpReply 280 17569 4805 5159 1.21 0.473 17543 4797 5159 1.21 99.852 26 8 191 1145088 0.638 0.350 0.001
cbdata clientReplyContext (22) 4320 1120 4725 12285 2.52 0.465 1119 4721 12285 2.52 99.911 1 5 958 583958 0.325 2.757 0.000
cbdata ClientSocketContext (21) 4256 1120 4655 12103 2.52 0.458 1119 4651 12103 2.52 99.911 1 5 944 583958 0.325 2.716 0.000
HttpRequest 1784 1139 1985 5086 2.52 0.195 1138 1983 5086 2.52 99.912 1 2 394 584066 0.326 1.139 0.000
cbdata MemBuf (13) 64 17790 1112 1195 1.21 0.109 17756 1110 1195 1.21 99.809 34 3 44 3950213 2.202 0.276 0.003
HttpHdrCc 96 10688 1002 1061 0.54 0.099 10641 998 1061 0.54 99.560 47 5 49 494286 0.275 0.052 0.000
4K Buffer 4096 215 860 880 1.12 0.085 207 828 880 1.12 96.279 8 32 120 394673 0.220 1.766 0.000
Comm::Connection 200 3692 722 1798 2.27 0.071 3681 719 1798 2.27 99.702 11 3 146 1708339 0.952 0.373 0.001
1KB Strings 1024 707 707 1846 1.86 0.070 634 634 1846 1.86 89.675 73 73 289 3827224 2.133 4.282 0.003
cbdata Server (18) 520 1273 647 1677 2.53 0.064 1272 646 1677 2.53 99.921 1 1 150 401690 0.224 0.228 0.000
cbdata IdleConnList (39) 4160 133 541 1187 1.25 0.053 125 508 1187 1.25 93.985 8 33 646 70268 0.039 0.319 0.000
cbdata ClientHttpRequest (20) 392 1120 429 1115 2.52 0.042 1119 429 1115 2.52 99.911 1 1 87 583958 0.325 0.250 0.000
cbdata idns_query (10) 8696 49 417 2523 3.10 0.041 12 102 2523 3.10 24.490 37 315 2234 253002 0.141 2.404 0.000
ClientInfo 432 862 364 364 0.00 0.036 862 364 364 0.00 100.000 0 0 0 0 0.000 0.000 0.000
cbdata TunnelStateData (26) 304 1108 329 856 2.52 0.032 1107 329 856 2.52 99.910 1 1 62 186431 0.104 0.062 0.000
cbdata clientStreamNode (23) 128 2240 280 728 2.52 0.028 2238 280 728 2.52 99.911 2 1 57 1167910 0.651 0.163 0.001
16K Buffer 16384 17 272 736 2.14 0.027 6 96 736 2.14 35.294 11 176 624 190937 0.106 3.418 0.000
MemBlob 48 4668 219 330 1.27 0.022 4667 219 330 1.27 99.979 1 1 20 3844886 2.143 0.202 0.003
fqdncache_entry 160 929 146 147 1.67 0.014 920 144 147 1.67 99.031 9 2 18 25113 0.014 0.004 0.000
ipcache_entry 128 1025 129 154 3.10 0.013 994 125 154 3.10 96.976 31 4 33 158901 0.089 0.022 0.000
Auth::Basic::User 192 351 66 85 1.66 0.006 350 66 85 1.66 99.715 1 1 20 285990 0.159 0.060 0.000
8K Buffer 8192 7 56 240 2.14 0.006 2 16 240 2.14 28.571 5 40 208 580579 0.324 5.197 0.000
cbdata helper_server (12) 240 207 49 49 1.81 0.005 207 49 49 1.81 100.000 0 0 0 0 0.000 0.000 0.000
2K Buffer 2048 22 44 106 2.14 0.004 7 14 106 2.14 31.818 15 30 96 9669241 5.389 21.638 0.007
squidaio_small_bufs 4096 8 32 60 2.87 0.003 0 0 60 2.87 0.000 8 32 60 827452 0.461 3.703 0.001
cbdata ACLFilledChecklist (25) 472 58 27 69 3.10 0.003 27 13 69 3.10 46.552 31 15 68 2036246 1.135 1.050 0.001
MimeEntry 128 177 23 23 9.60 0.002 177 23 23 9.60 100.000 0 0 0 0 0.000 0.000 0.000
AuthUserIP 64 345 22 28 1.66 0.002 345 22 28 1.66 100.000 0 0 7 924 0.001 0.000 0.000
acl_proxy_auth_match_cache 40 345 14 18 1.66 0.001 345 14 18 1.66 100.000 0 0 4 10 0.000 0.000 0.000
cbdata Logfile (15) 4192 3 13 13 9.60 0.001 3 13 13 9.60 100.000 0 0 0 0 0.000 0.000 0.000
Auth::Basic::UserRequest 48 182 9 37 1.86 0.001 151 8 37 1.86 82.967 31 2 9 279564 0.156 0.015 0.000
AuthUserHashPointer 24 348 9 11 1.66 0.001 348 9 11 1.66 100.000 0 0 3 14 0.000 0.000 0.000
Acl::AndNode 136 61 9 9 9.60 0.001 61 9 9 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata ClientRequestContext (24) 104 55 6 15 3.10 0.001 24 3 15 3.10 43.636 31 4 15 594478 0.331 0.068 0.000
Acl::NotNode 136 42 6 6 9.60 0.001 42 6 6 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata HttpStateData (38) 312 16 5 14 2.14 0.000 5 2 14 2.14 31.250 11 4 12 178489 0.099 0.061 0.000
cbdata helper (11) 168 21 4 4 9.60 0.000 21 4 4 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata store_client (31) 160 22 4 10 2.14 0.000 7 2 10 2.14 31.818 15 3 8 431019 0.240 0.075 0.000
cbdata FwdState (36) 176 20 4 9 2.14 0.000 6 2 9 2.14 30.000 14 3 8 181140 0.101 0.035 0.000
cbdata external_acl (2) 192 18 4 4 9.60 0.000 18 4 4 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata Tree (6) 192 16 3 3 9.60 0.000 16 3 3 9.60 100.000 0 0 0 0 0.000 0.000 0.000
external_acl_format 64 43 3 3 9.60 0.000 43 3 3 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata ConnOpener (28) 136 20 3 19 1.97 0.000 2 1 19 1.97 10.000 18 3 19 386491 0.215 0.057 0.000
squidaio_micro_bufs 128 20 3 10 2.41 0.000 0 0 10 2.41 0.000 20 3 10 206396 0.115 0.029 0.000
cbdata externalAclState (14) 88 27 3 11 3.10 0.000 1 1 11 3.10 3.704 26 3 11 2298241 1.281 0.221 0.002
acl_ip_data 96 24 3 3 9.60 0.000 24 3 3 9.60 100.000 0 0 1 7 0.000 0.000 0.000
ACLExternal 128 18 3 3 9.60 0.000 18 3 3 9.60 100.000 0 0 0 0 0.000 0.000 0.000
aio_ctrl 112 20 3 9 2.41 0.000 0 0 9 2.41 0.000 20 3 9 1034453 0.577 0.127 0.001
aio_request 104 20 3 8 2.41 0.000 0 0 8 2.41 0.000 20 3 8 1034468 0.577 0.118 0.001
cbdata UFSStoreState (32) 176 11 2 5 2.59 0.000 0 0 5 2.59 0.000 11 2 5 105893 0.059 0.020 0.000
ACLStrategised 136 13 2 2 9.60 0.000 13 2 2 9.60 100.000 0 0 0 0 0.000 0.000 0.000
wordlist 16 80 2 2 9.60 0.000 80 2 2 9.60 100.000 0 0 1 25 0.000 0.000 0.000
cbdata generic_cbdata (19) 32 40 2 10 3.10 0.000 8 1 10 3.10 20.000 32 1 9 250316 0.140 0.009 0.000
UFSStoreState::_queued_write 32 35 2 14 1.98 0.000 0 0 14 1.98 0.000 35 2 14 2373675 1.323 0.083 0.002
cbdata DiskThreadsDiskFile (33) 88 11 1 3 2.59 0.000 0 0 3 2.59 0.000 11 1 3 105893 0.059 0.010 0.000
Helper::Request 48 19 1 3 2.14 0.000 1 1 3 2.14 5.263 18 1 3 2075987 1.157 0.109 0.001
cbdata external_acl_data (3) 48 18 1 1 9.60 0.000 18 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
acl_name_list 72 11 1 1 9.60 0.000 11 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ev_entry 48 15 1 1 1.23 0.000 11 1 1 1.23 73.333 4 1 1 65014 0.036 0.003 0.000
HttpHdrScTarget 88 8 1 2 1.21 0.000 8 1 2 1.21 100.000 0 0 1 79 0.000 0.000 0.000
ACLStrategised 136 5 1 1 9.60 0.000 5 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
aio_thread 40 16 1 1 9.60 0.000 16 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata BodyPipe (41) 152 4 1 2 3.19 0.000 1 1 2 3.19 25.000 3 1 2 15494 0.009 0.003 0.000
cbdata ReadRequest (34) 72 8 1 2 2.87 0.000 0 0 2 2.87 0.000 8 1 2 827452 0.461 0.065 0.001
cbdata CbDataList (40) 96 6 1 3 2.14 0.000 2 1 3 2.14 33.333 4 1 2 2414811 1.346 0.253 0.002
link_list 16 35 1 7 1.98 0.000 0 0 7 1.98 0.000 35 1 7 2415273 1.346 0.042 0.002
ACLDestinationIP 120 4 1 1 9.60 0.000 4 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata CbDataList (4) 40 12 1 1 9.60 0.000 12 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLStringData 56 8 1 1 9.60 0.000 8 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata CbDataList (1) 40 10 1 1 9.60 0.000 10 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLSourceIP 120 3 1 1 9.60 0.000 3 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
RegexList 88 4 1 1 9.60 0.000 4 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLCertificateData 112 3 1 1 9.60 0.000 3 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
AclDenyInfoList 32 10 1 1 9.60 0.000 10 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata IoResult (35) 40 8 1 1 2.87 0.000 0 0 1 2.87 0.000 8 1 1 827452 0.461 0.036 0.001
cbdata TcpAcceptor (17) 104 3 1 1 9.60 0.000 3 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLRegexData 16 19 1 1 9.60 0.000 19 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLStrategised 136 2 1 1 9.60 0.000 2 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLStrategised 136 2 1 1 9.60 0.000 2 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLStrategised 136 2 1 1 9.60 0.000 2 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLUserData 64 4 1 1 9.60 0.000 4 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata ps_state (27) 256 1 1 1 9.60 0.000 0 0 1 9.60 0.000 1 1 1 382981 0.213 0.107 0.000
cbdata ErrorState (37) 240 1 1 2 2.53 0.000 0 0 2 2.53 0.000 1 1 2 178284 0.099 0.047 0.000
cbdata RemovalPolicy (8) 104 2 1 1 9.60 0.000 2 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
UFSStoreState::_queued_read 40 5 1 1 3.50 0.000 0 0 1 3.50 0.000 5 1 1 41065 0.023 0.002 0.000
Acl::AnyOf 144 1 1 1 9.60 0.000 1 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata CbDataList (5) 64 2 1 1 9.60 0.000 2 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLProxyAuth 128 1 1 1 9.60 0.000 1 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata RedirectStateData (44) 64 2 1 1 3.77 0.000 0 0 1 3.77 0.000 2 1 1 6111 0.003 0.000 0.000
HttpHdrSc 16 8 1 1 1.21 0.000 8 1 1 1.21 100.000 0 0 1 79 0.000 0.000 0.000
cbdata StoreSearchHashIndex (30) 104 1 1 1 9.60 0.000 1 1 1 9.60 100.000 0 0 1 6 0.000 0.000 0.000
ACLDomainData 16 6 1 1 9.60 0.000 6 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLHTTPHeaderData 48 2 1 1 9.60 0.000 2 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLSslErrorData 16 6 1 1 9.60 0.000 6 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
HttpHdrContRange 24 4 1 1 2.40 0.000 4 1 1 2.40 100.000 0 0 1 7924 0.004 0.000 0.000
cbdata AclSizeLimit (7) 48 2 1 1 9.60 0.000 2 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
Auth::QueueNode 32 3 1 1 0.85 0.000 0 0 1 0.85 0.000 3 1 1 3828 0.002 0.000 0.000
ACLServerNameData 16 5 1 1 9.60 0.000 5 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata WriteRequest (42) 80 1 1 1 9.32 0.000 0 0 1 9.32 0.000 1 1 1 2381284 1.327 0.208 0.002
ACLAtStepData 24 3 1 1 9.60 0.000 3 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
StoreSwapLogData 72 1 1 1 9.60 0.000 0 0 1 9.60 0.000 1 1 1 646204 0.360 0.051 0.000
cbdata RemovalPurgeWalker (45) 72 1 1 1 3.15 0.000 0 0 1 3.15 0.000 1 1 1 7898 0.004 0.001 0.000
ACLMethodData 16 3 1 1 9.60 0.000 3 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
dwrite_q 48 1 1 1 9.60 0.000 0 0 1 9.60 0.000 1 1 1 3027600 1.687 0.159 0.002
cbdata StateData (29) 48 1 1 1 2.27 0.000 0 0 1 2.27 0.000 1 1 1 10760 0.006 0.001 0.000
ACLNoteData 40 1 1 1 9.60 0.000 1 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
cbdata IoResult (43) 40 1 1 1 9.32 0.000 0 0 1 9.32 0.000 1 1 1 2381284 1.327 0.104 0.002
ACLASN 16 2 1 1 9.60 0.000 2 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLHierCodeData 32 1 1 1 9.60 0.000 1 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
ACLTimeData 32 1 1 1 9.60 0.000 1 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
CacheDigest 32 1 1 1 9.60 0.000 1 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
StoreMetaMD5 32 1 1 1 9.60 0.000 0 0 1 9.60 0.000 1 1 1 105136 0.059 0.004 0.000
StoreMetaSTDLFS 32 1 1 1 9.60 0.000 0 0 1 9.60 0.000 1 1 1 105136 0.059 0.004 0.000
StoreMetaURL 32 1 1 1 9.60 0.000 0 0 1 9.60 0.000 1 1 1 105136 0.059 0.004 0.000
StoreMetaObjSize 32 1 1 1 9.60 0.000 0 0 1 9.60 0.000 1 1 1 105136 0.059 0.004 0.000
StoreMetaVary 32 1 1 1 9.03 0.000 0 0 1 9.03 0.000 1 1 1 21330 0.012 0.001 0.000
HttpHdrRange 32 1 1 1 2.80 0.000 0 0 1 2.80 0.000 1 1 1 17337 0.010 0.001 0.000
FwdServer 24 1 1 1 9.60 0.000 0 0 1 9.60 0.000 1 1 1 382981 0.213 0.010 0.000
ACLProtocolData 16 1 1 1 9.60 0.000 1 1 1 9.60 100.000 0 0 0 0 0.000 0.000 0.000
HttpHdrRangeSpec 16 1 1 1 2.80 0.000 0 0 1 2.80 0.000 1 1 1 17337 0.010 0.000 0.000
32K Buffer 32768 0 0 96 2.33 0.000 0 0 96 2.33 -1.000 0 0 96 37 0.000 0.001 0.000
64K Buffer 65536 0 0 128 3.70 0.000 0 0 128 3.70 -1.000 0 0 128 1 0.000 0.000 0.000
dlink_node 24 0 0 1 9.59 0.000 0 0 1 9.59 -1.000 0 0 1 146 0.000 0.000 0.000
cbdata RebuildState (16) 688 0 0 1 9.60 0.000 0 0 1 9.60 -1.000 0 0 1 0 0.000 0.000 0.000
Total 1 5654759 1016434 1045039 0.76 100.000 5652896 1015122 1041138 0.77 99.871 1863 1313 9295 172333656 96.047 97.847 0.123
Cumulative allocated volume: 91.517 GB
Current overhead: 45448 bytes (0.004%)
Idle pool limit: 5.00 MB
Total Pools created: 145
Pools ever used: 137 (shown above)
Currently in use: 104
String Pool Impact
(%strings) (%volume)
Short Strings 57 14
Medium Strings 24 19
Long Strings 19 60
1KB Strings 0 0
4KB Strings 0 1
16KB Strings 0 5
Other Strings 0 0
Large buffers: 0 (0 KB)

> On 10 May 2016, at 6:02 PM, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote:
> 
> On 10/05/2016 2:35 p.m., Dan Charlesworth wrote:
>> A small percentage of deployments of our squid-based product are using oodles of memory—there doesn’t seem to be a limit to it.
>> 
>> I’m wondering what the best way might be to analyse what squid is reserving it all for in the latest 3.5 release?
>> 
>> The output of squidclient mgr:cache_mem is completely incomprehensible to me.
> 
> Try mgr:mem report. It is TSV (tab-separated values) file format.
> 
>  squidclient mgr:mem > mem.tsv
> 
> ... and load mem.tsv using your favourite spreadsheet program. The
> column titles should then be self-explanatory.
> 
> Amos
> 
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> http://lists.squid-cache.org/listinfo/squid-users


_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux