Re: How to analyse squid memory usage

[Dan Charlesworth <dan@xxxxxxxxxxx>]

I’ve now got mgr:mem output from a leaky box for comparison but I’m having a hard time spotting where the problem might be.

Would anyone more experienced mind taking at these and seeing if anything jumps out as a source of the high memory usage?

  - The leaky example has 8GB of server memory and 5.7GB was allocated to squid when the snapshot was taken.

  - The normal example has 12GB of server memory and 2.9GB was allocated to Squid when the snapshot was taken.

HTTP/1.1 200 OK
Server: squid/3.5.17
Mime-Version: 1.0
Date: Thu, 12 May 2016 00:12:05 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 12 May 2016 00:12:05 GMT
Last-Modified: Thu, 12 May 2016 00:12:05 GMT
X-Cache: MISS from livestream
X-Cache-Lookup: MISS from livestream:3128
Via: 1.1 livestream (squid/3.5.17)
Connection: close
Current memory usage:
Pool	Obj Size	Chunks							Allocated					In Use					Idle			Allocations Saved			Rate
	(bytes)	KB/ch	obj/ch	(#)	used	free	part	%Frag	(#)	(KB)	high (KB)	high (hrs)	%Tot	(#)	(KB)	high (KB)	high (hrs)	%alloc	(#)	(KB)	high (KB)	(#)	%cnt	%vol	(#)/sec
mem_node	4136								45734	184723	184723	0.00	37.800	45730	184707	184723	0.00	99.991	4	17	3502	2273600	6.913	47.195	0.002
cbdata clientReplyContext (22)	4320								9911	41813	41813	0.10	8.556	9820	41429	41813	0.10	99.082	91	384	806	87900	0.267	1.906	0.000
cbdata ClientSocketContext (21)	4256								9912	41197	41197	0.10	8.430	9820	40815	41197	0.10	99.072	92	383	794	87896	0.267	1.877	0.000
ExternalACLEntry	256								150345	37587	37587	0.00	7.691	150344	37586	37587	0.00	99.999	1	1	1	156184	0.475	0.201	0.000
4K Buffer	4096								7451	29804	29804	0.05	6.099	7425	29700	29804	0.05	99.651	26	104	324	32777	0.100	0.674	0.000
Medium Strings	128								195046	24381	24381	0.00	4.989	195044	24381	24381	0.00	99.999	2	1	30	2725210	8.286	1.751	0.002
Short Strings	40								550190	21492	21492	0.00	4.398	550190	21492	21492	0.00	100.000	0	0	56	8500598	25.847	1.707	0.006
Long Strings	512								38575	19288	19288	0.00	3.947	38563	19282	19288	0.00	99.969	12	6	66	237675	0.723	0.611	0.000
HttpRequest	1784								9911	17267	17267	0.10	3.533	9820	17109	17267	0.10	99.082	91	159	333	118243	0.360	1.059	0.000
StoreEntry	120								101030	11840	11840	0.00	2.423	101026	11839	11840	0.00	99.996	4	1	10	45604	0.139	0.027	0.000
1KB Strings	1024								10290	10290	10706	0.20	2.106	10238	10238	10706	0.20	99.495	52	52	387	761341	2.315	3.913	0.001
HttpHeaderEntry	56								142397	7788	7788	0.00	1.594	142397	7788	7788	0.00	100.000	0	0	30	1086968	3.305	0.305	0.001
NotePairs::Entry	48								139149	6523	6523	0.00	1.335	139136	6522	6523	0.00	99.991	13	1	13	1970338	5.991	0.475	0.001
MemObject	328								15836	5073	5073	0.00	1.038	15832	5072	5073	0.00	99.975	4	2	23	45873	0.139	0.076	0.000
HttpReply	280								18267	4995	4995	0.00	1.022	18265	4995	4995	0.00	99.989	2	1	19	107498	0.327	0.151	0.000
cbdata ClientHttpRequest (20)	392								9912	3795	3795	0.10	0.776	9820	3760	3795	0.10	99.072	92	36	74	87896	0.267	0.173	0.000
16KB Strings	16384								191	3056	4576	0.18	0.625	127	2032	4576	0.18	66.492	64	1024	2320	46119	0.140	3.792	0.000
cbdata Server (18)	520								5095	2588	2623	0.15	0.529	5023	2551	2623	0.15	98.587	72	37	99	46614	0.142	0.122	0.000
cbdata clientStreamNode (23)	128								19822	2478	2478	0.10	0.507	19640	2455	2478	0.10	99.082	182	23	43	175506	0.534	0.113	0.000
LRU policy node	24								102107	2394	2394	0.00	0.490	102107	2394	2394	0.00	100.000	0	0	3	1941	0.006	0.000	0.000
MemBlob	48								35713	1675	1678	0.10	0.343	35663	1672	1678	0.10	99.860	50	3	27	1427569	4.341	0.344	0.001
MD5 digest	16								101030	1579	1579	0.00	0.323	101026	1579	1579	0.00	99.996	4	1	2	53044	0.161	0.004	0.000
cbdata MemBuf (12)	64								18390	1150	1150	0.00	0.235	18390	1150	1150	0.00	100.000	0	0	5	622687	1.893	0.200	0.000
cbdata store_client (29)	160								7343	1148	1148	0.05	0.235	7324	1145	1148	0.05	99.741	19	3	14	63379	0.193	0.051	0.000
Comm::Connection	200								5575	1089	1177	0.18	0.223	5520	1079	1177	0.18	99.013	55	11	67	157168	0.478	0.158	0.000
HttpHdrCc	96								5663	531	531	0.00	0.109	5661	531	531	0.00	99.965	2	1	6	67643	0.206	0.033	0.000
cbdata ServerBump (34)	104								4930	501	501	0.10	0.102	4881	496	501	0.10	99.006	49	5	10	24871	0.076	0.013	0.000
cbdata ClientRequestContext (24)	104								4763	484	484	0.02	0.099	4756	484	484	0.02	99.853	7	1	3	93983	0.286	0.049	0.000
ipcache_entry	128								3264	408	408	0.02	0.083	3248	406	408	0.02	99.510	16	2	5	17518	0.053	0.011	0.000
cbdata idns_query (9)	8696								40	340	340	0.14	0.070	0	0	340	0.14	0.000	40	340	340	48771	0.148	2.129	0.000
16K Buffer	16384								20	320	624	0.51	0.065	7	112	624	0.51	35.000	13	208	448	23640	0.072	1.944	0.000
4KB Strings	4096								72	288	300	0.15	0.059	62	248	300	0.15	86.111	10	40	60	3037	0.009	0.062	0.000
cbdata IdleConnList (33)	4160								52	212	610	0.33	0.043	26	106	610	0.33	50.000	26	106	553	10769	0.033	0.225	0.000
ClientInfo	432								447	189	189	0.00	0.039	447	189	189	0.00	100.000	0	0	0	0	0.000	0.000	0.000
cbdata TunnelStateData (26)	304								312	93	117	0.18	0.019	168	50	117	0.18	53.846	144	43	47	29865	0.091	0.046	0.000
fqdncache_entry	160								464	73	73	0.01	0.015	460	72	73	0.01	99.138	4	1	1	12187	0.037	0.010	0.000
8K Buffer	8192								7	56	64	0.51	0.011	3	24	64	0.51	42.857	4	32	48	5287	0.016	0.217	0.000
2K Buffer	2048								16	32	62	0.48	0.007	9	18	62	0.48	56.250	7	14	56	1645814	5.004	16.917	0.001
cbdata helper_server (11)	240								117	28	28	0.42	0.006	117	28	28	0.42	100.000	0	0	0	0	0.000	0.000	0.000
MimeEntry	128								177	23	23	8.20	0.005	177	23	23	8.20	100.000	0	0	0	0	0.000	0.000	0.000
squidaio_small_bufs	4096								4	16	48	0.46	0.003	0	0	48	0.46	0.000	4	16	48	251683	0.765	5.174	0.000
cbdata Logfile (14)	4192								3	13	13	8.20	0.003	3	13	13	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata ACLFilledChecklist (25)	472								18	9	15	0.43	0.002	4	2	15	0.43	22.222	14	7	15	333924	1.015	0.791	0.000
UFSStoreState::_queued_write	32								247	8	8	0.04	0.002	0	0	8	0.04	0.000	247	8	8	880943	2.679	0.141	0.001
Acl::AndNode	136								50	7	7	8.20	0.001	50	7	7	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata HttpStateData (31)	312								20	7	12	0.51	0.001	7	3	12	0.51	35.000	13	4	9	23459	0.071	0.037	0.000
Acl::NotNode	136								36	5	5	8.20	0.001	36	5	5	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata FwdState (30)	176								25	5	9	0.34	0.001	14	3	9	0.34	56.000	11	2	7	23640	0.072	0.021	0.000
cbdata ConnOpener (28)	136								30	4	5	0.48	0.001	13	2	5	0.48	43.333	17	3	5	44184	0.134	0.030	0.000
link_list	16								247	4	4	0.04	0.001	0	0	4	0.04	0.000	247	4	4	883827	2.687	0.071	0.001
cbdata external_acl (2)	192								17	4	4	8.20	0.001	17	4	4	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata helper (10)	168								19	4	4	8.20	0.001	19	4	4	8.20	100.000	0	0	0	0	0.000	0.000	0.000
external_acl_format	64								42	3	3	8.20	0.001	42	3	3	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata Tree (6)	192								13	3	3	8.20	0.000	13	3	3	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLExternal	128								15	2	2	8.20	0.000	15	2	2	8.20	100.000	0	0	0	0	0.000	0.000	0.000
acl_ip_data	96								15	2	2	8.20	0.000	15	2	2	8.20	100.000	0	0	1	5	0.000	0.000	0.000
cbdata UFSStoreState (35)	176								8	2	4	0.46	0.000	1	1	4	0.46	12.500	7	2	3	9949	0.030	0.009	0.000
cbdata externalAclState (13)	88								14	2	3	0.43	0.000	2	1	3	0.43	14.286	12	2	3	485244	1.475	0.214	0.000
cbdata generic_cbdata (19)	32								33	2	2	0.14	0.000	0	0	2	0.14	0.000	33	2	2	40342	0.123	0.006	0.000
wordlist	16								61	1	1	8.20	0.000	61	1	1	8.20	100.000	0	0	1	19	0.000	0.000	0.000
HttpHdrScTarget	88								11	1	2	0.41	0.000	11	1	2	0.41	100.000	0	0	1	8	0.000	0.000	0.000
cbdata BodyPipe (32)	152								6	1	2	0.48	0.000	5	1	2	0.48	83.333	1	1	2	1813	0.006	0.001	0.000
acl_name_list	72								11	1	1	8.20	0.000	11	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata external_acl_data (3)	48								15	1	1	8.20	0.000	15	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata ErrorState (41)	240								3	1	2	1.74	0.000	0	0	2	1.74	0.000	3	1	2	3981	0.012	0.005	0.000
cbdata DiskThreadsDiskFile (36)	88								8	1	2	0.46	0.000	1	1	2	0.46	12.500	7	1	2	9949	0.030	0.004	0.000
ACLStrategised	136								5	1	1	8.20	0.000	5	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLStrategised	136								5	1	1	8.20	0.000	5	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ev_entry	48								14	1	2	0.48	0.000	11	1	2	0.48	78.571	3	1	1	49613	0.151	0.012	0.000
Helper::Request	48								14	1	2	0.42	0.000	2	1	2	0.42	14.286	12	1	2	455488	1.385	0.110	0.000
aio_thread	40								16	1	1	8.19	0.000	16	1	1	8.19	100.000	0	0	0	0	0.000	0.000	0.000
aio_ctrl	112								5	1	2	0.46	0.000	0	0	2	0.46	0.000	5	1	2	262911	0.799	0.148	0.000
aio_request	104								5	1	2	0.46	0.000	0	0	2	0.46	0.000	5	1	2	262918	0.799	0.137	0.000
squidaio_micro_bufs	128								4	1	2	0.44	0.000	0	0	2	0.44	0.000	4	1	2	11179	0.034	0.007	0.000
cbdata CbDataList (4)	40								12	1	1	8.20	0.000	12	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLStringData	56								8	1	1	8.20	0.000	8	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata CbDataList (1)	40								10	1	1	8.20	0.000	10	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLSourceIP	120								3	1	1	8.20	0.000	3	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLDestinationIP	120								3	1	1	8.20	0.000	3	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLCertificateData	112								3	1	1	8.20	0.000	3	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
AclDenyInfoList	32								10	1	1	8.20	0.000	10	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata TcpAcceptor (16)	104								3	1	1	8.20	0.000	3	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLRegexData	16								18	1	1	8.20	0.000	18	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata ReadRequest (37)	72								4	1	1	0.46	0.000	0	0	1	0.46	0.000	4	1	1	251683	0.765	0.091	0.000
cbdata CbDataList (42)	96								3	1	2	0.50	0.000	0	0	2	0.50	0.000	3	1	2	541376	1.646	0.261	0.000
ACLStrategised	136								2	1	1	8.20	0.000	2	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLStrategised	136								2	1	1	8.20	0.000	2	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLStrategised	136								2	1	1	8.20	0.000	2	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
RegexList	88								3	1	1	8.20	0.000	3	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata ps_state (27)	256								1	1	1	1.19	0.000	0	0	1	1.19	0.000	1	1	1	54661	0.166	0.070	0.000
cbdata RemovalPolicy (7)	104								2	1	1	8.20	0.000	2	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLUserData	64								3	1	1	8.20	0.000	3	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
HttpHdrSc	16								11	1	1	0.41	0.000	11	1	1	0.41	100.000	0	0	1	8	0.000	0.000	0.000
UFSStoreState::_queued_read	40								4	1	1	0.44	0.000	0	0	1	0.44	0.000	4	1	1	2828	0.009	0.001	0.000
cbdata IoResult (38)	40								4	1	1	0.46	0.000	0	0	1	0.46	0.000	4	1	1	251683	0.765	0.051	0.000
cbdata CbDataList (5)	64								2	1	1	8.20	0.000	2	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata StoreSearchHashIndex (17)	104								1	1	1	8.20	0.000	1	1	1	8.20	100.000	0	0	1	7	0.000	0.000	0.000
ACLHTTPHeaderData	48								2	1	1	8.20	0.000	2	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLSslErrorData	16								6	1	1	8.20	0.000	6	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata WriteRequest (39)	80								1	1	1	8.19	0.000	0	0	1	8.19	0.000	1	1	1	881667	2.681	0.354	0.001
ACLAtStepData	24								3	1	1	8.20	0.000	3	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
StoreSwapLogData	72								1	1	1	8.20	0.000	0	0	1	8.20	0.000	1	1	1	95999	0.292	0.035	0.000
ACLMethodData	16								3	1	1	8.20	0.000	3	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
dwrite_q	48								1	1	1	8.20	0.000	0	0	1	8.20	0.000	1	1	1	977678	2.973	0.236	0.001
HttpHdrContRange	24								2	1	1	2.88	0.000	0	0	1	2.88	0.000	2	1	1	345	0.001	0.000	0.000
ACLNoteData	40								1	1	1	8.20	0.000	1	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
cbdata IoResult (40)	40								1	1	1	8.19	0.000	0	0	1	8.19	0.000	1	1	1	881667	2.681	0.177	0.001
ACLASN	16								2	1	1	8.20	0.000	2	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLDomainData	16								2	1	1	8.20	0.000	2	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLHierCodeData	32								1	1	1	8.20	0.000	1	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLTimeData	32								1	1	1	8.20	0.000	1	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
ACLServerNameData	16								2	1	1	8.20	0.000	2	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
CacheDigest	32								1	1	1	8.20	0.000	1	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
StoreMetaMD5	32								1	1	1	8.19	0.000	0	0	1	8.19	0.000	1	1	1	9852	0.030	0.002	0.000
StoreMetaSTDLFS	32								1	1	1	8.19	0.000	0	0	1	8.19	0.000	1	1	1	9852	0.030	0.002	0.000
StoreMetaURL	32								1	1	1	8.19	0.000	0	0	1	8.19	0.000	1	1	1	9852	0.030	0.002	0.000
StoreMetaObjSize	32								1	1	1	8.19	0.000	0	0	1	8.19	0.000	1	1	1	9852	0.030	0.002	0.000
StoreMetaVary	32								1	1	1	8.19	0.000	0	0	1	8.19	0.000	1	1	1	1884	0.006	0.000	0.000
HttpHdrRange	32								1	1	1	2.88	0.000	0	0	1	2.88	0.000	1	1	1	202	0.001	0.000	0.000
FwdServer	24								1	1	1	8.20	0.000	0	0	1	8.20	0.000	1	1	1	54662	0.166	0.007	0.000
ACLProtocolData	16								1	1	1	8.20	0.000	1	1	1	8.20	100.000	0	0	0	0	0.000	0.000	0.000
HttpHdrRangeSpec	16								1	1	1	2.88	0.000	0	0	1	2.88	0.000	1	1	1	202	0.001	0.000	0.000
32K Buffer	32768								0	0	64	0.88	0.000	0	0	64	0.88	-1.000	0	0	64	2	0.000	0.000	0.000
dlink_node	24								0	0	1	8.19	0.000	0	0	1	8.19	-1.000	0	0	1	70	0.000	0.000	0.000
cbdata nsvc (8)	72								0	0	1	3.36	0.000	0	0	1	3.36	-1.000	0	0	1	4	0.000	0.000	0.000
cbdata RebuildState (15)	688								0	0	1	8.20	0.000	0	0	1	8.20	-1.000	0	0	1	0	0.000	0.000	0.000
Total	1								1770758	488686	488686	0.00	100.000	1768829	485604	485604	0.00	99.369	1929	3083	6006	30974128	94.181	96.746	0.022
Cumulative allocated volume: 19.925 GB
Current overhead: 41966 bytes (0.008%)
Idle pool limit: 5.00 MB
Total Pools created: 134
Pools ever used: 126 (shown above)
Currently in use: 95
String Pool	Impact
	(%strings)	(%volume)
Short Strings	69	28
Medium Strings	25	31
Long Strings	5	25
1KB Strings	1	13
4KB Strings	0	0
16KB Strings	0	3
Other Strings	0	0
Large buffers: 0 (0 KB)

Pool	Obj Size	Chunks							Allocated					In Use					Idle			Allocations Saved			Rate
	(bytes)	KB/ch	obj/ch	(#)	used	free	part	%Frag	(#)	(KB)	high (KB)	high (hrs)	%Tot	(#)	(KB)	high (KB)	high (hrs)	%alloc	(#)	(KB)	high (KB)	(#)	%cnt	%vol	(#)/sec
mem_node	4136								64944	262313	265205	2.87	25.807	64854	261950	265205	2.87	99.861	90	364	4036	4445407	2.478	20.090	0.003
ExternalACLEntry	256								998772	249693	249693	0.00	24.566	998771	249693	249693	0.00	100.000	1	1	1	708911	0.395	0.198	0.001
Long Strings	512								437504	218752	224054	0.71	21.522	437498	218749	224054	0.71	99.999	6	3	839	4201687	2.342	2.351	0.003
Medium Strings	128								543962	67996	68087	0.05	6.690	543714	67965	68087	0.05	99.954	248	31	170	12529486	6.983	1.752	0.009
StoreEntry	120								442960	51910	56978	3.15	5.107	442949	51909	56978	3.15	99.998	11	2	102	332274	0.185	0.044	0.001
Short Strings	40								1323856	51714	53226	0.72	5.088	1323436	51697	53226	0.72	99.968	420	17	520	63712274	35.509	2.785	0.045
NotePairs::Entry	48								669665	31391	31391	0.00	3.088	669636	31390	31391	0.00	99.996	29	2	30	9098556	5.071	0.477	0.007
16KB Strings	16384								1232	19712	52272	2.52	1.939	1231	19696	52272	2.52	99.919	1	16	4576	782012	0.436	14.000	0.001
LRU policy node	24								460379	10791	11683	2.88	1.062	460370	10790	11683	2.88	99.998	9	1	27	69551	0.039	0.002	0.000
HttpHeaderEntry	56								179215	9801	10985	1.21	0.964	178979	9788	10985	1.21	99.868	236	13	350	13955943	7.778	0.854	0.010
MD5 digest	16								442960	6922	7597	3.15	0.681	442949	6922	7597	3.15	99.998	11	1	14	398517	0.222	0.007	0.001
MemObject	328								17563	5626	6040	0.98	0.553	17539	5618	6040	0.98	99.863	24	8	223	378802	0.211	0.136	0.000
4KB Strings	4096								1259	5036	5036	0.01	0.495	1247	4988	5036	0.01	99.047	12	48	748	127748	0.071	0.572	0.000
HttpReply	280								17569	4805	5159	1.21	0.473	17543	4797	5159	1.21	99.852	26	8	191	1145088	0.638	0.350	0.001
cbdata clientReplyContext (22)	4320								1120	4725	12285	2.52	0.465	1119	4721	12285	2.52	99.911	1	5	958	583958	0.325	2.757	0.000
cbdata ClientSocketContext (21)	4256								1120	4655	12103	2.52	0.458	1119	4651	12103	2.52	99.911	1	5	944	583958	0.325	2.716	0.000
HttpRequest	1784								1139	1985	5086	2.52	0.195	1138	1983	5086	2.52	99.912	1	2	394	584066	0.326	1.139	0.000
cbdata MemBuf (13)	64								17790	1112	1195	1.21	0.109	17756	1110	1195	1.21	99.809	34	3	44	3950213	2.202	0.276	0.003
HttpHdrCc	96								10688	1002	1061	0.54	0.099	10641	998	1061	0.54	99.560	47	5	49	494286	0.275	0.052	0.000
4K Buffer	4096								215	860	880	1.12	0.085	207	828	880	1.12	96.279	8	32	120	394673	0.220	1.766	0.000
Comm::Connection	200								3692	722	1798	2.27	0.071	3681	719	1798	2.27	99.702	11	3	146	1708339	0.952	0.373	0.001
1KB Strings	1024								707	707	1846	1.86	0.070	634	634	1846	1.86	89.675	73	73	289	3827224	2.133	4.282	0.003
cbdata Server (18)	520								1273	647	1677	2.53	0.064	1272	646	1677	2.53	99.921	1	1	150	401690	0.224	0.228	0.000
cbdata IdleConnList (39)	4160								133	541	1187	1.25	0.053	125	508	1187	1.25	93.985	8	33	646	70268	0.039	0.319	0.000
cbdata ClientHttpRequest (20)	392								1120	429	1115	2.52	0.042	1119	429	1115	2.52	99.911	1	1	87	583958	0.325	0.250	0.000
cbdata idns_query (10)	8696								49	417	2523	3.10	0.041	12	102	2523	3.10	24.490	37	315	2234	253002	0.141	2.404	0.000
ClientInfo	432								862	364	364	0.00	0.036	862	364	364	0.00	100.000	0	0	0	0	0.000	0.000	0.000
cbdata TunnelStateData (26)	304								1108	329	856	2.52	0.032	1107	329	856	2.52	99.910	1	1	62	186431	0.104	0.062	0.000
cbdata clientStreamNode (23)	128								2240	280	728	2.52	0.028	2238	280	728	2.52	99.911	2	1	57	1167910	0.651	0.163	0.001
16K Buffer	16384								17	272	736	2.14	0.027	6	96	736	2.14	35.294	11	176	624	190937	0.106	3.418	0.000
MemBlob	48								4668	219	330	1.27	0.022	4667	219	330	1.27	99.979	1	1	20	3844886	2.143	0.202	0.003
fqdncache_entry	160								929	146	147	1.67	0.014	920	144	147	1.67	99.031	9	2	18	25113	0.014	0.004	0.000
ipcache_entry	128								1025	129	154	3.10	0.013	994	125	154	3.10	96.976	31	4	33	158901	0.089	0.022	0.000
Auth::Basic::User	192								351	66	85	1.66	0.006	350	66	85	1.66	99.715	1	1	20	285990	0.159	0.060	0.000
8K Buffer	8192								7	56	240	2.14	0.006	2	16	240	2.14	28.571	5	40	208	580579	0.324	5.197	0.000
cbdata helper_server (12)	240								207	49	49	1.81	0.005	207	49	49	1.81	100.000	0	0	0	0	0.000	0.000	0.000
2K Buffer	2048								22	44	106	2.14	0.004	7	14	106	2.14	31.818	15	30	96	9669241	5.389	21.638	0.007
squidaio_small_bufs	4096								8	32	60	2.87	0.003	0	0	60	2.87	0.000	8	32	60	827452	0.461	3.703	0.001
cbdata ACLFilledChecklist (25)	472								58	27	69	3.10	0.003	27	13	69	3.10	46.552	31	15	68	2036246	1.135	1.050	0.001
MimeEntry	128								177	23	23	9.60	0.002	177	23	23	9.60	100.000	0	0	0	0	0.000	0.000	0.000
AuthUserIP	64								345	22	28	1.66	0.002	345	22	28	1.66	100.000	0	0	7	924	0.001	0.000	0.000
acl_proxy_auth_match_cache	40								345	14	18	1.66	0.001	345	14	18	1.66	100.000	0	0	4	10	0.000	0.000	0.000
cbdata Logfile (15)	4192								3	13	13	9.60	0.001	3	13	13	9.60	100.000	0	0	0	0	0.000	0.000	0.000
Auth::Basic::UserRequest	48								182	9	37	1.86	0.001	151	8	37	1.86	82.967	31	2	9	279564	0.156	0.015	0.000
AuthUserHashPointer	24								348	9	11	1.66	0.001	348	9	11	1.66	100.000	0	0	3	14	0.000	0.000	0.000
Acl::AndNode	136								61	9	9	9.60	0.001	61	9	9	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata ClientRequestContext (24)	104								55	6	15	3.10	0.001	24	3	15	3.10	43.636	31	4	15	594478	0.331	0.068	0.000
Acl::NotNode	136								42	6	6	9.60	0.001	42	6	6	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata HttpStateData (38)	312								16	5	14	2.14	0.000	5	2	14	2.14	31.250	11	4	12	178489	0.099	0.061	0.000
cbdata helper (11)	168								21	4	4	9.60	0.000	21	4	4	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata store_client (31)	160								22	4	10	2.14	0.000	7	2	10	2.14	31.818	15	3	8	431019	0.240	0.075	0.000
cbdata FwdState (36)	176								20	4	9	2.14	0.000	6	2	9	2.14	30.000	14	3	8	181140	0.101	0.035	0.000
cbdata external_acl (2)	192								18	4	4	9.60	0.000	18	4	4	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata Tree (6)	192								16	3	3	9.60	0.000	16	3	3	9.60	100.000	0	0	0	0	0.000	0.000	0.000
external_acl_format	64								43	3	3	9.60	0.000	43	3	3	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata ConnOpener (28)	136								20	3	19	1.97	0.000	2	1	19	1.97	10.000	18	3	19	386491	0.215	0.057	0.000
squidaio_micro_bufs	128								20	3	10	2.41	0.000	0	0	10	2.41	0.000	20	3	10	206396	0.115	0.029	0.000
cbdata externalAclState (14)	88								27	3	11	3.10	0.000	1	1	11	3.10	3.704	26	3	11	2298241	1.281	0.221	0.002
acl_ip_data	96								24	3	3	9.60	0.000	24	3	3	9.60	100.000	0	0	1	7	0.000	0.000	0.000
ACLExternal	128								18	3	3	9.60	0.000	18	3	3	9.60	100.000	0	0	0	0	0.000	0.000	0.000
aio_ctrl	112								20	3	9	2.41	0.000	0	0	9	2.41	0.000	20	3	9	1034453	0.577	0.127	0.001
aio_request	104								20	3	8	2.41	0.000	0	0	8	2.41	0.000	20	3	8	1034468	0.577	0.118	0.001
cbdata UFSStoreState (32)	176								11	2	5	2.59	0.000	0	0	5	2.59	0.000	11	2	5	105893	0.059	0.020	0.000
ACLStrategised	136								13	2	2	9.60	0.000	13	2	2	9.60	100.000	0	0	0	0	0.000	0.000	0.000
wordlist	16								80	2	2	9.60	0.000	80	2	2	9.60	100.000	0	0	1	25	0.000	0.000	0.000
cbdata generic_cbdata (19)	32								40	2	10	3.10	0.000	8	1	10	3.10	20.000	32	1	9	250316	0.140	0.009	0.000
UFSStoreState::_queued_write	32								35	2	14	1.98	0.000	0	0	14	1.98	0.000	35	2	14	2373675	1.323	0.083	0.002
cbdata DiskThreadsDiskFile (33)	88								11	1	3	2.59	0.000	0	0	3	2.59	0.000	11	1	3	105893	0.059	0.010	0.000
Helper::Request	48								19	1	3	2.14	0.000	1	1	3	2.14	5.263	18	1	3	2075987	1.157	0.109	0.001
cbdata external_acl_data (3)	48								18	1	1	9.60	0.000	18	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
acl_name_list	72								11	1	1	9.60	0.000	11	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ev_entry	48								15	1	1	1.23	0.000	11	1	1	1.23	73.333	4	1	1	65014	0.036	0.003	0.000
HttpHdrScTarget	88								8	1	2	1.21	0.000	8	1	2	1.21	100.000	0	0	1	79	0.000	0.000	0.000
ACLStrategised	136								5	1	1	9.60	0.000	5	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
aio_thread	40								16	1	1	9.60	0.000	16	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata BodyPipe (41)	152								4	1	2	3.19	0.000	1	1	2	3.19	25.000	3	1	2	15494	0.009	0.003	0.000
cbdata ReadRequest (34)	72								8	1	2	2.87	0.000	0	0	2	2.87	0.000	8	1	2	827452	0.461	0.065	0.001
cbdata CbDataList (40)	96								6	1	3	2.14	0.000	2	1	3	2.14	33.333	4	1	2	2414811	1.346	0.253	0.002
link_list	16								35	1	7	1.98	0.000	0	0	7	1.98	0.000	35	1	7	2415273	1.346	0.042	0.002
ACLDestinationIP	120								4	1	1	9.60	0.000	4	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata CbDataList (4)	40								12	1	1	9.60	0.000	12	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLStringData	56								8	1	1	9.60	0.000	8	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata CbDataList (1)	40								10	1	1	9.60	0.000	10	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLSourceIP	120								3	1	1	9.60	0.000	3	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
RegexList	88								4	1	1	9.60	0.000	4	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLCertificateData	112								3	1	1	9.60	0.000	3	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
AclDenyInfoList	32								10	1	1	9.60	0.000	10	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata IoResult (35)	40								8	1	1	2.87	0.000	0	0	1	2.87	0.000	8	1	1	827452	0.461	0.036	0.001
cbdata TcpAcceptor (17)	104								3	1	1	9.60	0.000	3	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLRegexData	16								19	1	1	9.60	0.000	19	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLStrategised	136								2	1	1	9.60	0.000	2	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLStrategised	136								2	1	1	9.60	0.000	2	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLStrategised	136								2	1	1	9.60	0.000	2	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLUserData	64								4	1	1	9.60	0.000	4	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata ps_state (27)	256								1	1	1	9.60	0.000	0	0	1	9.60	0.000	1	1	1	382981	0.213	0.107	0.000
cbdata ErrorState (37)	240								1	1	2	2.53	0.000	0	0	2	2.53	0.000	1	1	2	178284	0.099	0.047	0.000
cbdata RemovalPolicy (8)	104								2	1	1	9.60	0.000	2	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
UFSStoreState::_queued_read	40								5	1	1	3.50	0.000	0	0	1	3.50	0.000	5	1	1	41065	0.023	0.002	0.000
Acl::AnyOf	144								1	1	1	9.60	0.000	1	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata CbDataList (5)	64								2	1	1	9.60	0.000	2	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLProxyAuth	128								1	1	1	9.60	0.000	1	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata RedirectStateData (44)	64								2	1	1	3.77	0.000	0	0	1	3.77	0.000	2	1	1	6111	0.003	0.000	0.000
HttpHdrSc	16								8	1	1	1.21	0.000	8	1	1	1.21	100.000	0	0	1	79	0.000	0.000	0.000
cbdata StoreSearchHashIndex (30)	104								1	1	1	9.60	0.000	1	1	1	9.60	100.000	0	0	1	6	0.000	0.000	0.000
ACLDomainData	16								6	1	1	9.60	0.000	6	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLHTTPHeaderData	48								2	1	1	9.60	0.000	2	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLSslErrorData	16								6	1	1	9.60	0.000	6	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
HttpHdrContRange	24								4	1	1	2.40	0.000	4	1	1	2.40	100.000	0	0	1	7924	0.004	0.000	0.000
cbdata AclSizeLimit (7)	48								2	1	1	9.60	0.000	2	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
Auth::QueueNode	32								3	1	1	0.85	0.000	0	0	1	0.85	0.000	3	1	1	3828	0.002	0.000	0.000
ACLServerNameData	16								5	1	1	9.60	0.000	5	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata WriteRequest (42)	80								1	1	1	9.32	0.000	0	0	1	9.32	0.000	1	1	1	2381284	1.327	0.208	0.002
ACLAtStepData	24								3	1	1	9.60	0.000	3	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
StoreSwapLogData	72								1	1	1	9.60	0.000	0	0	1	9.60	0.000	1	1	1	646204	0.360	0.051	0.000
cbdata RemovalPurgeWalker (45)	72								1	1	1	3.15	0.000	0	0	1	3.15	0.000	1	1	1	7898	0.004	0.001	0.000
ACLMethodData	16								3	1	1	9.60	0.000	3	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
dwrite_q	48								1	1	1	9.60	0.000	0	0	1	9.60	0.000	1	1	1	3027600	1.687	0.159	0.002
cbdata StateData (29)	48								1	1	1	2.27	0.000	0	0	1	2.27	0.000	1	1	1	10760	0.006	0.001	0.000
ACLNoteData	40								1	1	1	9.60	0.000	1	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
cbdata IoResult (43)	40								1	1	1	9.32	0.000	0	0	1	9.32	0.000	1	1	1	2381284	1.327	0.104	0.002
ACLASN	16								2	1	1	9.60	0.000	2	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLHierCodeData	32								1	1	1	9.60	0.000	1	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
ACLTimeData	32								1	1	1	9.60	0.000	1	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
CacheDigest	32								1	1	1	9.60	0.000	1	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
StoreMetaMD5	32								1	1	1	9.60	0.000	0	0	1	9.60	0.000	1	1	1	105136	0.059	0.004	0.000
StoreMetaSTDLFS	32								1	1	1	9.60	0.000	0	0	1	9.60	0.000	1	1	1	105136	0.059	0.004	0.000
StoreMetaURL	32								1	1	1	9.60	0.000	0	0	1	9.60	0.000	1	1	1	105136	0.059	0.004	0.000
StoreMetaObjSize	32								1	1	1	9.60	0.000	0	0	1	9.60	0.000	1	1	1	105136	0.059	0.004	0.000
StoreMetaVary	32								1	1	1	9.03	0.000	0	0	1	9.03	0.000	1	1	1	21330	0.012	0.001	0.000
HttpHdrRange	32								1	1	1	2.80	0.000	0	0	1	2.80	0.000	1	1	1	17337	0.010	0.001	0.000
FwdServer	24								1	1	1	9.60	0.000	0	0	1	9.60	0.000	1	1	1	382981	0.213	0.010	0.000
ACLProtocolData	16								1	1	1	9.60	0.000	1	1	1	9.60	100.000	0	0	0	0	0.000	0.000	0.000
HttpHdrRangeSpec	16								1	1	1	2.80	0.000	0	0	1	2.80	0.000	1	1	1	17337	0.010	0.000	0.000
32K Buffer	32768								0	0	96	2.33	0.000	0	0	96	2.33	-1.000	0	0	96	37	0.000	0.001	0.000
64K Buffer	65536								0	0	128	3.70	0.000	0	0	128	3.70	-1.000	0	0	128	1	0.000	0.000	0.000
dlink_node	24								0	0	1	9.59	0.000	0	0	1	9.59	-1.000	0	0	1	146	0.000	0.000	0.000
cbdata RebuildState (16)	688								0	0	1	9.60	0.000	0	0	1	9.60	-1.000	0	0	1	0	0.000	0.000	0.000
Total	1								5654759	1016434	1045039	0.76	100.000	5652896	1015122	1041138	0.77	99.871	1863	1313	9295	172333656	96.047	97.847	0.123
Cumulative allocated volume: 91.517 GB
Current overhead: 45448 bytes (0.004%)
Idle pool limit: 5.00 MB
Total Pools created: 145
Pools ever used: 137 (shown above)
Currently in use: 104
String Pool	Impact
	(%strings)	(%volume)
Short Strings	57	14
Medium Strings	24	19
Long Strings	19	60
1KB Strings	0	0
4KB Strings	0	1
16KB Strings	0	5
Other Strings	0	0
Large buffers: 0 (0 KB)

Thanks!

> On 11 May 2016, at 2:37 PM, Dan Charlesworth <dan@xxxxxxxxxxx> wrote:
> 
> Thanks Amos -
> 
> Not sure how self-explanatory the output is, though.
> 
> I’ve attached the output from a site with a 12GB server where top was showing 2.9GB allocated to squid (this is normal e.g. “the control"). But the mem output shows the allocated total as ~1GB, apparently?
> 
> Maybe things will become clearer once I have a “leaky” server’s output to compare with it.
> 
> <bdc-mem.tsv>
> 
>> On 10 May 2016, at 6:02 PM, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote:
>> 
>> On 10/05/2016 2:35 p.m., Dan Charlesworth wrote:
>>> A small percentage of deployments of our squid-based product are using oodles of memory—there doesn’t seem to be a limit to it.
>>> 
>>> I’m wondering what the best way might be to analyse what squid is reserving it all for in the latest 3.5 release?
>>> 
>>> The output of squidclient mgr:cache_mem is completely incomprehensible to me.
>> 
>> Try mgr:mem report. It is TSV (tab-separated values) file format.
>> 
>> squidclient mgr:mem > mem.tsv
>> 
>> ... and load mem.tsv using your favourite spreadsheet program. The
>> column titles should then be self-explanatory.
>> 
>> Amos
>> 
>> _______________________________________________
>> squid-users mailing list
>> squid-users@xxxxxxxxxxxxxxxxxxxxx
>> http://lists.squid-cache.org/listinfo/squid-users
> 

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users