Hi,
(sending again because ezlm didn't like my attaches before)
we've got squid 2.7.STABLE7, running on Ubuntu 10.04, as reverse
proxy for OWA 2007 (or Exchange 2007; excuse me if there are differences, I
don't know much about Windows products). It has been recompiled with
"--enable-ssl" to be able to answer SSL to the clients and to communicate
using SSL with the backend. We followed the howto at [1] and other, and then
modified the configuration a bit to better suit our needs.
It works well, and it's very fast. But it leaks memory like a sieve.
It goes from 250MB to 2.2GB in 24 hours. In cache.log there are a lot of
lines like these:
2010/10/18 09:36:29| fwdNegotiateSSL: Error negotiating SSL connection on FD
5485: error:00000000:lib(0):func(0):reason(0) (5/-1/104)
2010/10/18 09:36:29| TCP connection to owa (XX.XX.XX.XX:443) failed
("owa" is the backend)
Besides that, nothing. No error, no warning, nothing than can give a
hint as to what's happening. Attached is a snippet of the configuration, with
IP addresses and domains obfuscated (customer's requirement). I think
everything important is there, but don't hesitate to ask for more if it's
needed. I attach also the output from "squidclient mgr:info" and "squidclient
mgr:mem", two hours (more or less) after restarting the process. We tried with
"memory_pools" on and off (current configuration), but it seems to make no
difference. We also tried with "client_db" disabled, but as it meant no
improvement we enabled it again.
Unfortunately, this is a production system, so I can't do much
testing without affecting users. But if something can shed light on the
issue, I can ask for a maintenance window.
I'm planning to update to 2.7.STABLE9 (Ubuntu package, recompiled
with SSL again), and also add the patch from the daily snapshot. It seems that
the only thing it adds is a "memBufClean" call, and to me it looks like
something good when speaking about memory leaks :-) I hope it helps.
Sorry if I make little sense in some of this, it's monday morning. Any
help, opinion or suggestion is welcome. Thanks in advance.
References:
[1]
http://www.tanti.org.uk/index.php/blogs/owencampbell/3-tech/3-proxy
--
Roberto Suarez Soto Allenta Consulting
robe@xxxxxxxxxxx www.allenta.com
+34 881 922 600
Este correo electrÃnico contiene informaciÃn estrictamente confidencial y es
de uso exclusivo del destinatario, quedando prohibida a cualquier otra
persona su revelaciÃn, copia, distribuciÃn, o el ejercicio de cualquier
acciÃn relativa a su contenido. Si ha recibido este mensaje por error, por
favor conteste a su remitente mediante correo electrÃnico y proceda a
borrarlo de su sistema. Rogamos nos comunique inmediatamente sobre cualquier
inconveniente que pueda tener usted en relaciÃn al envÃo de este tipo de
correo electrÃnico.
Sus datos personales serÃn tratados de forma confidencial y no serÃn cedidos
a terceros ajenos a ALLENTA CONSULTING, S.L. En cualquier caso, podrà ejercer
los derecho de oposiciÃn, acceso, rectificaciÃn y cancelaciÃn de acuerdo con
lo establecido en la Ley OrgÃnica 15/99, de 13 de diciembre, de ProtecciÃn de
Datos de CarÃcter Personal dirigiÃndose a ALLENTA CONSULTING, S.L. en
C/Enrique MariÃas 36, 2Â piso, oficina 8, 15009 â A CoruÃa o en la direcciÃn
de electrÃnico info@xxxxxxxxxxx
Current memory usage:
Pool Obj Size Allocated In Use Idle Allocations Saved Hit Rate
(bytes) (#) (KB) high (KB) high (hrs) impact (%total) (#) (KB) high (KB) portion (%alloc) (#) (KB) high (KB) (number) (%num) (%vol) (%num) (number)
2K Buffer 2048 1551 3102 3196 0.11 4 1551 3102 3196 100 0 0 0 0 0.00 0.00 0.00 485938
4K Buffer 4096 3392 13568 13780 0.11 19 3392 13568 13780 100 0 0 0 0 0.00 0.00 0.00 65261
8K Buffer 8192 1491 11928 12040 0.13 17 1491 11928 12040 100 0 0 0 0 0.00 0.00 0.00 664814
16K Buffer 16384 0 0 16 1.65 0 0 0 16 -1 0 0 0 0 0.00 0.00 0.00 3025
32K Buffer 32768 1 32 128 0.33 0 1 32 128 100 0 0 0 0 0.00 0.00 0.00 540
Store Client Buffer 4096 3367 13468 13680 0.11 19 3367 13468 13680 100 0 0 0 0 0.00 0.00 0.00 1109374
acl 64 14 1 1 1.65 0 14 1 1 100 0 0 0 0 0.00 0.00 0.00 14
acl_ip_data 24 8 1 1 1.65 0 8 1 1 100 0 0 0 0 0.00 0.00 0.00 8
acl_list 24 35 1 1 1.65 0 35 1 1 100 0 0 0 0 0.00 0.00 0.00 35
CacheDigest 32 1 1 1 1.65 0 1 1 1 100 0 0 0 0 0.00 0.00 0.00 1
dread_ctrl 56 0 0 1 1.57 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 23082
dwrite_q 48 0 0 1 1.65 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 659
FwdServer 24 3367 79 81 0.11 0 3367 79 81 100 0 0 0 0 0.00 0.00 0.00 87827
HttpReply 168 4946 812 828 0.11 1 4946 812 828 100 0 0 0 0 0.00 0.00 0.00 178007
HttpHeaderEntry 40 40342 1576 1607 0.11 2 40342 1576 1607 100 0 0 0 0 0.00 0.00 0.00 3532152
HttpHdrCc 40 3078 121 123 0.11 0 3078 121 123 100 0 0 0 0 0.00 0.00 0.00 194810
HttpHdrRangeSpec 16 0 0 1 1.47 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 29
HttpHdrRange 16 0 0 1 1.47 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 1137
HttpHdrContRange 24 0 0 1 1.47 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 7039
intlist 16 3 1 1 1.65 0 3 1 1 100 0 0 0 0 0.00 0.00 0.00 3
MemObject 272 3396 903 917 0.11 1 3396 903 917 100 0 0 0 0 0.00 0.00 0.00 96070
mem_node 4112 1627 6534 6767 0.11 9 1627 6534 6767 100 0 0 0 0 0.00 0.00 0.00 619392
relist 80 1 1 1 1.65 0 1 1 1 100 0 0 0 0 0.00 0.00 0.00 1
request_t 1384 6738 9107 9251 0.11 13 6738 9107 9251 100 0 0 0 0 0.00 0.00 0.00 177849
StoreEntry 88 4020 346 351 0.11 0 4020 346 351 100 0 0 0 0 0.00 0.00 0.00 87710
wordlist 16 6 1 1 1.65 0 6 1 1 100 0 0 0 0 0.00 0.00 0.00 14
ClientInfo 344 1262 424 424 0.00 1 1262 424 424 100 0 0 0 0 0.00 0.00 0.00 2227
MD5 digest 16 4020 63 64 0.11 0 4020 63 64 100 0 0 0 0 0.00 0.00 0.00 87754
storeSwapTLV 24 0 0 1 1.57 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 33622
storeSwapLogData 72 0 0 1 1.65 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 643
Short Strings 36 36315 1277 1302 0.11 2 36315 1277 1302 100 0 0 0 0 0.00 0.00 0.00 4199514
Medium Strings 128 14641 1831 1867 0.11 3 14641 1831 1867 100 0 0 0 0 0.00 0.00 0.00 923460
Long Strings 512 42 21 33 0.23 0 42 21 33 100 0 0 0 0 0.00 0.00 0.00 100256
cbdata acl_access (1) 56 32 2 2 1.65 0 32 2 2 100 0 0 0 0 0.00 0.00 0.00 32
cbdata aclCheck_t (2) 344 1 1 2 1.46 0 1 1 2 100 0 0 0 0 0.00 0.00 0.00 423835
cbdata clientHttpRequest (3) 1112 3390 3682 3738 0.11 5 3390 3682 3738 100 0 0 0 0 0.00 0.00 0.00 93347
cbdata ConnStateData (4) 320 3392 1060 1077 0.11 1 3392 1060 1077 100 0 0 0 0 0.00 0.00 0.00 64105
cbdata ErrorState (5) 160 6 1 2 1.51 0 6 1 2 100 0 0 0 0 0.00 0.00 0.00 4864
cbdata FwdState (6) 112 3366 369 374 0.11 1 3366 369 374 100 0 0 0 0 0.00 0.00 0.00 84469
cbdata generic_cbdata (7) 32 0 0 1 1.65 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 10
cbdata HttpStateData (12) 152 3365 500 508 0.11 1 3365 500 508 100 0 0 0 0 0.00 0.00 0.00 84475
cbdata peer (13) 712 2 2 2 1.65 0 2 2 2 100 0 0 0 0 0.00 0.00 0.00 2
cbdata ps_state (14) 200 0 0 1 1.65 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 84469
cbdata RemovalPolicy (15) 104 2 1 1 1.65 0 2 1 1 100 0 0 0 0 0.00 0.00 0.00 2
cbdata RemovalPurgeWalker (17) 72 0 0 1 1.65 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 5913
cbdata store_client (18) 128 3368 421 428 0.11 1 3368 421 428 100 0 0 0 0 0.00 0.00 0.00 96309
event 48 17 1 1 1.51 0 17 1 1 100 0 0 0 0 0.00 0.00 0.00 31599
UFS IO State data 8 0 0 1 0.16 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 9188
cbdata http_port_list (19) 128 1 1 1 1.65 0 1 1 1 100 0 0 0 0 0.00 0.00 0.00 1
cbdata https_port_list (20) 232 3 1 1 1.65 0 3 1 1 100 0 0 0 0 0.00 0.00 0.00 3
cbdata body_size (21) 64 1 1 1 1.65 0 1 1 1 100 0 0 0 0 0.00 0.00 0.00 1
LRU policy node 24 628 15 15 1.17 0 628 15 15 100 0 0 0 0 0.00 0.00 0.00 642
cbdata ConnectStateData (22) 96 0 0 1 1.64 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 70948
close_handler 24 13489 317 322 0.11 0 13489 317 322 100 0 0 0 0 0.00 0.00 0.00 382204
ipcache_entry 128 7 1 1 1.65 0 7 1 1 100 0 0 0 0 0.00 0.00 0.00 8
fqdncache_entry 160 6 1 2 1.65 0 6 1 2 100 0 0 0 0 0.00 0.00 0.00 7
cbdata Logfile (26) 4192 2 9 9 1.65 0 2 9 9 100 0 0 0 0 0.00 0.00 0.00 2
cbdata RebuildState (28) 648 0 0 1 1.65 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 1
pconn_data 32 0 0 1 0.65 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 17693
pconn_fds 32 0 0 1 0.65 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 17693
cbdata storeIOState (30) 128 0 0 1 0.16 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 9188
cbdata LocateVaryState (31) 136 0 0 1 1.57 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 185
VaryData 32 0 0 1 1.57 0 0 0 1 -1 0 0 0 0 0.00 0.00 0.00 185
cbdata StoreUpdateState (32) 4152 0 0 5 1.18 0 0 0 5 -1 0 0 0 0 0.00 0.00 0.00 8
Total - 164742 71571 72615 0.11 100 164742 71571 72615 100 0 0 0 0 0.00 0.00 0.00 14163656
Cumulative allocated volume: 15.00 GB
Current overhead: 19633 bytes (0.027%)
Idle pool limit: 0.00 MB
memPoolAlloc calls: 14163656
memPoolFree calls: 13998913
String Pool Impact
(%strings) (%volume)
Short Strings 71 41
Medium Strings 29 59
Long Strings 0 1
Other Strings 0 0
Large buffers: 0 (0 KB)
Squid Object Cache: Version 2.7.STABLE7
Start Time: Mon, 15 Oct 2010 06:20:44 GMT
Current Time: Mon, 15 Oct 2010 07:51:10 GMT
Connection information for squid:
Number of clients accessing cache: 1184
Number of HTTP requests received: 79626
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 880.6
Average ICP messages per minute since start: 0.0
Select loop called: 4410984 times, 1.230 ms avg
Cache information for squid:
Request Hit Ratios: 5min: 13.9%, 60min: 10.4%
Byte Hit Ratios: 5min: 2.0%, 60min: 2.5%
Request Memory Hit Ratios: 5min: 0.2%, 60min: 0.8%
Request Disk Hit Ratios: 5min: 45.5%, 60min: 71.2%
Storage Swap size: 88552 KB
Storage Mem size: 6528 KB
Mean Object Size: 141.46 KB
Requests given to unlinkd: 8
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.16775 0.10281
Cache Misses: 0.22004 0.14252
Cache Hits: 0.00000 0.00000
Near Hits: 0.00000 0.00286
Not-Modified Replies: 0.00000 0.00000
DNS Lookups: 0.00000 0.00000
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 5425.408 seconds
CPU Time: 270.260 seconds
CPU Usage: 4.98%
CPU Usage, 5 minute avg: 8.04%
CPU Usage, 60 minute avg: 5.24%
Process Data Segment Size via sbrk(): 622536 KB
Maximum Resident Size: 1665616 KB
Page faults with physical i/o: 2
Memory usage for squid via mallinfo():
Total space in arena: 622536 KB
Ordinary blocks: 622025 KB 2397 blks
Small blocks: 0 KB 0 blks
Holding blocks: 3204 KB 1 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 511 KB
Total in use: 625229 KB 100%
Total free: 511 KB 0%
Total size: 625740 KB
Memory accounted for:
Total accounted: 71229 KB
memPoolAlloc calls: 12556001
memPoolFree calls: 12392165
File descriptor usage for squid:
Maximum number of file descriptors: 8192
Largest file desc currently in use: 6741
Number of file desc currently in use: 6735
Files queued for open: 0
Available number of file descriptors: 1457
Reserved number of file descriptors: 100
Store Disk files open: 0
IO loop method: epoll
Internal Data Structures:
3998 StoreEntries
3374 StoreEntries with MemObjects
28 Hot Object Cache Items
626 on-disk objects
https_port YY.YY.YY.75:443 cert=/etc/ssl/certs/wildcard.crt key=/etc/ssl/private/wildcard.key cafile=/etc/ssl/certs/wildcard.ca-bundle vhost defaultsite=mail.domain.com
https_port YY.YY.YY.175:443 cert=/etc/ssl/certs/wildcard.crt key=/etc/ssl/private/wildcard.key cafile=/etc/ssl/certs/wildcard.ca-bundle vhost defaultsite=mail.domain.com
cache_peer XX.XX.XX.XX parent 443 0 no-query no-digest proxy-only originserver login=PASS ssl sslversion=4 name=owa front-end-https=auto sslflags=DONT_VERIFY_PEER
acl EXCH dstdomain mail.domain.com webmail.domain.com
cache_peer_access owa allow EXCH
cache_peer_access owa deny all
never_direct allow EXCH
http_access allow EXCH
http_access deny all
miss_access allow EXCH
miss_access deny all
# For broken Nokia clients
ignore_expect_100 on
high_memory_warning 2048 MB
memory_pools off
cache_mem 50 MB
