We are still chasing this one down but made a major breakthrough. The leak is related to squid in intercept mode + SSL decryption + origin with invalid certs. In our case, the majority of the cases were related to Windows Update and Windows Defender domains, so a stopgap solution is to bypass decryption for these sites (eg, .update.microsoft.com). If you do, don't use dstdomain ACL, as the domain is not available at the time of the checking. Use something like ssl::server_name[_regex].
Hope this helps!
On Fri, Jan 27, 2023 at 2:28 PM Gustavo Carvalho <gustavocarv4872@xxxxxxxxx> wrote:
Hi Hamilton, thanks for helping!
I wish I could provide this log while squid is crashing, but there
have been no incidents since wednesday. From what I've heard, the RAM
on that server's VM has been increased to 32GB.
Anyway, here is the squidclient mgr:mem log output. I hope it can be helpful.
On Thu, Jan 26, 2023 at 5:43 PM Hamilton Coutinho
<hamilton.coutinho@xxxxxxxxx> wrote:
>
> Hi Gustavo,
>
> I'm seeing the same thing. I could narrow down (but can't say with 100% confidence) to the code that does certificate verification when configured for SSL decryption. What is the output of squidclient mgr:mem for you? Do you see unexplainably high counts for in-use objects like HttpRequest, PeekingPeerConnector, Comm::Connection, Security::ErrorDetail?
>
>
> On Thu, Jan 26, 2023 at 12:31 PM Gustavo Carvalho <gustavocarv4872@xxxxxxxxx> wrote:
>>
>> Hi,
>>
>> I have Squid 5.6 on a FreeBSD 13.1 server with 16GB RAM
>>
>> I noticed that squid starts to consume a lot of ram until it starts to
>> consume swap space. When this happens, browsing becomes extremely
>> slow.
>>
>> This is happening at least once a week when I have to restart squid to
>> get it back to normal.
>>
>> Any ideas?
>>
>> ############# Wed Jan 25 08:30:00 -03 2023 #############
>>
>> HTTP/1.1 200 OK
>> Server: squid
>> Mime-Version: 1.0
>> Date: Wed, 25 Jan 2023 11:30:00 GMT
>> Content-Type: text/plain;charset=utf-8
>> Expires: Wed, 25 Jan 2023 11:30:00 GMT
>> Last-Modified: Wed, 25 Jan 2023 11:30:00 GMT
>> X-Cache: MISS from xxxx.xxxx.xxxx
>> X-Cache-Lookup: MISS from xxxx.xxxx.xxxx:3128
>> Via: 1.1 xxxx.xxxx.xxxx (squid)
>> Connection: close
>>
>> Squid Object Cache: Version 5.6
>> Build Info:
>> Service Name: squid
>> Start Time: Thu, 19 Jan 2023 20:25:17 GMT
>> Current Time: Wed, 25 Jan 2023 11:30:00 GMT
>> Connection information for squid:
>> Number of clients accessing cache: 224
>> Number of HTTP requests received: 7541590
>> Number of ICP messages received: 0
>> Number of ICP messages sent: 0
>> Number of queued ICP replies: 0
>> Number of HTCP messages received: 0
>> Number of HTCP messages sent: 0
>> Request failure ratio: 0.00
>> Average HTTP requests per minute since start: 930.5
>> Average ICP messages per minute since start: 0.0
>> Select loop called: 78733524 times, 6.176 ms avg
>> Cache information for squid:
>> Hits as % of all requests: 5min: 8.4%, 60min: 12.1%
>> Hits as % of bytes sent: 5min: 21.6%, 60min: 14.1%
>> Memory hits as % of hit requests: 5min: 90.8%, 60min: 75.9%
>> Disk hits as % of hit requests: 5min: 4.0%, 60min: 19.7%
>> Storage Swap size: 2829956 KB
>> Storage Swap capacity: 90.0% used, 10.0% free
>> Storage Mem size: 16172 KB
>> Storage Mem capacity: 98.7% used, 1.3% free
>> Mean Object Size: 28.95 KB
>> Requests given to unlinkd: 186982
>> Median Service Times (seconds) 5 min 60 min:
>> HTTP Requests (All): 0.00562 0.01847
>> Cache Misses: 0.15048 0.23230
>> Cache Hits: 0.00000 0.00000
>> Near Hits: 0.14252 0.13498
>> Not-Modified Replies: 0.00865 0.03066
>> DNS Lookups: 0.00000 0.00372
>> ICP Queries: 0.00000 0.00000
>> Resource usage for squid:
>> UP Time: 486282.612 seconds
>> CPU Time: 65555.712 seconds
>> CPU Usage: 13.48%
>> CPU Usage, 5 minute avg: 26.89%
>> CPU Usage, 60 minute avg: 68.00%
>> Maximum Resident Size: 37896960 KB
>> Page faults with physical i/o: 10843
>> Memory accounted for:
>> Total accounted: -1459461 KB
>> memPoolAlloc calls: 11408
>> memPoolFree calls: 1888969689
>> File descriptor usage for squid:
>> Maximum number of file descriptors: 4096
>> Largest file desc currently in use: 2149
>> Number of file desc currently in use: 679
>> Files queued for open: 0
>> Available number of file descriptors: 3417
>> Reserved number of file descriptors: 100
>> Store Disk files open: 0
>> Internal Data Structures:
>> 97906 StoreEntries
>> 3002 StoreEntries with MemObjects
>> 2838 Hot Object Cache Items
>> 97742 on-disk objects
>>
>> ------ pfctl -si ------
>>
>> Status: Enabled for 25 days 22:58:24 Debug: Urgent
>>
>> State Table Total Rate
>> current entries 8085
>> searches 6650475717 2965.4/s
>> inserts 133521957 59.5/s
>> removals 133552376 59.5/s
>> Counters
>> match 605960865 270.2/s
>> bad-offset 0 0.0/s
>> fragment 1 0.0/s
>> short 54 0.0/s
>> normalize 659 0.0/s
>> memory 0 0.0/s
>> bad-timestamp 0 0.0/s
>> congestion 0 0.0/s
>> ip-option 0 0.0/s
>> proto-cksum 0 0.0/s
>> state-mismatch 104674 0.0/s
>> state-insert 38501 0.0/s
>> state-limit 0 0.0/s
>> src-limit 0 0.0/s
>> synproxy 0 0.0/s
>> map-failed 0 0.0/s
>>
>> ------ sysctl -a | grep swap ------
>>
>> swap_pager: out of swap space
>> swp_pager_getswapspace(32): failed
>> swap_pager: out of swap space
>> swp_pager_getswapspace(31): failed
>> swap_pager: out of swap space
>> swp_pager_getswapspace(1): failed
>> 1 PART da0p2 2147483648 512 i 2 o 544768 ty freebsd-swap xs GPT xt
>> 516e7cb5-6ecf-11d6-8ff8-00022d09712b
>> 0 MD md1 94371840 512 u 1 s 512 f 0 fs 0 l 94371840 t swap label
>> 0 MD md0 62914560 512 u 0 s 512 f 0 fs 0 l 62914560 t swap label
>> z0xfffff80003ec5800 [shape=box,label="SWAP\nswap\nr#3"];
>> <name>swap</name>
>> <type>swap</type>
>> <type>swap</type>
>> <type>freebsd-swap</type>
>> vm.swap_enabled: 1
>> vm.domain.0.stats.unswappable: 2044
>> vm.swap_idle_threshold2: 10
>> vm.swap_idle_threshold1: 2
>> vm.swap_idle_enabled: 0
>> vm.disable_swapspace_pageouts: 0
>> vm.stats.vm.v_swappgsout: 3154299
>> vm.stats.vm.v_swappgsin: 510404
>> vm.stats.vm.v_swapout: 174446
>> vm.stats.vm.v_swapin: 62590
>> vm.stats.swap.free_completed: 54375
>> vm.stats.swap.free_deferred: 56992
>> vm.nswapdev: 1
>> vm.swap_fragmentation:
>> vm.swap_async_max: 4
>> vm.swap_maxpages: 32572800
>> vm.swap_total: 2147483648
>> vm.swap_reserved: 384676114432
>>
>> ------ /usr/sbin/swapinfo -h ------
>>
>> Device Size Used Avail Capacity
>> /dev/da0p2 2.0G 2.0G 8.0K 100%
>>
>>
>> ############# squid.conf #############
>>
>> http_port 3128 ssl-bump generate-host-certificates=on
>> dynamic_cert_mem_cache_size=20MB cert=/xxxx/conf/certs/ca.crt
>> key=/xxxx/conf/certs/ca.key
>> http_port 3129 intercept
>> https_port 3130 intercept ssl-bump generate-host-certificates=on
>> dynamic_cert_mem_cache_size=20MB cert=/xxxx/conf/certs/ca.crt
>> key=/xxxx/conf/certs/ca.key
>> visible_hostname xxxx.xxxx.xxxx
>> max_filedescriptors 4096
>> maximum_object_size 4096 KB
>> minimum_object_size 0 KB
>> maximum_object_size_in_memory 256 KB
>> fqdncache_size 1024
>> cache_mgr xxxx@xxxx
>> dns_nameservers 127.0.0.1
>> cache_replacement_policy heap LFUDA
>> memory_replacement_policy heap GDSF
>> cache_mem 16 MB
>> cache_dir ufs /xxxx/chroot/osproxy/cache 3072 32 256
>> forwarded_for on
>> memory_pools off
>> logformat xxxx %ts|%6tr|%>a|%Ss|%03>Hs|%<st|%rm|%un|%mt|%ea|%ru
>> logfile_rotate 0
>> httpd_suppress_version_string on
>> strip_query_terms off
>> _______________________________________________
>> squid-users mailing list
>> squid-users@xxxxxxxxxxxxxxxxxxxxx
>> http://lists.squid-cache.org/listinfo/squid-users
>
>
>
> --
> Hamilton
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> http://lists.squid-cache.org/listinfo/squid-users
Hamilton
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
