Search squid archive

Re: Squid use all memory ram

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



We are still chasing this one down but made a major breakthrough. The leak is related to squid in intercept mode + SSL decryption + origin with invalid certs. In our case, the majority of the cases were related to Windows Update and Windows Defender domains, so a stopgap solution is to bypass decryption for these sites (eg, .update.microsoft.com). If you do, don't use dstdomain ACL, as the domain is not available at the time of the checking. Use something like ssl::server_name[_regex].

Hope this helps!

On Fri, Jan 27, 2023 at 2:28 PM Gustavo Carvalho <gustavocarv4872@xxxxxxxxx> wrote:
Hi Hamilton, thanks for helping!

I wish I could provide this log while squid is crashing, but there
have been no incidents since wednesday. From what I've heard, the RAM
on that server's VM has been increased to 32GB.

Anyway, here is the squidclient mgr:mem log output. I hope it can be helpful.

On Thu, Jan 26, 2023 at 5:43 PM Hamilton Coutinho
<hamilton.coutinho@xxxxxxxxx> wrote:
>
> Hi Gustavo,
>
> I'm seeing the same thing. I could narrow down (but can't say with 100% confidence) to the code that does certificate verification when configured for SSL decryption. What is the output of squidclient mgr:mem for you? Do you see unexplainably high counts for in-use objects like HttpRequest, PeekingPeerConnector, Comm::Connection, Security::ErrorDetail?
>
>
> On Thu, Jan 26, 2023 at 12:31 PM Gustavo Carvalho <gustavocarv4872@xxxxxxxxx> wrote:
>>
>> Hi,
>>
>> I have Squid 5.6 on a FreeBSD 13.1 server with 16GB RAM
>>
>> I noticed that squid starts to consume a lot of ram until it starts to
>> consume swap space. When this happens, browsing becomes extremely
>> slow.
>>
>> This is happening at least once a week when I have to restart squid to
>> get it back to normal.
>>
>> Any ideas?
>>
>> ############# Wed Jan 25 08:30:00 -03 2023 #############
>>
>> HTTP/1.1 200 OK
>> Server: squid
>> Mime-Version: 1.0
>> Date: Wed, 25 Jan 2023 11:30:00 GMT
>> Content-Type: text/plain;charset=utf-8
>> Expires: Wed, 25 Jan 2023 11:30:00 GMT
>> Last-Modified: Wed, 25 Jan 2023 11:30:00 GMT
>> X-Cache: MISS from xxxx.xxxx.xxxx
>> X-Cache-Lookup: MISS from xxxx.xxxx.xxxx:3128
>> Via: 1.1 xxxx.xxxx.xxxx (squid)
>> Connection: close
>>
>> Squid Object Cache: Version 5.6
>> Build Info:
>> Service Name: squid
>> Start Time: Thu, 19 Jan 2023 20:25:17 GMT
>> Current Time: Wed, 25 Jan 2023 11:30:00 GMT
>> Connection information for squid:
>>      Number of clients accessing cache: 224
>>      Number of HTTP requests received: 7541590
>>      Number of ICP messages received: 0
>>      Number of ICP messages sent: 0
>>      Number of queued ICP replies: 0
>>      Number of HTCP messages received: 0
>>      Number of HTCP messages sent: 0
>>      Request failure ratio: 0.00
>>      Average HTTP requests per minute since start: 930.5
>>      Average ICP messages per minute since start: 0.0
>>      Select loop called: 78733524 times, 6.176 ms avg
>> Cache information for squid:
>>      Hits as % of all requests: 5min: 8.4%, 60min: 12.1%
>>      Hits as % of bytes sent: 5min: 21.6%, 60min: 14.1%
>>      Memory hits as % of hit requests: 5min: 90.8%, 60min: 75.9%
>>      Disk hits as % of hit requests: 5min: 4.0%, 60min: 19.7%
>>      Storage Swap size: 2829956 KB
>>      Storage Swap capacity: 90.0% used, 10.0% free
>>      Storage Mem size: 16172 KB
>>      Storage Mem capacity: 98.7% used,  1.3% free
>>      Mean Object Size: 28.95 KB
>>      Requests given to unlinkd: 186982
>> Median Service Times (seconds)  5 min    60 min:
>> HTTP Requests (All):   0.00562  0.01847
>>      Cache Misses:          0.15048  0.23230
>>      Cache Hits:            0.00000  0.00000
>>      Near Hits:             0.14252  0.13498
>>      Not-Modified Replies:  0.00865  0.03066
>>      DNS Lookups:           0.00000  0.00372
>>      ICP Queries:           0.00000  0.00000
>> Resource usage for squid:
>>      UP Time: 486282.612 seconds
>>      CPU Time: 65555.712 seconds
>>      CPU Usage: 13.48%
>>      CPU Usage, 5 minute avg: 26.89%
>>      CPU Usage, 60 minute avg: 68.00%
>>      Maximum Resident Size: 37896960 KB
>>      Page faults with physical i/o: 10843
>> Memory accounted for:
>>      Total accounted:       -1459461 KB
>>      memPoolAlloc calls:     11408
>>      memPoolFree calls:  1888969689
>> File descriptor usage for squid:
>>      Maximum number of file descriptors:   4096
>>      Largest file desc currently in use:   2149
>>      Number of file desc currently in use:  679
>>      Files queued for open:                   0
>>      Available number of file descriptors: 3417
>>      Reserved number of file descriptors:   100
>>      Store Disk files open:                   0
>> Internal Data Structures:
>>      97906 StoreEntries
>>      3002 StoreEntries with MemObjects
>>      2838 Hot Object Cache Items
>>      97742 on-disk objects
>>
>> ------ pfctl -si ------
>>
>> Status: Enabled for 25 days 22:58:24          Debug: Urgent
>>
>> State Table                          Total             Rate
>>   current entries                     8085
>>   searches                      6650475717         2965.4/s
>>   inserts                        133521957           59.5/s
>>   removals                       133552376           59.5/s
>> Counters
>>   match                          605960865          270.2/s
>>   bad-offset                             0            0.0/s
>>   fragment                               1            0.0/s
>>   short                                 54            0.0/s
>>   normalize                            659            0.0/s
>>   memory                                 0            0.0/s
>>   bad-timestamp                          0            0.0/s
>>   congestion                             0            0.0/s
>>   ip-option                              0            0.0/s
>>   proto-cksum                            0            0.0/s
>>   state-mismatch                    104674            0.0/s
>>   state-insert                       38501            0.0/s
>>   state-limit                            0            0.0/s
>>   src-limit                              0            0.0/s
>>   synproxy                               0            0.0/s
>>   map-failed                             0            0.0/s
>>
>> ------ sysctl -a | grep swap  ------
>>
>> swap_pager: out of swap space
>> swp_pager_getswapspace(32): failed
>> swap_pager: out of swap space
>> swp_pager_getswapspace(31): failed
>> swap_pager: out of swap space
>> swp_pager_getswapspace(1): failed
>> 1 PART da0p2 2147483648 512 i 2 o 544768 ty freebsd-swap xs GPT xt
>> 516e7cb5-6ecf-11d6-8ff8-00022d09712b
>> 0 MD md1 94371840 512 u 1 s 512 f 0 fs 0 l 94371840 t swap label
>> 0 MD md0 62914560 512 u 0 s 512 f 0 fs 0 l 62914560 t swap label
>> z0xfffff80003ec5800 [shape=box,label="SWAP\nswap\nr#3"];
>>       <name>swap</name>
>>     <type>swap</type>
>>     <type>swap</type>
>>     <type>freebsd-swap</type>
>> vm.swap_enabled: 1
>> vm.domain.0.stats.unswappable: 2044
>> vm.swap_idle_threshold2: 10
>> vm.swap_idle_threshold1: 2
>> vm.swap_idle_enabled: 0
>> vm.disable_swapspace_pageouts: 0
>> vm.stats.vm.v_swappgsout: 3154299
>> vm.stats.vm.v_swappgsin: 510404
>> vm.stats.vm.v_swapout: 174446
>> vm.stats.vm.v_swapin: 62590
>> vm.stats.swap.free_completed: 54375
>> vm.stats.swap.free_deferred: 56992
>> vm.nswapdev: 1
>> vm.swap_fragmentation:
>> vm.swap_async_max: 4
>> vm.swap_maxpages: 32572800
>> vm.swap_total: 2147483648
>> vm.swap_reserved: 384676114432
>>
>> ------ /usr/sbin/swapinfo -h  ------
>>
>> Device              Size     Used    Avail Capacity
>> /dev/da0p2          2.0G     2.0G     8.0K   100%
>>
>>
>> ############# squid.conf #############
>>
>> http_port 3128 ssl-bump generate-host-certificates=on
>> dynamic_cert_mem_cache_size=20MB cert=/xxxx/conf/certs/ca.crt
>> key=/xxxx/conf/certs/ca.key
>> http_port 3129 intercept
>> https_port 3130 intercept ssl-bump generate-host-certificates=on
>> dynamic_cert_mem_cache_size=20MB cert=/xxxx/conf/certs/ca.crt
>> key=/xxxx/conf/certs/ca.key
>> visible_hostname xxxx.xxxx.xxxx
>> max_filedescriptors 4096
>> maximum_object_size 4096 KB
>> minimum_object_size 0 KB
>> maximum_object_size_in_memory 256 KB
>> fqdncache_size 1024
>> cache_mgr xxxx@xxxx
>> dns_nameservers 127.0.0.1
>> cache_replacement_policy                heap LFUDA
>> memory_replacement_policy               heap GDSF
>> cache_mem 16 MB
>> cache_dir               ufs /xxxx/chroot/osproxy/cache 3072 32 256
>> forwarded_for on
>> memory_pools off
>> logformat xxxx %ts|%6tr|%>a|%Ss|%03>Hs|%<st|%rm|%un|%mt|%ea|%ru
>> logfile_rotate 0
>> httpd_suppress_version_string on
>> strip_query_terms off
>> _______________________________________________
>> squid-users mailing list
>> squid-users@xxxxxxxxxxxxxxxxxxxxx
>> http://lists.squid-cache.org/listinfo/squid-users
>
>
>
> --
> Hamilton
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> http://lists.squid-cache.org/listinfo/squid-users


--
Hamilton
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users

[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux