Hey Ivan,
From what i remember there is a calculation for how much k per conn should squid use.
another thing is that squid is not returning memory once ot took it.
Amos knows about this and might be able to respond.
Eliezer
On Wed, Aug 5, 2020, 20:34 Ivan Bulatovic <ivan.bulatovic@xxxxxxxxx> wrote:
Hi Eliezer,
In the original message I sent to the squid-users mail list, I
attached listings from mgr:info and mgr:mem. The server is definitely
using a lot of connections (close to 200K connections), which is why I
increased the open files limits in linux as well as in squid.conf. And
there are a lot of requests per second, when the server is running. I
could understand that it needs a lot of memory for in-transit cache,
but that memory should be later released back to OS, once the
requests load goes down. However, that is not the case, even days
after there is no load on the server, it stays at 11GB of RAM and 5.5
GB of swap used. The second I restart the squid process, everything
goes to normal, memory is released. That is why I suspect there is
some memory leak somewhere.
The server is a Ubuntu 18.04 LTS VM (running on Hyper-V 2019 server),
with 8 virtual processors and 12GB of RAM (although I can increase
that if that is the problem, but I thought that without caching this
would be more than enough).
I am not using dynamic memory on Hyper-V (it is turned off for this VM).
Best regards,
Ivan
On Wed, Aug 5, 2020 at 7:14 PM NgTech LTD <ngtech1ltd@xxxxxxxxx> wrote:
>
> I think that the mgr:info or another page there contains the amount of requests per second etc.
> also netstat or ss -ntp might give some basic understanding about this server size.
>
> are you using dynamic memory on the hyper-v hypervisor?
>
> Eliezer
>
> On Wed, Aug 5, 2020, 19:59 Ivan Bulatovic <ivan.bulatovic@xxxxxxxxx> wrote:
>>
>> Hi Alex,
>>
>> Thank you very much for your help.
>>
>> I opened a bug on bugs.squid-cache.org
>> (https://bugs.squid-cache.org/show_bug.cgi?id=5071).
>>
>> Best regards,
>> Ivan
>>
>> On Mon, Aug 3, 2020 at 10:02 PM Alex Rousskov
>> <rousskov@xxxxxxxxxxxxxxxxxxxxxxx> wrote:
>> >
>> > On 8/3/20 9:11 AM, Ivan Bulatovic wrote:
>> >
>> > > Looks like squid has some serious memory issues when under heavy load
>> > > (90 servers that crawl Internet sites).
>> >
>> > > Maximum Resident Size: 41500720 KB
>> >
>> > If the above (unreliable) report matches your observations using system
>> > tools like "top", then it is indeed likely that your Squid is suffering
>> > from a memory leak -- 41GB is usually too much for most non-caching
>> > Squid instances.
>> >
>> > Identifying the leak may take some time, and I am not volunteering to do
>> > the necessary legwork personally, but the Squid Project does fix
>> > virtually all runtime leaks that we know about. If you want to speed up
>> > the process, one of the best things you can do is to run Squid under
>> > valgrind with a good suppression file. This requires building Squid with
>> > a special ./configure option. Several testing iterations may be
>> > necessary. If you are willing to do this, please file a bug report and
>> > somebody will guide you through the steps.
>> >
>> >
>> > > It just eats up memory, and
>> > > does not free it up even days after it is being used (with no load on
>> > > the proxy for days).
>> >
>> > Some memory retention is expected by default. See
>> > http://www.squid-cache.org/Doc/config/memory_pools/
>> >
>> > Unfortunately, AFAICT, your mgr:mem output does not show any obvious
>> > leaks -- all numbers are very small. If something is leaking a lot, then
>> > it is probably not pooled by Squid.
>> >
>> >
>> > HTH,
>> >
>> > Alex.
>> >
>> >
>> > > On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic wrote:
>> > >>
>> > >> Hi all,
>> > >>
>> > >> I am trying to configure squid to run as a forward proxy with no
>> > >> caching (cache deny all) with an option to choose the outgoing IP
>> > >> address based on the username. So all squid has to do is to use a
>> > >> certain outgoing IP address for a certain user, return the data from
>> > >> the server to that user and cache nothing.
>> > >>
>> > >> For that I created a special authentication helper and used the ACLs
>> > >> and tcp_outgoing_address to create a lot of users and outgoing IP
>> > >> addresses (about 260 at the moment). Example (not the real IP I use,
>> > >> of course):
>> > >>
>> > >> acl use_IP1 proxy_auth user1
>> > >> tcp_outgoing_address 1.2.3.4 use_IP1
>> > >>
>> > >> I also configured the squid to use 4 workers, but this happens even
>> > >> when I use only one worker (default)
>> > >>
>> > >> And this works. However, under heavy load, Squid eats all of the RAM
>> > >> and then starts going to swap. And the memory usage does not drop when
>> > >> I remove all the load from squid (I shut down all clients).
>> > >>
>> > >> I left it to see if the memory will be freed but even after leaving it
>> > >> for an hour the info page reports this:
>> > >> Cache information for squid:
>> > >> Hits as % of all requests: 5min: 0.0%, 60min: 0.0%
>> > >> Hits as % of bytes sent: 5min: 0.0%, 60min: 1.1%
>> > >> Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
>> > >> Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
>> > >> Storage Swap size: 0 KB
>> > >> Storage Swap capacity: 0.0% used, 100.0% free
>> > >> Storage Mem size: 0 KB
>> > >> Storage Mem capacity: 0.0% used, 100.0% free
>> > >> Mean Object Size: 0.00 KB
>> > >> Requests given to unlinkd: 0
>> > >>
>> > >> Resource usage for squid:
>> > >> UP Time: 255334.875 seconds
>> > >> CPU Time: 7122.436 seconds
>> > >> CPU Usage: 2.79%
>> > >> CPU Usage, 5 minute avg: 0.05%
>> > >> CPU Usage, 60 minute avg: 37.66%
>> > >> Maximum Resident Size: 41500720 KB
>> > >> Page faults with physical i/o: 1003410
>> > >>
>> > >> And here is the listing of free and top commands (with no load on the server):
>> > >>
>> > >> # free -h
>> > >> total used free shared buff/cache available
>> > >> Mem: 11G 10G 791M 676K 491M 1.0G
>> > >> Swap: 11G 5.5G 6.5G
>> > >>
>> > >> # top
>> > >> top - 14:12:32 up 3 days, 1:30, 1 user, load average: 0.00, 0.00, 0.00
>> > >> Tasks: 177 total, 1 running, 102 sleeping, 0 stopped, 0 zombie
>> > >> %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> > >> %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> > >> %Cpu2 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> > >> %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> > >> %Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> > >> %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> > >> %Cpu6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> > >> %Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> > >> KiB Mem : 91.2/12251688
>> > >> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> > >> ]
>> > >> KiB Swap: 45.8/12582904
>> > >> [||||||||||||||||||||||||||||||||||||||||||||||
>> > >> ]
>> > >>
>> > >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> > >> 7851 proxy 20 0 6946872 2.514g 8084 S 0.0 21.5 29:43.74 squid
>> > >> 7832 proxy 20 0 6711480 2.464g 8040 S 0.0 21.1 29:58.17 squid
>> > >> 7814 proxy 20 0 6834928 2.454g 10024 S 0.0 21.0 29:47.56 squid
>> > >> 7843 proxy 20 0 6906252 2.436g 8208 S 0.0 20.8 29:15.60 squid
>> > >> 1329 root 20 0 2416672 167272 12680 S 0.0 1.4 136:18.57 metricbeat
>> > >> 1321 root 20 0 1831804 48364 11648 S 0.0 0.4 14:32.10 filebeat
>> > >> 474 root 19 -1 127796 17576 17144 S 0.0 0.1 0:27.01
>> > >> systemd-journal
>> > >> 7811 proxy 20 0 549384 14168 8372 S 0.0 0.1 0:20.87 squid
>> > >> 1166 root 20 0 1749724 10596 4468 S 0.0 0.1 0:31.83 snapd
>> > >> 43940 proxy 20 0 28884 9608 5384 S 0.0 0.1 0:00.14 python3
>> > >> 43941 proxy 20 0 28884 9552 5328 S 0.0 0.1 0:00.10 python3
>> > >> 43939 proxy 20 0 28884 9524 5308 S 0.0 0.1 0:00.12 python3
>> > >> 43938 proxy 20 0 28884 9452 5232 S 0.0 0.1 0:00.16 python3
>> > >> 48848 root 20 0 105688 6960 5968 S 0.0 0.1 0:00.02 sshd
>> > >> 48974 janitor 20 0 108120 5380 4372 S 0.0 0.0 0:00.00 sshd
>> > >> 1 root 20 0 86360 4364 2488 S 0.0 0.0 32:46.22 systemd
>> > >> ...
>> > >> ... lines ommited
>> > >> ...
>> > >>
>> > >> In the attachment you can find the printout from squidclient mgr:info
>> > >> and squidclient mgr:mem. These are both taken at the moment when there
>> > >> is no more load on the proxy. I also included my squid.conf file
>> > >> (minus the two files where acls are defined and outgoing IP addresses,
>> > >> these two contain only acl and tcp_outgoing_address lines as in the
>> > >> example above).
>> > >>
>> > >> Machine info:
>> > >> OS: Ubuntu 18.04 with latest updates
>> > >> Squid version 4.12 (from diladele repository)
>> > >> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
>> > >>
>> > >> I can not understand what is eating all of the memory, if I disabled the cache.
>> > >>
>> > >> Maybe I configured something wrong but I can not find what.
>> > >>
>> > >> Thank you for any help you can provide.
>> > >>
>> > >> Best regards,
>> > >> Ivan
>> > > _______________________________________________
>> > > squid-users mailing list
>> > > squid-users@xxxxxxxxxxxxxxxxxxxxx
>> > > http://lists.squid-cache.org/listinfo/squid-users
>> > >
>> >
>> _______________________________________________
>> squid-users mailing list
>> squid-users@xxxxxxxxxxxxxxxxxxxxx
>> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
