So I'm testing squid 3.5.26 on an m3.xlarge w/ 14GB of RAM. Squid is the only "real" service running (sshd and the like). I'm running 4 workers, and 2 rock cache. The workers seem to be growing unbounded and given ~30min or so will cause the kernel to start killing off processes until memory is freed. Yes, my clients (32 of them) are hitting this at about 250 URL's/min which doesn't seem that crazy, but ¯\_(ツ)_/¯ cache_mem 1 GB resulted in workers exceeding 4GB resident. So I tried 500 MB, same problem. Now I'm down to 250 MB and I'm still seeing workers using 3-4GB of RAM after a few minutes and still growing which is surprising since the docs indicate I should expect total memory to be roughly 3x cache_mem. mgr:info reports: Squid Object Cache: Version 3.5.26 Build Info: Service Name: squid Start Time: Mon, 25 Sep 2017 22:53:22 GMT Current Time: Mon, 25 Sep 2017 23:15:21 GMT Connection information for squid: Number of clients accessing cache: 12 Number of HTTP requests received: 568290 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 25851.6 Average ICP messages per minute since start: 0.0 Select loop called: 10686802 times, 2.009 ms avg Cache information for squid: Hits as % of all requests: 5min: 7.5%, 60min: 9.8% Hits as % of bytes sent: 5min: 12.3%, 60min: 17.3% Memory hits as % of hit requests: 5min: 36.2%, 60min: 40.0% Disk hits as % of hit requests: 5min: 27.7%, 60min: 25.5% Storage Swap size: 4481632 KB Storage Swap capacity: 4.3% used, 95.7% free Storage Mem size: 254656 KB Storage Mem capacity: 99.5% used, 0.5% free Mean Object Size: 26.30 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.00800 0.00569 Cache Misses: 0.03766 0.04489 Cache Hits: 0.00030 0.00000 Near Hits: 0.05364 0.07135 Not-Modified Replies: 0.00236 0.00168 DNS Lookups: 0.04438 0.04540 ICP Queries: 0.00000 0.00000 Resource usage for squid: UP Time: 1319.019 seconds CPU Time: 1617.476 seconds CPU Usage: 122.63% CPU Usage, 5 minute avg: 107.50% CPU Usage, 60 minute avg: 124.63% Maximum Resident Size: 60715904 KB Page faults with physical i/o: 8 Memory accounted for: Total accounted: 44968 KB memPoolAlloc calls: 158302074 memPoolFree calls: 159053228 File descriptor usage for squid: Maximum number of file descriptors: 98304 Largest file desc currently in use: 1891 Number of file desc currently in use: 3423 Files queued for open: 0 Available number of file descriptors: 94881 Reserved number of file descriptors: 600 Store Disk files open: 8 Internal Data Structures: 1702 StoreEntries 454 StoreEntries with MemObjects 4475 Hot Object Cache Items 170394 on-disk objects top when things go bad: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21450 squid 20 0 5379004 3.984g 619260 D 28.5 27.9 8:54.37 (squid-4) -f /etc/squid/squid.conf 21451 squid 20 0 5253164 3.850g 601248 D 26.6 27.0 7:46.71 (squid-3) -f /etc/squid/squid.conf 21453 squid 20 0 4750376 3.299g 504192 D 17.3 23.1 5:36.19 (squid-1) -f /etc/squid/squid.conf 21452 squid 20 0 4448280 2.994g 482136 D 16.6 21.0 5:43.58 (squid-2) -f /etc/squid/squid.conf 21449 squid 20 0 1292376 358544 346416 D 1.4 2.4 0:45.83 (squid-disk-5) -f /etc/squid/squid.conf 21448 squid 20 0 1292376 356924 344788 D 1.2 2.4 0:46.51 (squid-disk-6) -f /etc/squid/squid.conf 21447 squid 20 0 943584 11624 20 S 0.0 0.1 0:00.19 (squid-coord-7) -f /etc/squid/squid.conf I'm trying to figure out why and how to fix. One thing I've read about the cache_mem knob is: "If circumstances require, this limit will be exceeded. Specifically, if your incoming request rate requires more than 'cache_mem' of memory to hold in-transit objects, Squid will exceed this limit to satisfy the new requests. When the load decreases, blocks will be freed until the high-water mark is reached. Thereafter, blocks will be used to store hot objects." Not sure if this is the cause of my problem? Maybe something else? The FAQ says try a different malloc, so tried recompiling with --enable-dlmalloc, but that had no impact. -- Aaron Turner https://synfin.net/ Twitter: @synfinatic My father once told me that respect for the truth comes close to being the basis for all morality. "Something cannot emerge from nothing," he said. This is profound thinking if you understand how unstable "the truth" can be. -- Frank Herbert, Dune _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
