Thank You Amos. Appreciate your help! Thanks & Regards, Naresh -----Original Message----- From: Amos Jeffries [mailto:squid3@xxxxxxxxxxxxx] Sent: Friday, August 11, 2017 12:50 PM To: squid-users@xxxxxxxxxxxxxxxxxxxxx Cc: Cherukuri, Naresh Subject: Re: Crash: every 19 hours: kernel: Out of memory: Kill process (squid) On 12/08/17 01:13, Cherukuri, Naresh wrote: > Amos, > > Please find below my squid conf and access logs and memory output in MB. > Appreciate any help. > > Memory Info: > > [root@******prod ~]# free -m > > total used free shared buffers cached > > Mem: 11845 4194 7651 41 190 1418 > > -/+ buffers/cache: 2585 9260 > > Swap: 25551 408 25143 > > Squidconf: > > [root@******prod squid]# more squid.conf > > # > > # Recommended minimum configuration: > > # > > max_filedesc 4096 Ouch. Squid requires between 2 and 6 sockets (FD) per client connection and clients tend to make upwards of 8 connections per domain being contacted (dozens per web page loaded). While HTTP/1.1 improvements can reduce that average a lot 4K FD cannot serve 7K clients well. The above number should be at least 4x the expected client count to cope with load, at least 8x would be better for peak times. > > acl manager proto cache_object > > visible_hostname ******prod > > logfile_rotate 10 > > access_log /cache/access.log > > acl localnet src 172.16.0.0/16 > > acl backoffice_users src 10.136.0.0/13 > > acl h****_backoffice_users src 10.142.0.0/15 > > acl re****_users src 10.128.0.0/13 > > acl hcity_r*****_users src 10.134.0.0/15 > So you are immediately confusing "users" with "clients". They are different, and at the Squid layer of networking the difference starts to matter. For example; are you aware that automatic software and devices without any user logged in can still perform network transactions through the proxy? usually it is not a problem, but if you are thinking of only *users* utilizing the proxy you could be in for a major surprise. > acl par**** url_regex par**** > > acl SSL_ports port 443 > > acl Safe_ports port 80 # http > > #acl Safe_ports port 21 # ftp > > acl Safe_ports port 443 # https > > #acl Safe_ports port 70 # gopher > > #acl Safe_ports port 210 # wais > > #acl Safe_ports port 1025-65535 # unregistered ports > > #acl Safe_ports port 280 # http-mgmt > > #acl Safe_ports port 488 # gss-http > > #acl Safe_ports port 591 # filemaker > > #acl Safe_ports port 777 # multiling http > NP: "Safe_ports" is meant to block HTTP connections made to ports whose defined protocols are known to be dangerous for HTTP messages to go to. Those protocols are so similar they can be confused with HTTP messages and things go badly wrong. By doing the above you are making the default security rules (if you re-enable them) decide that any web API using a non-80 port is prohibited to all your clients. eg. services like http://pc-sep.pcwhq.par****.net:8014/ in your logs. The above change to the default http_access security rules behaviour is probably why you had to move your custom config to the top of the http_access list - thus bypassing everything making use of the above ports definitions. > acl CONNECT method CONNECT > > acl backoffice_allowed_sites url_regex "/etc/squid/backoffice_allowed_sites" > > acl h***_backoffice_allowed_sites url_regex > "/etc/squid/backoffice_allowed_sites" > > acl backoffice_blocked_sites url_regex "/etc/squid/backoffice_blocklist" > > acl h***_backoffice_blocked_sites url_regex > "/etc/squid/backoffice_blocklist" > > acl re****_allowed_sites url_regex "/etc/squid/re****_allowed_sites" > > acl h****_reg****_allowed_sites url_regex > "/etc/squid/h***_reg*****_allowed_sites" Are all these URLs *actually* needing regex? regex is the second slowest and most memory consuming type of ACL Squid provides. If you can reduce that to just domain names (which can have wildcard subdomains), the dstdomain ACL type would improve both memory and performance a fair bit. > > # > > http_access allow localnet reg***_allowed_sites > > http_access deny backoffice_users backoffice_blocked_sites > > http_access deny h***_backoffice_users backoffice_blocked_sites > > http_access allow backoffice_users backoffice_allowed_sites > > http_access allow h***_backoffice_users backoffice_allowed_sites > > http_access allow reg****_users reg****_allowed_sites > > http_access allow h***_reg****_users h***_reg****_allowed_sites > > no_cache deny par**** > "no_cache" has not existed for many years. Even "cache" directive is now deprecated. Most likely you want to use "store_miss deny ..." to prevent caching of objects. > http_access deny all > "deny all" ... so none of the below security protections will work. Your custom http_access lines should be down .... > #http_access allow manager localhost > > #http_access deny manager > (sure, manager report access can be moved or removed. In fact current recommendation is to place it after the CONNECT rule below. BUT, be aware that removing it *allows* all clients to access the Squid management interfaces. Probably not what you intended with the above. Only sysadmin should need that access. ) > # Deny requests to certain unsafe ports > > http_access deny !Safe_ports > > # Deny CONNECT to other than secure SSL ports > > #http_access deny CONNECT !SSL_ports > > http_access allow CONNECT SSL_ports Oh boy. You are lucky these were disabled by the above "deny all". The default config rule was specifically crafted to prevent anonymous tunneling to send arbitrary bytes to arbitrary ports with no proxy control possible. Only HTTPS (port 443) is safe enough to deliver by default. Other ports can be permitted if you need by adding to the SSL_Ports ACL. Absolutely DO NOT change that to an "allow CONNECT" like above. > > # We strongly recommend the following be uncommented to protect innocent > > # web applications running on the proxy server who think the only > > # one who can access services on "localhost" is a local user > > http_access deny to_localhost > ... Your custom http_access lines should be down here. After the basic security protections. > # Example rule allowing access from your local networks. > > # Adapt localnet in the ACL section to list your (internal) IP networks > > # from where browsing should be allowed > > #http_access allow localnet > > http_access allow localhost > > # And finally deny all other access to this proxy > > http_access deny all > > # Squid normally listens to port 3128 > > http_port 3128 ssl-bump \ > > key=/etc/squid/pc****sslcerts/pc*****prod.pkey \ > > cert=/etc/squid/pc******sslcerts/pc*****prod.crt \ > cert= before key=. The most recent Squid will complain loudly, and may not start if there is no cert to associate with the key. > generate-host-certificates=on dynamic_cert_mem_cache_size=4MB > You should also add the option sslflags=NO_DEFAULT_CA here. This lack is probably a bit part of your memory problem as OpenSSL grabs a huge amount of memory per client connection (ouch!) to store the "globally trusted CA" certificates - which are pointless on that type of connection and never used in your setup. > acl step1 at_step SslBump1 > > ssl_bump peek step1 > > #ssl_bump bump all > > ssl_bump bump backoffice_users !localnet !h***_backoffice_users > !reg****_users !h***_reg***_users !par*** > Outwardly that may look reasonable, but it can be simplified a fair bit. eg. A message sent by client whose IP is in the range 10.136.0.0/13 cannot simultaneously be sent using an IP in the range 172.16.0.0/16 or 10.128.0.0/13. You only need to exclude (with '!') when the excluded ACL can match things that would otherwise be caught by the other ACLs in the list. For example the h****_backoffice_users is a subset of backoffice_users, and the par*** ACL matches a wholly different criteria than src-IP. > #sslproxy_capath /etc/ssl/certs > > sslproxy_cert_error allow all > > always_direct allow all > > sslproxy_flags DONT_VERIFY_PEER > Just no. Remove the above three lines. Then fix any TLS/SSL issues you find in a proper way - which will be different per-problem, and worthwhile fixing. Since 3.2 Squid has been able to mimic errors in the fake certs it generates so silencing them is more harmful than good. If there are errors that actually cannot be fixed AND happen to be safe for a proxy to hide from the end-user [be very sure of that first], then set only that error into an "sslproxy_cert_error allow ..." rule. Leave all other errors going on through to the client software, often they are better able to cope with it cleanly than the proxy can. and for the log; All these 403 look like the relevant URLs are not in your *_sites ACL definitions. > > 1502424001.504 0 10.138.142.6 TCP_DENIED/403 4175 GET > http://update.scansoft.com/GetCertificate.asp? - HIER_NONE/- text/html > > 1502424001.533 329 10.140.230.6 TAG_NONE/200 0 CONNECT > watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 - > > 1502424001.543 0 10.141.80.6 TCP_DENIED/403 4167 GET > http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html > > 1502424001.546 331 10.140.230.6 TAG_NONE/200 0 CONNECT > watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 - > > 1502424001.551 29923 10.130.27.24 TCP_MISS_ABORTED/000 0 GET > http://pc-sep.pcwhq.par****.net:8014/secars/secars.dll? - > HIER_DIRECT/10.1.2.35 - > That is a little more worrying, your web server at 10.1.2.35 appears not to be producing a response for at least 30sec. > > Cachelog errors I am seeing daily: > > Error negotiating SSL connection on FD 26: error:140A1175:SSL > routines:SSL_BYTES_TO_CIPHER_LIST:inappropriate fallback (1/-1) > > Error negotiating SSL connection on FD 1175: error:14094416:SSL > routines:SSL3_READ_BYTES:sslv3 alert certificate unknown (1/0) > First thing to do is ensure that your OpenSSL library is up to date. That should resolve most cipher and TLS protocol related issues. Second thing is to ensure that the qa-certificates package (or whatever your OS calls it) is kept up to date. It changes every few weeks. Squid-3.x cannot download CA certs on demand so this is particularly important, and you may need to configure the sslproxy_foreign_intermediate_certs directive with additional and intermediate CA - only as needed and after checking them though. > 2017/08/02 09:01:02 kid1| Error negotiating SSL on FD 989: > error:00000000:lib(0):func(0):reason(0) (5/-1/104) ##Very rare i found > few not frequently > These opaque error codes tends to be non-TLS being pushed into port 443. It varies based on what software your clients are running. > 2017/08/02 09:01:43 kid1| Queue overload, rejecting # too many times > > 2017/08/02 09:01:45 kid1| Error negotiating SSL connection on FD 1749: > (104) Connection reset by peer ## too many times > > 2017/08/02 10:12:58 kid1| WARNING: Closing client connection due to > lifetime timeout ## only one > > 2017/08/07 22:37:56 kid1| comm_open: socket failure: (24) Too many open > files Either lack of FD or your OS has set a per-process open FD limit far too low for what Squid requires. Either HDD disk files or network socket limits result in the above message. > > 2017/08/07 22:39:37 kid1| WARNING: Error Pages Missing Language: en > > 2017/08/07 22:39:37 kid1| '/usr/share/squid/errors/en-us/ERR_DNS_FAIL': > (24) Too many open files > > 2017/08/07 22:39:37 kid1| WARNING: Error Pages Missing Language: en-us > Your system appears to be missing the Squid langpack for error page localization. Or if you have one it needs updating to match the error pages your Squid version uses. <http://www.squid-cache.org/Versions/langpack/> > 2017/08/07 22:01:42 kid1| WARNING: All 32/32 ssl_crtd processes are busy. > > 2017/08/07 22:01:42 kid1| WARNING: 32 pending requests queued > > 2017/08/07 22:01:42 kid1| WARNING: Consider increasing the number of > ssl_crtd processes in your config file. As Squid said, you can try increasing the crtd processes being run. Though the above says 32 running and your squid.conf sslcrtd_children directive says maximum of 8 to be run. Alternatively, since you are bumping you may want to try Squid-4. The SSL-Bump code and behaviour is quite a bit better in the latest version even though it is in beta still. > > 2017/08/11 00:58:56 kid1| WARNING: Closing client connection due to > lifetime timeout > > 2017/08/09 12:55:45 kid1| WARNING! Your cache is running out of > filedescriptors > Yes, well. I covered that already, the above just confirms it. Amos _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
