On 12/08/17 01:13, Cherukuri, Naresh wrote:
Amos,
Please find below my squid conf and access logs and memory output in MB.
Appreciate any help.
Memory Info:
[root@******prod ~]# free -m
total used free shared buffers cached
Mem: 11845 4194 7651 41 190 1418
-/+ buffers/cache: 2585 9260
Swap: 25551 408 25143
Squidconf:
[root@******prod squid]# more squid.conf
#
# Recommended minimum configuration:
#
max_filedesc 4096
Ouch. Squid requires between 2 and 6 sockets (FD) per client connection
and clients tend to make upwards of 8 connections per domain being
contacted (dozens per web page loaded). While HTTP/1.1 improvements can
reduce that average a lot 4K FD cannot serve 7K clients well.
The above number should be at least 4x the expected client count to cope
with load, at least 8x would be better for peak times.
acl manager proto cache_object
visible_hostname ******prod
logfile_rotate 10
access_log /cache/access.log
acl localnet src 172.16.0.0/16
acl backoffice_users src 10.136.0.0/13
acl h****_backoffice_users src 10.142.0.0/15
acl re****_users src 10.128.0.0/13
acl hcity_r*****_users src 10.134.0.0/15
So you are immediately confusing "users" with "clients". They are
different, and at the Squid layer of networking the difference starts to
matter.
For example; are you aware that automatic software and devices without
any user logged in can still perform network transactions through the
proxy? usually it is not a problem, but if you are thinking of only
*users* utilizing the proxy you could be in for a major surprise.
acl par**** url_regex par****
acl SSL_ports port 443
acl Safe_ports port 80 # http
#acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
#acl Safe_ports port 70 # gopher
#acl Safe_ports port 210 # wais
#acl Safe_ports port 1025-65535 # unregistered ports
> #acl Safe_ports port 280 # http-mgmt
#acl Safe_ports port 488 # gss-http
#acl Safe_ports port 591 # filemaker
#acl Safe_ports port 777 # multiling http
NP: "Safe_ports" is meant to block HTTP connections made to ports whose
defined protocols are known to be dangerous for HTTP messages to go to.
Those protocols are so similar they can be confused with HTTP messages
and things go badly wrong.
By doing the above you are making the default security rules (if you
re-enable them) decide that any web API using a non-80 port is
prohibited to all your clients.
eg. services like http://pc-sep.pcwhq.par****.net:8014/ in your logs.
The above change to the default http_access security rules behaviour is
probably why you had to move your custom config to the top of the
http_access list - thus bypassing everything making use of the above
ports definitions.
acl CONNECT method CONNECT
acl backoffice_allowed_sites url_regex "/etc/squid/backoffice_allowed_sites"
acl h***_backoffice_allowed_sites url_regex
"/etc/squid/backoffice_allowed_sites"
acl backoffice_blocked_sites url_regex "/etc/squid/backoffice_blocklist"
acl h***_backoffice_blocked_sites url_regex
"/etc/squid/backoffice_blocklist"
acl re****_allowed_sites url_regex "/etc/squid/re****_allowed_sites"
acl h****_reg****_allowed_sites url_regex
"/etc/squid/h***_reg*****_allowed_sites"
Are all these URLs *actually* needing regex? regex is the second slowest
and most memory consuming type of ACL Squid provides.
If you can reduce that to just domain names (which can have wildcard
subdomains), the dstdomain ACL type would improve both memory and
performance a fair bit.
#
http_access allow localnet reg***_allowed_sites
http_access deny backoffice_users backoffice_blocked_sites
http_access deny h***_backoffice_users backoffice_blocked_sites
http_access allow backoffice_users backoffice_allowed_sites
http_access allow h***_backoffice_users backoffice_allowed_sites
http_access allow reg****_users reg****_allowed_sites
http_access allow h***_reg****_users h***_reg****_allowed_sites
no_cache deny par****
"no_cache" has not existed for many years. Even "cache" directive is now
deprecated.
Most likely you want to use "store_miss deny ..." to prevent caching of
objects.
http_access deny all
"deny all" ... so none of the below security protections will work.
Your custom http_access lines should be down ....
#http_access allow manager localhost
#http_access deny manager
(sure, manager report access can be moved or removed. In fact current
recommendation is to place it after the CONNECT rule below.
BUT, be aware that removing it *allows* all clients to access the Squid
management interfaces. Probably not what you intended with the above.
Only sysadmin should need that access.
)
# Deny requests to certain unsafe ports
http_access deny !Safe_ports
# Deny CONNECT to other than secure SSL ports
#http_access deny CONNECT !SSL_ports
http_access allow CONNECT SSL_ports
Oh boy. You are lucky these were disabled by the above "deny all".
The default config rule was specifically crafted to prevent anonymous
tunneling to send arbitrary bytes to arbitrary ports with no proxy
control possible. Only HTTPS (port 443) is safe enough to deliver by
default.
Other ports can be permitted if you need by adding to the SSL_Ports ACL.
Absolutely DO NOT change that to an "allow CONNECT" like above.
# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
http_access deny to_localhost
... Your custom http_access lines should be down here. After the basic
security protections.
# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
#http_access allow localnet
http_access allow localhost
# And finally deny all other access to this proxy
http_access deny all
# Squid normally listens to port 3128
http_port 3128 ssl-bump \
key=/etc/squid/pc****sslcerts/pc*****prod.pkey \
cert=/etc/squid/pc******sslcerts/pc*****prod.crt \
cert= before key=. The most recent Squid will complain loudly, and may
not start if there is no cert to associate with the key.
generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
You should also add the option sslflags=NO_DEFAULT_CA here.
This lack is probably a bit part of your memory problem as OpenSSL grabs
a huge amount of memory per client connection (ouch!) to store the
"globally trusted CA" certificates - which are pointless on that type of
connection and never used in your setup.
acl step1 at_step SslBump1
ssl_bump peek step1
#ssl_bump bump all
ssl_bump bump backoffice_users !localnet !h***_backoffice_users
!reg****_users !h***_reg***_users !par***
Outwardly that may look reasonable, but it can be simplified a fair bit.
eg. A message sent by client whose IP is in the range 10.136.0.0/13
cannot simultaneously be sent using an IP in the range 172.16.0.0/16 or
10.128.0.0/13.
You only need to exclude (with '!') when the excluded ACL can match
things that would otherwise be caught by the other ACLs in the list. For
example the h****_backoffice_users is a subset of backoffice_users, and
the par*** ACL matches a wholly different criteria than src-IP.
#sslproxy_capath /etc/ssl/certs
sslproxy_cert_error allow all
always_direct allow all
sslproxy_flags DONT_VERIFY_PEER
Just no. Remove the above three lines. Then fix any TLS/SSL issues you
find in a proper way - which will be different per-problem, and
worthwhile fixing.
Since 3.2 Squid has been able to mimic errors in the fake certs it
generates so silencing them is more harmful than good.
If there are errors that actually cannot be fixed AND happen to be safe
for a proxy to hide from the end-user [be very sure of that first], then
set only that error into an "sslproxy_cert_error allow ..." rule. Leave
all other errors going on through to the client software, often they are
better able to cope with it cleanly than the proxy can.
and for the log;
All these 403 look like the relevant URLs are not in your *_sites ACL
definitions.
1502424001.504 0 10.138.142.6 TCP_DENIED/403 4175 GET
http://update.scansoft.com/GetCertificate.asp? - HIER_NONE/- text/html
1502424001.533 329 10.140.230.6 TAG_NONE/200 0 CONNECT
watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 -
1502424001.543 0 10.141.80.6 TCP_DENIED/403 4167 GET
http://update.scansoft.com/Version.asp? - HIER_NONE/- text/html
1502424001.546 331 10.140.230.6 TAG_NONE/200 0 CONNECT
watson.telemetry.microsoft.com:443 - HIER_DIRECT/65.55.252.202 -
1502424001.551 29923 10.130.27.24 TCP_MISS_ABORTED/000 0 GET
http://pc-sep.pcwhq.par****.net:8014/secars/secars.dll? -
HIER_DIRECT/10.1.2.35 -
That is a little more worrying, your web server at 10.1.2.35 appears not
to be producing a response for at least 30sec.
Cachelog errors I am seeing daily:
Error negotiating SSL connection on FD 26: error:140A1175:SSL
routines:SSL_BYTES_TO_CIPHER_LIST:inappropriate fallback (1/-1)
Error negotiating SSL connection on FD 1175: error:14094416:SSL
routines:SSL3_READ_BYTES:sslv3 alert certificate unknown (1/0)
First thing to do is ensure that your OpenSSL library is up to date.
That should resolve most cipher and TLS protocol related issues.
Second thing is to ensure that the qa-certificates package (or whatever
your OS calls it) is kept up to date. It changes every few weeks.
Squid-3.x cannot download CA certs on demand so this is particularly
important, and you may need to configure the
sslproxy_foreign_intermediate_certs directive with additional and
intermediate CA - only as needed and after checking them though.
2017/08/02 09:01:02 kid1| Error negotiating SSL on FD 989:
error:00000000:lib(0):func(0):reason(0) (5/-1/104) ##Very rare i found
few not frequently
These opaque error codes tends to be non-TLS being pushed into port 443.
It varies based on what software your clients are running.
2017/08/02 09:01:43 kid1| Queue overload, rejecting # too many times
2017/08/02 09:01:45 kid1| Error negotiating SSL connection on FD 1749:
(104) Connection reset by peer ## too many times
2017/08/02 10:12:58 kid1| WARNING: Closing client connection due to
lifetime timeout ## only one
2017/08/07 22:37:56 kid1| comm_open: socket failure: (24) Too many open
files
Either lack of FD or your OS has set a per-process open FD limit far too
low for what Squid requires. Either HDD disk files or network socket
limits result in the above message.
2017/08/07 22:39:37 kid1| WARNING: Error Pages Missing Language: en
2017/08/07 22:39:37 kid1| '/usr/share/squid/errors/en-us/ERR_DNS_FAIL':
(24) Too many open files
2017/08/07 22:39:37 kid1| WARNING: Error Pages Missing Language: en-us
Your system appears to be missing the Squid langpack for error page
localization. Or if you have one it needs updating to match the error
pages your Squid version uses.
<http://www.squid-cache.org/Versions/langpack/>
2017/08/07 22:01:42 kid1| WARNING: All 32/32 ssl_crtd processes are busy.
2017/08/07 22:01:42 kid1| WARNING: 32 pending requests queued
2017/08/07 22:01:42 kid1| WARNING: Consider increasing the number of
ssl_crtd processes in your config file.
As Squid said, you can try increasing the crtd processes being run.
Though the above says 32 running and your squid.conf sslcrtd_children
directive says maximum of 8 to be run.
Alternatively, since you are bumping you may want to try Squid-4. The
SSL-Bump code and behaviour is quite a bit better in the latest version
even though it is in beta still.
2017/08/11 00:58:56 kid1| WARNING: Closing client connection due to
lifetime timeout
2017/08/09 12:55:45 kid1| WARNING! Your cache is running out of
filedescriptors
Yes, well. I covered that already, the above just confirms it.
Amos
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
