On 27/07/2013 11:10 p.m., Ahmad wrote:
hi ,
i have 2 squid machine and both of them has the same load of users and BW
but of them has 32 G ram and the other has 48 G ram
here is a comparision between them :
1-server of 32 G ram
squidclient -p 2222 mgr:info | grep 'file des'
Maximum number of file descriptors: 65536
Largest file desc currently in use: 52462
Number of file desc currently in use: 39277
Available number of file descriptors: 26258
Reserved number of file descriptors: 100
2-server of 48 G ram
squidclient -p 2222 mgr:info | grep 'file des'
Maximum number of file descriptors: 65536
Largest file desc currently in use: 52497
Number of file desc currently in use: 50191
Available number of file descriptors: 15345
Reserved number of file descriptors: 100
note that server 1 has larger number of server 2 relative to Number of
file desc currently in use ,
also , i note that server 1 is making better than server 2 in bw saving .
my question is , does this relative to ram ???
Yes and no.
It is relative to RAM, but not close enough to be an indicator or Squid.
FD are used for all of listening sockets, files access, client/server
sockets, UDP sockets, IDS sockets. Each of these has very different
memory usage requirements and some have quite a bit of internal variance
relative to the transaction type underway as well.
another issue , how do i know that i need to increase my rams ???
If you are close to filling it and reducing cache_mem is not an option.
also i want my squid.conf to be checked for any thing could be enhance my
server .
Which particular version of Squid?
here is squid.conf file and it is identical to both servers .
=========================================================================
here is squid.conf file :
include /etc/squid3/vc_squid_3.conf
What is in that file may affect performance. Since you have not provided
it I am going to have to assume it does not.
####################################
############################################################
acl localnetz src 10.11.0.0/16
http_access deny localnetz
#################################################################
###############Redirection of private ips to webpage#############
#################################################################
acl localnetx src 10.12.0.0/16
acl localnety dst 192.168.70.2/32
http_access allow localnetx localnety
http_access deny localnetx
############filtering without squidguard###########
acl blockkeywords dstdomain "/etc/squid3/newsquid-porn.acl"
http_access deny blockkeywords
##########################################################
acl pornreg dstdom_regex "/etc/squid3/squid-regex.acl"
http_access deny pornreg
############################################################
####################################################
####################################################
#acl NO-CACHE-SITES dstdomain "/etc/squid3/not-to-cache-sites.txt"
#no_cache deny NO-CACHE-SITES
################################################################
#
# Recommended minimum configuration:
#
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
#################################################################################
# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged)
machines
acl mysubnet src 185.6.16.0/22 176.58.64.0/20 188.161.104.0/21
213.244.82.129/32
http_access allow mysubnet
###################################################################################
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
######################################################
#
# Recommended minimum Access Permission configuration:
#
Sigh. This block from here ...
# Only allow cachemgr access from localhost
http_access allow manager localhost
http_access deny manager
################################################
# Deny requests to certain unsafe ports
http_access deny !Safe_ports
###################################################################
# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports
#####################################################################
# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost
... to here is the default security settings for Squid. Protecting you
against quite a large number of malicious traffic and designed
explicitly to do so at the highest possible speed.
By placing your own http_access rules above it you are making Squid run
the processing for all those local rules *before* it detects things such
as an attempt to send Spam email through your proxy. Any one or more of
your local rules may even be *allowing* such an attack to take place
accidentally.
==> such as any one of the machines in "mysubnet" have free access to
do so if they become infected with a spambot.
Ideally for any forward-proxy or interception-proxy like yours local
rules should all be placed underneath the disclaimer here:
##################################################################
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
###################################################################
# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost
##############################################################
# And finally deny all other access to this proxy
http_access deny all
###########################################################
# Squid normally listens to port 3128
http_port 1xxxxx
http_port xxxxxx
http_port xxxxx tproxy
###########################################################
# We recommend you to use at least the following line.
hierarchy_stoplist cgi-bin ?
Recommendation has changed. You can remove that line. The setting
appears to be useless in your configuration anyway.
###############################################################
# Uncomment and adjust the following to add a disk cache directory.
cache_dir aufs /cache1 90000 32 256
cache_dir aufs /cache2 90000 32 256
cache_dir aufs /cache3 90000 32 256
###########################################################
# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid
###############################################################
# Add any of your own refresh_pattern entries above these.
########### Refresh_pattern Config:
#####################################################################################################################
#for the music:
refresh_pattern -i
\.(mp2|mp3|mid|midi|mp[234]|wav|ram|ra|rm|au|3gp|m4r|m4a)(\?.*|$) 5259487
999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache
ignore-private
#for the movies:
refresh_pattern -i
\.(mpg|mpeg|mp4|m4v|mov|avi|asf|wmv|wma|dat|flv|swf)(\?.*|$) 5259487 999%
5259487 override-expire ignore-reload reload-into-ims ignore-no-cache
ignore-private
#for pictures:
refresh_pattern -i
\.(jpeg|jpg|jpe|jp2|gif|tiff?|pcx|png|bmp|pic|ico)(\?.*|$) 5259487 999%
5259487 override-expire ignore-reload reload-into-ims ignore-no-cache
ignore-private
#for MS docs:
refresh_pattern -i
\.(chm|dll|doc|docx|xls|xlsx|ppt|pptx|pps|ppsx|mdb|mdbx)(\?.*|$) 5259487
999% 5259487 override-expire ignore-reload reload-into-ims ignore-no-cache
ignore-private
I *really* hope you know what you are doing here. Ignoring the "private"
cache control on documents and email archives will cause them to
potentially be delivered to multiple people. Some of who may not be
wanting or supposed to receive them. With "dll" you are potentially even
sending one persons login credentials to everyone.
#for various other docs:
refresh_pattern -i \.(txt|conf|cfm|psd|wmf|emf|vsd|pdf|rtf|odt)(\?.*|$)
5259487 999% 5259487 override-expire ignore-reload reload-into-ims
ignore-no-cache ignore-private
#for the well-known compressed/excutable files:
refresh_pattern -i
\.(class|jar|exe|gz|bz|bz2|tar|tgz|zip|gzip|arj|ace|bin|cab|msi|rar)(\?.*|$)
5259487 999% 5259487 override-expire ignore-reload reload-into-ims
ignore-no-cache ignore-private
#for various client-side Web docs:
refresh_pattern -i \.(htm|html|mhtml|css|js)(\?.*|$) 1440 90% 86400
override-expire ignore-reload reload-into-ims
FMI: Which softwares or sites traffic is causing you to add
"ignore-private" to most of those patterns? That setting in particular
is a serious breach of the HTTP protocol. The software must be badly
broken to explicitly send "Cache-Control:private" on non-private
information, and you will definitely be causing major problems to
security systems like Captchas.
#####################################################################################################################
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
###############################################################
cache_mem 1000 MB
########### WCCP2 Config#############
wccp2_router xxxxxx
wccp_version 2
wccp2_forwarding_method 2
wccp2_return_method 2
wccp2_assignment_method 2
wccp2_service dynamic 60
wccp2_service_info 60 protocol=tcp flags=src_ip_hash priority=250 ports=80
wccp2_service dynamic 70
wccp2_service_info 70 protocol=tcp flags=dst_ip_hash,ports_source
priority=250 ports=80
###################################################################
dns_nameservers xxxxx xxxxx 8.8.8.8
cache_effective_user proxy
cache_effective_group proxy
visible_hostname squid
visible_hostname is supposed to be a FQDN and unique per-machine. If you
have configured each machines hostname settings correctly and registered
the name in DNS you do not need the above setting at all. Squid will
auto-detect the one appropriate to the machine it is running on.
############################################################
########### Performance Related Config:
relaxed_header_parser on
Well, it is related to performance. But the OFF setting is what makes
reject or ignore traffic with syntax errors. The rejection happens fast,
so avoiding time wasted processing those transactions. But it may be
super-annoying for any of your clients using slightly broken software
which "works" through other proxies.
In general there are far too many broken software out there and this
needs to be left at its default setting (ON or remove from the config
completely), but if you are able to turn it OFF that would be wonderful
news.
vary_ignore_expire on
If you have a Squid-3.1 or later that can be removed. Those versions do
not send HTTP/1.0 to servers.
##########################################
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
maximum_object_size_in_memory 64 KB
You have 1000 MB of cache_mem available. It may be possible to tune this
higher for faster HIT on larger objects. But you need to tune that
yourself with testing of how it affects the traffic and HIT rates.
###########################################
ipcache_size 2048
ipcache_low 96
ipcache_high 99
memory_pools off
YMMV on this one. The pools prevent Squid having to cycle through the
system malloc systems on every byte of memory allocated or deallocated,
and there are some things with very small sizes being
allocated/deallocated constantly in Squid. There is some overhead inside
Squid CPU usage instead.
If you must disable this you will be worth building Squid against a
highly optimized allocator library. The system default one is usually
not very great for high performance. Tcalloc was the best last time I
had to go through and rate them (a few years ago now so things may have
changed).
pipeline_prefetch on
############################################
httpd_suppress_version_string on
server_persistent_connections on
client_persistent_connections on
pconn_timeout 2 minutes
persistent_request_timeout 1 minute
###########################################
forwarded_for on
max_filedescriptors 65536
max_open_disk_fds 65536
YMMV, but I personally do not think it is a good idea to have the total
and disk I/O FD limits the same. There are multiple network uses which
will need to be using FD so you risk the disks grabbing them _all_ if a
large disk garbage collection is running when you reached the max. That
would block opening new client and server connections completely until
the disk I/O was completed.
relaxed_header_parser on
You already configured that to ON.
reload_into_ims on
client_lifetime 15 minutes
read_timeout 5 minutes
request_timeout 1 minutes
ie_refresh on
ignore_expect_100 on
This will be a drag on anyone using HTTP Expect functionality. The best
cure there is to upgrade to Squid-3.2 and drop this setting entirely.
Second-best is to leave it OFF, which is the default in Squid-3.1. Yes
clients will get 417 responses out of Squid but they are expected to
deal with those properly and being nice by ignoring Expect headers will
only result in traffic hanging while all involved FD are locked up
unusable for undefined length of time when the client hits a timeout
(which it may never do).
vary_ignore_expire on
###############################
################################
httpd_suppress_version_string on
server_persistent_connections on
client_persistent_connections on
pconn_timeout 2 minutes
persistent_request_timeout 1 minute
#shutdown_lifetime 20 seconds
#############################
cache_swap_low 96
cache_swap_high 99
###############################
HTH
Amos
