Re: Re: is there any thing wrong from cache manager logs ?!!

[Eliezer Croitoru <eliezer@xxxxxxxxxxxx>]

Hey there,

It's a bit weird..
Lets try it from another angle one at a time..
First user the basic 256MB MEMcache  which is less the 512MB.
Remove any cache_dir that do exits..
This way we have pushed squid into the CPU and mem land.
Having low memory is nothing to be afraid of..
The main issue is why would you get into a position that squid that can 
pump lots of users traffic is not responding to you very well.
If you have an unresponsive server to even SSH the next thing is basic 
PING tests.
ARPING
PING
TCPING
SSH_TCP_PING (telnet\nc port 22)
HTTPING

note that SWAP is not a bad thing..
swap means that the OS recognizes a need to take the load off the RAM 
into disk and back.
This specific function allows many "unresponsive" servers to be 
"responsive" in a case of need by not other then the help of "HDD".

Ok so we have one instance that should take 58 requests per second no 
problem which my tiny INTEL ATOM server is taking no problem almost at 
least couple times a day.
maximum FD will be about 1k(1024) in use which should no be too much of 
a problem for any tiny proxy.

And now again, what OS are you using? why do you need 500k FD allowed 
for a system that should use about 16k FD just for a test? 65k FS should 
be enough for most basic small ISP setup.

So only one instance of squid no workers at all no ROCK storage adding 
users should show you how the service works..
(I can calm you down that I took up and down a server that serves more 
then 1k users live on it.)

Indeed named can load on the server but since it's serving only squid I 
assume you do have another local DNS server in place so point to it.
Using 8.8.8.8 public DNS will not solve your problems but rather make 
them worse unless you do have 1-30ms response time from it.

You can also do DNS cache warm-up..

Again what is the degradation you are talking about?
Try to download a static file from some couple random mirrors of let say 
some linux distro or a mirror of another file you have in mind like 
microsoft..

I have looked at the graph and I do not understand what is the problem 
while there is a degradation?
SSH can be one hell of thing while doing couple reverse DNS lookups for 
some IP that doesn't have one or once the local dns that is defined in 
/etc/resolv.conf is not responding.
Also try to use the same dns server for both OS and squid.

Just a tiny test I want to add for all the above:
start any local http server that you like and prefer like apache, nginx, 
lighthttpd, GoAhead-Webserver, micro_httpd or any other and put it on 
the "lan" or "network segment" with the squid server and try to download 
files of 512bytes 1KB 1MB 9MB 15MB 100MB and up.
also try to access the cache-mgr page using:
http://MY_CACHE_IP:zzzzzz/squid-internal-mgr/mem

at the same time there is a problem and after\before.

The graph that you supply doesn't really explains the problem in any of 
the network measurements at least I can understand.

At the same time try to use the squid proxy as a simple forward proxy 
that is defined in the browser settings and make sure what are the results.

Firefox have a tiny nice SHIFT+F5 interface that shows network access 
times for every page that is loaded.
When you do have the issue you can try to see what network issues are 
happening and what the simple client is having trouble with?
For example the problem can be in another level then you actually think off.

Try to take a small look at:
http://wiki.squid-cache.org/ConfigExamples/UbuntuTproxy4Wccp2

Which uses the tunnel method rather then the mac address rewritting 
method 2 wccp forwarding.
(I am merely saying that I know it works very well for me)
What device does the WCCP interception??

There are bugs out-there in squid but I would like to see the bug..
I am not sure but what are the "free -m" status on this server while 
squid is not running at all?

I am until now try to think of a test that will show and explain the 
problem in hands.
There is an issue that I remember about FD limit that is being forced by 
the bash\startup script on squid master instance but I am not sure if 
this is the source to the issue or it's another thing.
please add to squid init.d\startup script "ulimit -a" and the output of 
that..

Are there any clues in the cache.log?

Best Regards,
Eliezer

On 11/07/2013 09:58 PM, Dr.x wrote:
> hi ,
> i have made a test for squid withing 30 minutes and ive logged alot of
> things during working :
> 1st of all , the current server is 8 g ram and quad core cpu " 8 cores "
> here is squid performance and degredation with time :
> <http://squid-web-proxy-cache.1019090.n4.nabble.com/file/n4663169/237842408.png>
>
>
> as we see from graph , squid is fine as 1st starting and after some time
> degradation start untill it reach down !!!!
> also from grahp we note the cpu loading , not congested to specific cores ,
> i see its fine .
>
> note to hint that during the squid running the cpus cores was fine and
> normal except free ram is getting decreasing !
>
> @Alex , during to ur question , the cpu dissipation was jumping to cores
> during top and pressing 1 then c
> i monitored it and cpu dissipation is normal !
> also ,
> @Alex, ive disabled "named" service of my server  and let squid look to
> 8.8.8.8 for dns resolving , i mean i dont want it to be makking problems .
>
> @Alex , about the "vary object loop" which i posted before , i found that
> when i define cache dir , i dont see this issue in logs , thats why in my
> squid.conf below i configured only 1 G  for rock type  and 1 drive.
>
> @Alex , ive reduced cache mem , and cache dir , and put mem pools off ,so
> that we dont face problems in memoery ,but not sure if what i did is fine or
> not.
>
> here is squid.conf file :
> #################################################################
> # Example rule allowing access from your local networks.
> # Adapt to list your (internal) IP networks from where browsing
> # should be allowed
> acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
> acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
> acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
> acl localnet src fc00::/7       # RFC 4193 local private network range
> acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged)
> machines
> acl mysubnet src zzzzzzzzzzzzzz/zzzzzzzzzzzzzzz
> acl SSL_ports port 443
> acl Safe_ports port 80          # http
> acl Safe_ports port 21          # ftp
> acl Safe_ports port 443         # https
> acl Safe_ports port 70          # gopher
> acl Safe_ports port 210         # wais
> acl Safe_ports port 1025-65535  # unregistered ports
> acl Safe_ports port 280         # http-mgmt
> acl Safe_ports port 488         # gss-http
> acl Safe_ports port 591         # filemaker
> acl Safe_ports port 777         # multiling http
> acl CONNECT method CONNECT
> ######################################################
> # Recommended minimum Access Permission configuration:
> # Deny requests to certain unsafe ports
> http_access deny !Safe_ports
> http_access allow localnet
> # Deny CONNECT to other than secure SSL ports
> http_access deny CONNECT !SSL_ports
> #######################################################
> ###########SMP oPTIONS#####################
> dns_v4_first on
> # 3 workers, using worker #1 as the frontend is important
> workers 3
> #########################################
> visible_hostname squid.drx
> ####################################################
> ####Filterring##############
> acl blockkeywords dstdomain "/etc/squid/koko.acl"
> http_access deny blockkeywords
> #################################################
> cache_log /var/log/squid/cache.log
> access_log stdio:/var/log/squid/access.log
> ###############################################
> http_port xxxx:zzzzzz
> http_port cccccccc:333333333333 tproxy
> ###############################################################
> http_access allow mysubnet
> ############################################################################################
> cache_mgr xxxxxxx
> cachemgr_passwd xxxxxxxxxxxx all
> # add user authentication and similar options here
> http_access allow manager localhost
> http_access allow manager mysubnet
> http_access allow mysubnet manager
> http_access deny manager
> #######################################################################
> cache_mem 512 MB
> ################################################
> maximum_object_size_in_memory 64 KB
> maximum_object_size 10 MB
> #######################################################################
> cache_swap_low 90
> cache_swap_high 95
> ###########################################################
> server_persistent_connections off
> client_persistent_connections off
> quick_abort_min 0 KB
> quick_abort_max 0 KB
> quick_abort_pct 95
> fqdncache_size 65535
> ipcache_size 65535
> ipcache_low 98
> ipcache_high 99
> #####################################################
> ########### WCCP2 Config#############
> wccp2_router x.x.x.x.x.
> wccp2_rebuild_wait off
> wccp_version 2
> wccp2_forwarding_method 2
> wccp2_return_method 2
> wccp2_assignment_method 2
> wccp2_service dynamic 92
> wccp2_service_info 92 protocol=tcp flags=src_ip_hash priority=250 ports=80
> wccp2_service dynamic 93
> wccp2_service_info 93 protocol=tcp flags=dst_ip_hash,ports_source
> priority=250 ports=80
> ######################################################
> cache_effective_user squid
> cache_effective_group squid
> #######################################################
> memory_replacement_policy heap GDSF
> cache_replacement_policy heap LFUDA
> #####################################################
> dns_nameservers  8.8.8.8
> #######################################################
> http_access deny all
> ##############################
> refresh_pattern ^ftp:           1440    20%     10080
> refresh_pattern ^gopher:        1440    0%      1440
> refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
> refresh_pattern .               0       20%     4320
> ####################################################
> cache_dir rock /CACHE1 1000 max-size=32768 swap-timeout=350
> max-swap-rate=350
> #####################################
> memory_pools off
> =================================================================
> i have monitored squid along time :
> here is monitoring logs :
>
>
> *at 10:25*
> *no users pumped to squid*
>
> top command and cache manage results :
>
> top - 14:27:50 up 1 day,  5:43,  3 users,  load average: 0.02, 0.04, 0.08
> Tasks: 152 total,   1 running, 151 sleeping,   0 stopped,   0 zombie
> Cpu0  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Mem:   8182336k total,  7006844k used,  1175492k free,   173512k buffers
> Swap:        0k total,        0k used,        0k free,  6527720k cached
> ============================================================================
> Squid Object Cache: Version 3.3.9
>
> Start Time:	Thu, 07 Nov 2013 18:22:47 GMT
> Current Time:	Thu, 07 Nov 2013 18:29:14 GMT
>
> Connection information for squid:
> 	Number of clients accessing cache:	8
> 	Number of HTTP requests received:	209
> 	Number of ICP messages received:	0
> 	Number of ICP messages sent:	0
> 	Number of queued ICP replies:	0
> 	Number of HTCP messages received:	0
> 	Number of HTCP messages sent:	0
> 	Request failure ratio:	 0.00
> 	Average HTTP requests per minute since start:	32.4
> 	Average ICP messages per minute since start:	0.0
> 	Select loop called: 685029 times, 2.755 ms avg
> Cache information for squid:
> 	Hits as % of all requests:	5min: 0.0%, 60min: 0.0%
> 	Hits as % of bytes sent:	5min: 2.2%, 60min: 2.2%
> 	Memory hits as % of hit requests:	5min: 0.0%, 60min: 0.0%
> 	Disk hits as % of hit requests:	5min: 0.0%, 60min: 0.0%
> 	Storage Swap size:	25584 KB
> 	Storage Swap capacity:	 2.5% used, 97.5% free
> 	Storage Mem size:	1568 KB
> 	Storage Mem capacity:	 0.3% used, 99.7% free
> 	Mean Object Size:	32.02 KB
> 	Requests given to unlinkd:	0
> Median Service Times (seconds)  5 min    60 min:
> 	HTTP Requests (All):   2.00247  2.00247
> 	Cache Misses:          2.00247  2.00247
> 	Cache Hits:            0.00000  0.00000
> 	Near Hits:             0.00000  0.00000
> 	Not-Modified Replies:  0.00000  0.00000
> 	DNS Lookups:           4.65962  4.65962
> 	ICP Queries:           0.00000  0.00000
> Resource usage for squid:
> 	UP Time:	387.118 seconds
> 	CPU Time:	35.047 seconds
> 	CPU Usage:	9.05%
> 	CPU Usage, 5 minute avg:	0.23%
> 	CPU Usage, 60 minute avg:	9.72%
> 	Process Data Segment Size via sbrk(): 250724 KB
> 	Maximum Resident Size: 1459216 KB
> 	Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
> 	Total space in arena:  251252 KB
> 	Ordinary blocks:       250303 KB    120 blks
> 	Small blocks:               0 KB      0 blks
> 	Holding blocks:        291440 KB     31 blks
> 	Free Small blocks:          0 KB
> 	Free Ordinary blocks:     949 KB
> 	Total in use:             949 KB 0%
> 	Total free:               949 KB 0%
> 	Total size:            542692 KB
> Memory accounted for:
> 	Total accounted:         1463 KB   0%
> 	memPool accounted:       1463 KB   0%
> 	memPool unaccounted:   541229 KB 100%
> 	memPoolAlloc calls:        52
> 	memPoolFree calls:      98503
> File descriptor usage for squid:
> 	Maximum number of file descriptors:   524288
> 	Largest file desc currently in use:     21
> 	Number of file desc currently in use:   33
> 	Files queued for open:                   0
> 	Available number of file descriptors: 524255
> 	Reserved number of file descriptors:   400
> 	Store Disk files open:                   1
> Internal Data Structures:
> 	   214 StoreEntries
> 	   214 StoreEntries with MemObjects
> 	    49 Hot Object Cache Items
> 	   799 on-disk objects
> ===================================================================
> Cache Manager menu
> sample_start_time = 1383848634.943911 (Thu, 07 Nov 2013 18:23:54 GMT)
> sample_end_time = 1383848994.404908 (Thu, 07 Nov 2013 18:29:54 GMT)
> client_http.requests = 0.643331/sec
> client_http.hits = 0.000000/sec
> client_http.errors = 0.006667/sec
> client_http.kbytes_in = 0.379999/sec
> client_http.kbytes_out = 56.286475/sec
> client_http.all_median_svc_time = 2.002465 seconds
> client_http.miss_median_svc_time = 2.002465 seconds
> client_http.nm_median_svc_time = 0.000000 seconds
> client_http.nh_median_svc_time = 0.000000 seconds
> client_http.hit_median_svc_time = 0.000000 seconds
> server.all.requests = 0.603331/sec
> server.all.errors = 0.000000/sec
> server.all.kbytes_in = 56.173142/sec
> server.all.kbytes_out = 0.386665/sec
> server.http.requests = 0.603331/sec
> server.http.errors = 0.000000/sec
> server.http.kbytes_in = 56.173142/sec
> server.http.kbytes_out = 0.386665/sec
> server.ftp.requests = 0.000000/sec
> server.ftp.errors = 0.000000/sec
> server.ftp.kbytes_in = 0.000000/sec
> server.ftp.kbytes_out = 0.000000/sec
> server.other.requests = 0.000000/sec
> server.other.errors = 0.000000/sec
> server.other.kbytes_in = 0.000000/sec
> server.other.kbytes_out = 0.000000/sec
> icp.pkts_sent = 0.000000/sec
> icp.pkts_recv = 0.000000/sec
> icp.queries_sent = 0.000000/sec
> icp.replies_sent = 0.000000/sec
> icp.queries_recv = 0.000000/sec
> icp.replies_recv = 0.000000/sec
> icp.replies_queued = 0.000000/sec
> icp.query_timeouts = 0.000000/sec
> icp.kbytes_sent = 0.000000/sec
> icp.kbytes_recv = 0.000000/sec
> icp.q_kbytes_sent = 0.000000/sec
> icp.r_kbytes_sent = 0.000000/sec
> icp.q_kbytes_recv = 0.000000/sec
> icp.r_kbytes_recv = 0.000000/sec
> icp.query_median_svc_time = 0.000000 seconds
> icp.reply_median_svc_time = 0.000000 seconds
> dns.median_svc_time = 4.659619 seconds
> unlink.requests = 0.000000/sec
> page_faults = 0.000000/sec
> select_loops = 1221.763190/sec
> select_fds = 24.383249/sec
> average_select_fd_period = 0.000000/fd
> median_select_fds = -1.000000
> swap.outs = 0.163333/sec
> swap.ins = 0.000000/sec
> swap.files_cleaned = 0.000000/sec
> aborted_requests = 0.026667/sec
> syscalls.disk.opens = 0.006667/sec
> syscalls.disk.closes = 0.006667/sec
> syscalls.disk.reads = 0.000000/sec
> syscalls.disk.writes = 0.163333/sec
> syscalls.disk.seeks = 0.000000/sec
> syscalls.disk.unlinks = 0.000000/sec
> syscalls.sock.accepts = 0.906662/sec
> syscalls.sock.sockets = 0.983330/sec
> syscalls.sock.connects = 0.616665/sec
> syscalls.sock.binds = 0.000000/sec
> syscalls.sock.closes = 1.676661/sec
> syscalls.sock.reads = 6.183312/sec
> syscalls.sock.writes = 16.389944/sec
> syscalls.sock.recvfroms = 0.673331/sec
> syscalls.sock.sendtos = 0.779997/sec
> cpu_time = 0.687000 seconds
> wall_time = 1200.007428 seconds
> cpu_usage = 0.057250%
>
> Generated Thu, 07 Nov 2013 18:29:54 GMT, by cachemgr.cgi/3.3.9@squid
> ==========================================================================
> *
> after pumping  250 users at  10:35*
>
> Cache Manager menu
>
>
> sample_start_time = 1383848994.404908 (Thu, 07 Nov 2013 18:29:54 GMT)
> sample_end_time = 1383849297.332922 (Thu, 07 Nov 2013 18:34:57 GMT)
> client_http.requests = 57.339965/sec
> client_http.hits = 2.996665/sec
> client_http.errors = 1.753332/sec
> client_http.kbytes_in = 42.156641/sec
> client_http.kbytes_out = 3265.764662/sec
> client_http.all_median_svc_time = 0.277925 seconds
> client_http.miss_median_svc_time = 0.298933 seconds
> client_http.nm_median_svc_time = 0.000000 seconds
> client_http.nh_median_svc_time = 0.090797 seconds
> client_http.hit_median_svc_time = 0.000000 seconds
> server.all.requests = 51.156635/sec
> server.all.errors = 0.000000/sec
> server.all.kbytes_in = 3269.724660/sec
> server.all.kbytes_out = 41.756641/sec
> server.http.requests = 51.156635/sec
> server.http.errors = 0.000000/sec
> server.http.kbytes_in = 3269.724660/sec
> server.http.kbytes_out = 41.756641/sec
> server.ftp.requests = 0.000000/sec
> server.ftp.errors = 0.000000/sec
> server.ftp.kbytes_in = 0.000000/sec
> server.ftp.kbytes_out = 0.000000/sec
> server.other.requests = 0.000000/sec
> server.other.errors = 0.000000/sec
> server.other.kbytes_in = 0.000000/sec
> server.other.kbytes_out = 0.000000/sec
> icp.pkts_sent = 0.000000/sec
> icp.pkts_recv = 0.000000/sec
> icp.queries_sent = 0.000000/sec
> icp.replies_sent = 0.000000/sec
> icp.queries_recv = 0.000000/sec
> icp.replies_recv = 0.000000/sec
> icp.replies_queued = 0.000000/sec
> icp.query_timeouts = 0.000000/sec
> icp.kbytes_sent = 0.000000/sec
> icp.kbytes_recv = 0.000000/sec
> icp.q_kbytes_sent = 0.000000/sec
> icp.r_kbytes_sent = 0.000000/sec
> icp.q_kbytes_recv = 0.000000/sec
> icp.r_kbytes_recv = 0.000000/sec
> icp.query_median_svc_time = 0.000000 seconds
> icp.reply_median_svc_time = 0.000000 seconds
> dns.median_svc_time = 5.745147 seconds
> unlink.requests = 0.000000/sec
> page_faults = 0.000000/sec
> select_loops = 6369.086425/sec
> select_fds = 1972.625470/sec
> average_select_fd_period = 0.000000/fd
> median_select_fds = -0.750000
> swap.outs = 16.923323/sec
> swap.ins = 0.106667/sec
> swap.files_cleaned = 0.000000/sec
> aborted_requests = 3.266665/sec
> syscalls.disk.opens = 1.109999/sec
> syscalls.disk.closes = 1.109999/sec
> syscalls.disk.reads = 0.176667/sec
> syscalls.disk.writes = 16.826667/sec
> syscalls.disk.seeks = 0.000000/sec
> syscalls.disk.unlinks = 0.000000/sec
> syscalls.sock.accepts = 101.956603/sec
> syscalls.sock.sockets = 87.023290/sec
> syscalls.sock.connects = 54.083300/sec
> syscalls.sock.binds = 53.529967/sec
> syscalls.sock.closes = 147.856586/sec
> syscalls.sock.reads = 681.629588/sec
> syscalls.sock.writes = 1133.832646/sec
> syscalls.sock.recvfroms = 18.026655/sec
> syscalls.sock.sendtos = 30.143313/sec
> cpu_time = 34.626000 seconds
> wall_time = 1200.000613 seconds
> cpu_usage = 2.885499%
> =================================================================
> Generated Thu, 07 Nov 2013 18:35:51 GMT, by cachemgr.cgi/3.3.9@squid
> ========================================
> Cache Manager menu
> Squid Object Cache: Version 3.3.9
>
> Start Time:	Thu, 07 Nov 2013 18:22:47 GMT
> Current Time:	Thu, 07 Nov 2013 18:36:28 GMT
>
> Connection information for squid:
> 	Number of clients accessing cache:	603
> 	Number of HTTP requests received:	24710
> 	Number of ICP messages received:	0
> 	Number of ICP messages sent:	0
> 	Number of queued ICP replies:	0
> 	Number of HTCP messages received:	0
> 	Number of HTCP messages sent:	0
> 	Request failure ratio:	 0.00
> 	Average HTTP requests per minute since start:	1804.5
> 	Average ICP messages per minute since start:	0.0
> 	Select loop called: 3529673 times, 1.022 ms avg
> Cache information for squid:
> 	Hits as % of all requests:	5min: 3.7%, 60min: 3.6%
> 	Hits as % of bytes sent:	5min: -0.1%, 60min: -0.1%
> 	Memory hits as % of hit requests:	5min: 36.1%, 60min: 36.1%
> 	Disk hits as % of hit requests:	5min: 1.8%, 60min: 1.8%
> 	Storage Swap size:	179632 KB
> 	Storage Swap capacity:	17.5% used, 82.5% free
> 	Storage Mem size:	149536 KB
> 	Storage Mem capacity:	28.5% used, 71.5% free
> 	Mean Object Size:	32.00 KB
> 	Requests given to unlinkd:	0
> Median Service Times (seconds)  5 min    60 min:
> 	HTTP Requests (All):   0.28318  0.28318
> 	Cache Misses:          0.32244  0.32244
> 	Cache Hits:            0.00000  0.00000
> 	Near Hits:             0.09080  0.09080
> 	Not-Modified Replies:  0.00000  0.00000
> 	DNS Lookups:           5.74515  5.74515
> 	ICP Queries:           0.00000  0.00000
> Resource usage for squid:
> 	UP Time:	821.616 seconds
> 	CPU Time:	86.324 seconds
> 	CPU Usage:	10.51%
> 	CPU Usage, 5 minute avg:	15.10%
> 	CPU Usage, 60 minute avg:	10.32%
> 	Process Data Segment Size via sbrk(): 278704 KB
> 	Maximum Resident Size: 2119536 KB
> 	Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
> 	Total space in arena:  279232 KB
> 	Ordinary blocks:       273204 KB   2572 blks
> 	Small blocks:               0 KB      0 blks
> 	Holding blocks:        291440 KB     31 blks
> 	Free Small blocks:          0 KB
> 	Free Ordinary blocks:    6028 KB
> 	Total in use:            6028 KB 1%
> 	Total free:              6028 KB 1%
> 	Total size:            570672 KB
> Memory accounted for:
> 	Total accounted:        21831 KB   4%
> 	memPool accounted:      21831 KB   4%
> 	memPool unaccounted:   548841 KB  96%
> 	memPoolAlloc calls:        52
> 	memPoolFree calls:    6375465
> File descriptor usage for squid:
> 	Maximum number of file descriptors:   524288
> 	Largest file desc currently in use:    886
> 	Number of file desc currently in use: 1146
> 	Files queued for open:                   0
> 	Available number of file descriptors: 523142
> 	Reserved number of file descriptors:   400
> 	Store Disk files open:                   2
> Internal Data Structures:
> 	   511 StoreEntries
> 	   511 StoreEntries with MemObjects
> 	  4673 Hot Object Cache Items
> 	  5613 on-disk objects
>
> Generated Thu, 07 Nov 2013 18:36:28 GMT, by cachemgr.cgi/3.3.9@squid
>
> =============================================================
>
> *at 8:40
> degradation started !!!*
> =================================================================
> Cache Manager menu
>
>
> Squid Object Cache: Version 3.3.9
>
> Start Time:	Thu, 07 Nov 2013 18:22:47 GMT
> Current Time:	Thu, 07 Nov 2013 18:40:50 GMT
>
> Connection information for squid:
> 	Number of clients accessing cache:	682
> 	Number of HTTP requests received:	44173
> 	Number of ICP messages received:	0
> 	Number of ICP messages sent:	0
> 	Number of queued ICP replies:	0
> 	Number of HTCP messages received:	0
> 	Number of HTCP messages sent:	0
> 	Request failure ratio:	 0.00
> 	Average HTTP requests per minute since start:	2446.7
> 	Average ICP messages per minute since start:	0.0
> 	Select loop called: 5938295 times, 0.824 ms avg
> Cache information for squid:
> 	Hits as % of all requests:	5min: 4.0%, 60min: 4.0%
> 	Hits as % of bytes sent:	5min: -0.1%, 60min: -0.1%
> 	Memory hits as % of hit requests:	5min: 34.2%, 60min: 36.6%
> 	Disk hits as % of hit requests:	5min: 2.8%, 60min: 2.5%
> 	Storage Swap size:	275024 KB
> 	Storage Swap capacity:	26.9% used, 73.1% free
> 	Storage Mem size:	227904 KB
> 	Storage Mem capacity:	43.5% used, 56.5% free
> 	Mean Object Size:	32.00 KB
> 	Requests given to unlinkd:	0
> Median Service Times (seconds)  5 min    60 min:
> 	HTTP Requests (All):   0.35277  0.31556
> 	Cache Misses:          0.39683  0.34658
> 	Cache Hits:            0.00000  0.00000
> 	Near Hits:             0.09080  0.09080
> 	Not-Modified Replies:  0.00000  0.00000
> 	DNS Lookups:           5.49589  5.57654
> 	ICP Queries:           0.00000  0.00000
> Resource usage for squid:
> 	UP Time:	1083.267 seconds
> 	CPU Time:	130.129 seconds
> 	CPU Usage:	12.01%
> 	CPU Usage, 5 minute avg:	17.10%
> 	CPU Usage, 60 minute avg:	11.87%
> 	Process Data Segment Size via sbrk(): 279416 KB
> 	Maximum Resident Size: 2440480 KB
> 	Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
> 	Total space in arena:  279944 KB
> 	Ordinary blocks:       274059 KB   2744 blks
> 	Small blocks:               0 KB      0 blks
> 	Holding blocks:        291440 KB     31 blks
> 	Free Small blocks:          0 KB
> 	Free Ordinary blocks:    5885 KB
> 	Total in use:            5885 KB 1%
> 	Total free:              5885 KB 1%
> 	Total size:            571384 KB
> Memory accounted for:
> 	Total accounted:        22503 KB   4%
> 	memPool accounted:      22503 KB   4%
> 	memPool unaccounted:   548881 KB  96%
> 	memPoolAlloc calls:        52
> 	memPoolFree calls:   11335629
> File descriptor usage for squid:
> 	Maximum number of file descriptors:   524288
> 	Largest file desc currently in use:    848
> 	Number of file desc currently in use: 1083
> 	Files queued for open:                   0
> 	Available number of file descriptors: 523205
> 	Reserved number of file descriptors:   400
> 	Store Disk files open:                   3
> Internal Data Structures:
> 	   508 StoreEntries
> 	   507 StoreEntries with MemObjects
> 	  7122 Hot Object Cache Items
> 	  8594 on-disk objects
>
>
>
> Generated Thu, 07 Nov 2013 18:40:49 GMT, by cachemgr.cgi/3.3.9@squid
> ======================================================================
> Cache Manager menu
>
>
> sample_start_time = 1383849354.404908 (Thu, 07 Nov 2013 18:35:54 GMT)
> sample_end_time = 1383849657.332992 (Thu, 07 Nov 2013 18:40:57 GMT)
> client_http.requests = 74.816660/sec
> client_http.hits = 4.343333/sec
> client_http.errors = 1.623333/sec
> client_http.kbytes_in = 54.336662/sec
> client_http.kbytes_out = 4725.706260/sec
> client_http.all_median_svc_time = 0.341861 seconds
> client_http.miss_median_svc_time = 0.396834 seconds
> client_http.nm_median_svc_time = 0.000000 seconds
> client_http.nh_median_svc_time = 0.090797 seconds
> client_http.hit_median_svc_time = 0.000000 seconds
> server.all.requests = 65.903327/sec
> server.all.errors = 0.000000/sec
> server.all.kbytes_in = 4729.139594/sec
> server.all.kbytes_out = 54.566662/sec
> server.http.requests = 65.903327/sec
> server.http.errors = 0.000000/sec
> server.http.kbytes_in = 4729.139594/sec
> server.http.kbytes_out = 54.566662/sec
> server.ftp.requests = 0.000000/sec
> server.ftp.errors = 0.000000/sec
> server.ftp.kbytes_in = 0.000000/sec
> server.ftp.kbytes_out = 0.000000/sec
> server.other.requests = 0.000000/sec
> server.other.errors = 0.000000/sec
> server.other.kbytes_in = 0.000000/sec
> server.other.kbytes_out = 0.000000/sec
> icp.pkts_sent = 0.000000/sec
> icp.pkts_recv = 0.000000/sec
> icp.queries_sent = 0.000000/sec
> icp.replies_sent = 0.000000/sec
> icp.queries_recv = 0.000000/sec
> icp.replies_recv = 0.000000/sec
> icp.replies_queued = 0.000000/sec
> icp.query_timeouts = 0.000000/sec
> icp.kbytes_sent = 0.000000/sec
> icp.kbytes_recv = 0.000000/sec
> icp.q_kbytes_sent = 0.000000/sec
> icp.r_kbytes_sent = 0.000000/sec
> icp.q_kbytes_recv = 0.000000/sec
> icp.r_kbytes_recv = 0.000000/sec
> icp.query_median_svc_time = 0.000000 seconds
> icp.reply_median_svc_time = 0.000000 seconds
> dns.median_svc_time = 5.495891 seconds
> unlink.requests = 0.000000/sec
> page_faults = 0.000000/sec
> select_loops = 9235.691514/sec
> select_fds = 2959.706378/sec
> average_select_fd_period = 0.000000/fd
> median_select_fds = -0.750000
> swap.outs = 19.856665/sec
> swap.ins = 0.456667/sec
> swap.files_cleaned = 0.000000/sec
> aborted_requests = 5.519999/sec
> syscalls.disk.opens = 0.353333/sec
> syscalls.disk.closes = 0.353333/sec
> syscalls.disk.reads = 0.563333/sec
> syscalls.disk.writes = 19.843324/sec
> syscalls.disk.seeks = 0.000000/sec
> syscalls.disk.unlinks = 0.000000/sec
> syscalls.sock.accepts = 127.843316/sec
> syscalls.sock.sockets = 110.706649/sec
> syscalls.sock.connects = 70.273327/sec
> syscalls.sock.binds = 69.683327/sec
> syscalls.sock.closes = 193.976642/sec
> syscalls.sock.reads = 1081.103225/sec
> syscalls.sock.writes = 1679.543177/sec
> syscalls.sock.recvfroms = 21.279997/sec
> syscalls.sock.sendtos = 33.633328/sec
> cpu_time = 50.542000 seconds
> wall_time = 1200.000335 seconds
> cpu_usage = 4.211832%
>
>
> enerated Thu, 07 Nov 2013 18:41:19 GMT, by cachemgr.cgi/3.3.9@squid
> ==============================================================
>
> *at 10:45 ,
>
> squid is very  bad and cant login to ssh terminal and interrupts if terminal
> succeed*
>
> ==================================
> top command
> top - 14:07:43 up 1 day,  5:23,  4 users,  load average: 0.11, 0.15, 0.15
> Tasks: 159 total,   1 running, 158 sleeping,   0 stopped,   0 zombie
> Cpu0  :  1.0%us,  0.3%sy,  0.0%ni, 93.9%id,  0.7%wa,  0.0%hi,  4.1%si,
> 0.0%st
> Cpu1  :  1.0%us,  1.0%sy,  0.0%ni, 97.2%id,  0.0%wa,  0.0%hi,  0.7%si,
> 0.0%st
> Cpu2  :  0.0%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.3%si,
> 0.0%st
> Cpu3  :  2.4%us,  1.7%sy,  0.0%ni, 94.8%id,  0.0%wa,  0.0%hi,  1.0%si,
> 0.0%st
> Cpu4  :  0.7%us,  0.7%sy,  0.0%ni, 97.3%id,  0.0%wa,  0.0%hi,  1.4%si,
> 0.0%st
> Cpu5  :  1.3%us,  0.7%sy,  0.0%ni, 96.3%id,  0.0%wa,  0.0%hi,  1.7%si,
> 0.0%st
> Cpu6  :  0.7%us,  0.7%sy,  0.0%ni, 97.0%id,  0.0%wa,  0.0%hi,  1.7%si,
> 0.0%st
> Cpu7  :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Mem:   8182336k total,  7957752k used,   224584k free,   172808k buffers
> Swap:        0k total,        0k used,        0k free,  6897624k cached
> ==================================================================
>
>
> Cache Manager menu
>
>
> Squid Object Cache: Version 3.3.9
>
> Start Time:	Thu, 07 Nov 2013 18:22:47 GMT
> Current Time:	Thu, 07 Nov 2013 18:46:38 GMT
>
> Connection information for squid:
> 	Number of clients accessing cache:	749
> 	Number of HTTP requests received:	76299
> 	Number of ICP messages received:	0
> 	Number of ICP messages sent:	0
> 	Number of queued ICP replies:	0
> 	Number of HTCP messages received:	0
> 	Number of HTCP messages sent:	0
> 	Request failure ratio:	 0.00
> 	Average HTTP requests per minute since start:	3198.4
> 	Average ICP messages per minute since start:	0.0
> 	Select loop called: 9250833 times, 0.712 ms avg
> Cache information for squid:
> 	Hits as % of all requests:	5min: 5.7%, 60min: 4.7%
> 	Hits as % of bytes sent:	5min: 0.1%, 60min: 0.0%
> 	Memory hits as % of hit requests:	5min: 46.1%, 60min: 40.8%
> 	Disk hits as % of hit requests:	5min: 3.3%, 60min: 3.0%
> 	Storage Swap size:	395280 KB
> 	Storage Swap capacity:	38.6% used, 61.4% free
> 	Storage Mem size:	312704 KB
> 	Storage Mem capacity:	59.6% used, 40.4% free
> 	Mean Object Size:	32.00 KB
> 	Requests given to unlinkd:	0
> Median Service Times (seconds)  5 min    60 min:
> 	HTTP Requests (All):   0.25301  0.28720
> 	Cache Misses:          0.28720  0.31417
> 	Cache Hits:            0.00000  0.00000
> 	Near Hits:             0.09080  0.09080
> 	Not-Modified Replies:  0.00000  0.00000
> 	DNS Lookups:           5.49589  5.49589
> 	ICP Queries:           0.00000  0.00000
> Resource usage for squid:
> 	UP Time:	1431.315 seconds
> 	CPU Time:	193.163 seconds
> 	CPU Usage:	13.50%
> 	CPU Usage, 5 minute avg:	18.24%
> 	CPU Usage, 60 minute avg:	13.46%
> 	Process Data Segment Size via sbrk(): 280820 KB
> 	Maximum Resident Size: 2865248 KB
> 	Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
> 	Total space in arena:  281348 KB
> 	Ordinary blocks:       274362 KB   2359 blks
> 	Small blocks:               0 KB      0 blks
> 	Holding blocks:        291440 KB     31 blks
> 	Free Small blocks:          0 KB
> 	Free Ordinary blocks:    6986 KB
> 	Total in use:            6986 KB 1%
> 	Total free:              6986 KB 1%
> 	Total size:            572788 KB
> Memory accounted for:
> 	Total accounted:        22509 KB   4%
> 	memPool accounted:      22509 KB   4%
> 	memPool unaccounted:   550279 KB  96%
> 	memPoolAlloc calls:        52
> 	memPoolFree calls:   19470245
> File descriptor usage for squid:
> 	Maximum number of file descriptors:   524288
> 	Largest file desc currently in use:    892
> 	Number of file desc currently in use: 1121
> 	Files queued for open:                   0
> 	Available number of file descriptors: 523167
> 	Reserved number of file descriptors:   400
> 	Store Disk files open:                   2
> Internal Data Structures:
> 	   497 StoreEntries
> 	   496 StoreEntries with MemObjects
> 	  9772 Hot Object Cache Items
> 	 12352 on-disk objects
>
>
>
> Generated Thu, 07 Nov 2013 18:46:37 GMT, by cachemgr.cgi/3.3.9@squid
> ==================================================================
> Cache Manager menu
> sample_start_time = 1383849714.405043 (Thu, 07 Nov 2013 18:41:54 GMT)
> sample_end_time = 1383850017.332992 (Thu, 07 Nov 2013 18:46:57 GMT)
> client_http.requests = 90.849991/sec
> client_http.hits = 6.389999/sec
> client_http.errors = 1.430000/sec
> client_http.kbytes_in = 70.366660/sec
> client_http.kbytes_out = 4938.722996/sec
> client_http.all_median_svc_time = 0.253008 seconds
> client_http.miss_median_svc_time = 0.287199 seconds
> client_http.nm_median_svc_time = 0.000000 seconds
> client_http.nh_median_svc_time = 0.090797 seconds
> client_http.hit_median_svc_time = 0.000000 seconds
> server.all.requests = 79.706659/sec
> server.all.errors = 0.000000/sec
> server.all.kbytes_in = 4931.719662/sec
> server.all.kbytes_out = 69.639993/sec
> server.http.requests = 79.706659/sec
> server.http.errors = 0.000000/sec
> server.http.kbytes_in = 4931.719662/sec
> server.http.kbytes_out = 69.639993/sec
> server.ftp.requests = 0.000000/sec
> server.ftp.errors = 0.000000/sec
> server.ftp.kbytes_in = 0.000000/sec
> server.ftp.kbytes_out = 0.000000/sec
> server.other.requests = 0.000000/sec
> server.other.errors = 0.000000/sec
> server.other.kbytes_in = 0.000000/sec
> server.other.kbytes_out = 0.000000/sec
> icp.pkts_sent = 0.000000/sec
> icp.pkts_recv = 0.000000/sec
> icp.queries_sent = 0.000000/sec
> icp.replies_sent = 0.000000/sec
> icp.queries_recv = 0.000000/sec
> icp.replies_recv = 0.000000/sec
> icp.replies_queued = 0.000000/sec
> icp.query_timeouts = 0.000000/sec
> icp.kbytes_sent = 0.000000/sec
> icp.kbytes_recv = 0.000000/sec
> icp.q_kbytes_sent = 0.000000/sec
> icp.r_kbytes_sent = 0.000000/sec
> icp.q_kbytes_recv = 0.000000/sec
> icp.r_kbytes_recv = 0.000000/sec
> icp.query_median_svc_time = 0.000000 seconds
> icp.reply_median_svc_time = 0.000000 seconds
> dns.median_svc_time = 5.415243 seconds
> unlink.requests = 0.000000/sec
> page_faults = 0.000000/sec
> select_loops = 9473.238553/sec
> select_fds = 3062.906428/sec
> average_select_fd_period = 0.000000/fd
> median_select_fds = -0.750000
> swap.outs = 23.526664/sec
> swap.ins = 0.493333/sec
> swap.files_cleaned = 0.000000/sec
> aborted_requests = 5.513333/sec
> syscalls.disk.opens = 0.373333/sec
> syscalls.disk.closes = 0.373333/sec
> syscalls.disk.reads = 0.870000/sec
> syscalls.disk.writes = 23.640000/sec
> syscalls.disk.seeks = 0.000000/sec
> syscalls.disk.unlinks = 0.000000/sec
> syscalls.sock.accepts = 154.756643/sec
> syscalls.sock.sockets = 131.653323/sec
> syscalls.sock.connects = 83.856659/sec
> syscalls.sock.binds = 83.043326/sec
> syscalls.sock.closes = 231.946647/sec
> syscalls.sock.reads = 1084.103251/sec
> syscalls.sock.writes = 1738.646543/sec
> syscalls.sock.recvfroms = 23.476662/sec
> syscalls.sock.sendtos = 37.269994/sec
> cpu_time = 54.046000 seconds
> wall_time = 1200.000237 seconds
> cpu_usage = 4.503832%
>
> Generated Thu, 07 Nov 2013 18:47:06 GMT, by cachemgr.cgi/3.3.9@squid
>
> =========================
>
>
>
> just to hint , the free memory before using squid is 1 g  and after some
> time   the free is about 200 M
> could this be the problem ???
> i did my best to let squid far from memory problems .
> =========
> currently i stopped squid and want to fix the issue before using the other
> servers and pump more users
>
> regards
>
>
>
>
> -----
> Dr.x
> --
> View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/is-there-any-thing-wrong-from-cache-manager-logs-tp4663156p4663169.html
> Sent from the Squid - Users mailing list archive at Nabble.com.