OK another small test:
##START
ls /proc/sys/net/netfilter/nf_conntrack_* |tee
ls /proc/sys/net/netfilter/nf_conntrack_* |xargs -n1 -l1 cat
ls /proc/net/nf_conntrack
##END
If the "/proc/net/nf_conntrack" file do exits then run these:
##START
cat /proc/net/nf_conntrack |wc -l
cat /proc/net/nf_conntrack |grep UDP |wc -l
cat /proc/net/nf_conntrack |grep ESTABLISHED |wc -l
cat /proc/net/nf_conntrack |grep TIME_WAIT |wc -l
cat /proc/net/nf_conntrack |grep CLOSE_WAIT |wc -l
cat /proc/net/nf_conntrack |grep LISTEN |wc -l
cat /proc/net/nf_conntrack |grep ":53" |wc -l
cat /proc/net/nf_conntrack |grep ":TPROXYPORT" |wc -l
##END
Which should show us couple statistics about the issue.
also if you want to take a snapshot of "top" you can try to start with:
top -n 1 -b
which will get lots of out put that you will need to consult the "man
top" first so you would get more info on how to clear un-needed info
from the snapshot.
Eliezer
On 11/08/2013 01:21 AM, Dr.x wrote:
Eliezer Croitoru-2 wrote
OK so after Amos did the calculations (Thanks) I assume that using lsof
will give us more clue about it.
The first thing to do is to start a ssh session into the server with the
setting:
ServerAliveInterval 30
Added into the /etc/ssh/ssh_config (on the client side)
When you do have this session running you wont have any troubles running
top or any other basic tests on the server while there is a degradation.
Now this command is what you will need in order to make the suspicious
more accurate and maybe lead to something that can help you:
"lsof -u squid -a -i 4 -n -P"
(squid is the default username on centos for the proxy user)
Dont try to run this command just like this out of the blue since the
output can be more then 60k lines long..
You should try to throw it into a file at the tmp dir so this:
"lsof -u squid -a -i 4 -n -P >/tmp/tmp_lsof.1"
Should be safe.
The next thing is to find out how many FD are in sum and how many are
ESTABLISHED etc.. so run these:
##START
lsof -u squid -a -i 4 -n -P >/tmp/tmp_lsof.1
cat /tmp/tmp_lsof.1 |wc -l
cat /tmp/tmp_lsof.1 |grep UDP |wc -l
cat /tmp/tmp_lsof.1 |grep ESTABLISHED |wc -l
cat /tmp/tmp_lsof.1 |grep TIME_WAIT |wc -l
cat /tmp/tmp_lsof.1 |grep CLOSE_WAIT |wc -l
cat /tmp/tmp_lsof.1 |grep LISTEN |wc -l
cat /tmp/tmp_lsof.1 |grep ":53" |wc -l
cat /tmp/tmp_lsof.1 |grep ":TPROXYPORT" |wc -l
##END
(TPROXYPORT IS the port from squid.conf)
Once you have all the above results before the degradation in it and
after we might have a clue about the source of the problem and whether
it comes from too much FD which are not being used but causing the
system to loop throw lots of them.
hi , 1st of all , does i face a loop ??
anyway , ive made another test
before the degredation , 1st 5 minutes of squid
[root@squid ~]# cat /tmp/tmp_lsof.1 |wc -l
3398
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep UDP |wc -l
6
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep ESTABLISHED |wc -l
3134
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep TIME_WAIT |wc -l
0
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep CLOSE_WAIT |wc -l
1
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep LISTEN |wc -l
8
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep ":53" |wc -l
73
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep ":3129" |wc -l
5
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]#
[root@squid ~]# lsof -u squid -a -i 4 -n -P >/tmp/tmp_lsof.1
cat /tmp/tmp_lsof.1 |wc -l
[root@squid ~]# cat /tmp/tmp_lsof.1 |wc -l
3366
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep UDP |wc -l
6
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep ESTABLISHED |wc -l
3118
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep TIME_WAIT |wc -l
0
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep CLOSE_WAIT |wc -l
2
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep LISTEN |wc -l
8
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep ":53" |wc -l
82
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep ":3129" |wc -l
5
==================================================
after degredation and system close to dead !
[root@squid ~]# cat /tmp/tmp_lsof.1 |wc -l
3505
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep UDP |wc -l
6
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep ESTABLISHED |wc -l
3262
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep TIME_WAIT |wc -l
0
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep CLOSE_WAIT |wc -l
5
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep LISTEN |wc -l
8
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep ":53" |wc -l
57
[root@squid ~]# cat /tmp/tmp_lsof.1 |grep ":3129" |wc -l
4
-----
Dr.x
--
View this message in context: http://squid-web-proxy-cache.1019090.n4.nabble.com/is-there-any-thing-wrong-from-cache-manager-logs-tp4663156p4663180.html
Sent from the Squid - Users mailing list archive at Nabble.com.
