Search squid archive

bad kernel logs on high traffic server

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi guys , I have server with about 1000 users and its okay with 2 workers

 

Now I added more 1000 and its now 2000 users with 4 workers

 

But sometime some process of squid get killed suddenly and here is log below ,

 

Here  is sysctl.conf

[root@Largerock-squid ~]# cat  /etc/sysctl.conf    

# Kernel sysctl configuration file for Red Hat Linux

#

# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and

# sysctl.conf(5) for more details.

 

# Controls IP packet forwarding

################net.ipv4.ip_forward = 0

net.ipv4.ip_forward = 1

# Controls source route verification

#################net.ipv4.conf.default.rp_filter = 1

net.ipv4.conf.default.rp_filter = 0

###################################

net.ipv4.conf.all.rp_filter = 0

net.ipv4.conf.eth0.rp_filter = 0

###############################

# Do not accept source routing

net.ipv4.conf.default.accept_source_route = 0

 

# Controls the System Request debugging functionality of the kernel

kernel.sysrq = 0

 

# Controls whether core dumps will append the PID to the core filename.

# Useful for debugging multi-threaded applications.

kernel.core_uses_pid = 1

 

# Controls the use of TCP syncookies

net.ipv4.tcp_syncookies = 1

 

# Disable netfilter on bridges.

net.bridge.bridge-nf-call-ip6tables = 0

net.bridge.bridge-nf-call-iptables = 0

net.bridge.bridge-nf-call-arptables = 0

 

# Controls the default maxmimum size of a mesage queue

kernel.msgmnb = 65536

 

# Controls the maximum size of a message, in bytes

kernel.msgmax = 65536

 

# Controls the maximum shared segment size, in bytes

kernel.shmmax = 68719476736

 

# Controls the maximum number of shared memory segments, in pages

kernel.shmall = 4294967296

################################

###############################

fs.file-max = 1131072

net.local.dgram.recvspace = 1262144

net.local.dgram.maxdgram = 116384

net.netfilter.nf_conntrack_max = 1130000

########################################

net.nf_conntrack_max = 1131072

#########################################

####kernel: possible SYN flooding on port 80. Sending cookies.#####

net.ipv4.tcp_syncookies=0

 

 

im wondering wt should I do ,   or modify my kernel linux

 

the error code is below :

 

 

 

Nov  7 14:52:16 Largerock-squid snmpd[1576]: Connection from UDP: [xxx]:57067xxx]

Nov  7 14:52:17 Largerock-squid kernel: snmpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

Nov  7 14:52:17 Largerock-squid kernel: snmpd cpuset=/ mems_allowed=0

Nov  7 14:52:17 Largerock-squid kernel: Pid: 1576, comm: snmpd Not tainted 2.6.32-431.el6.x86_64 #1

Nov  7 14:52:17 Largerock-squid kernel: Call Trace:

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff810d05b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff81122960>] ? dump_header+0x90/0x1b0

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8122798c>] ? security_real_capable_noaudit+0x3c/0x70

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff81122de2>] ? oom_kill_process+0x82/0x2a0

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff81122d21>] ? select_bad_process+0xe1/0x120

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff81123220>] ? out_of_memory+0x220/0x3c0

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8112fb3c>] ? __alloc_pages_nodemask+0x8ac/0x8d0

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff81167a9a>] ? alloc_pages_current+0xaa/0x110

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8111fd57>] ? __page_cache_alloc+0x87/0x90

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8111f73e>] ? find_get_page+0x1e/0xa0

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff81120cf7>] ? filemap_fault+0x1a7/0x500

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8114a084>] ? __do_fault+0x54/0x530

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8114a657>] ? handle_pte_fault+0xf7/0xb00

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff81447c54>] ? move_addr_to_kernel+0x64/0x70

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff810129de>] ? copy_user_generic+0xe/0x20

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8114b28a>] ? handle_mm_fault+0x22a/0x300

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff81448119>] ? sys_sendto+0x139/0x190

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff810149b9>] ? read_tsc+0x9/0x20

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff810a70a1>] ? ktime_get_ts+0xb1/0xf0

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8119fb18>] ? poll_select_copy_remaining+0xf8/0x150

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8152d45e>] ? do_page_fault+0x3e/0xa0

Nov  7 14:52:17 Largerock-squid kernel: [<ffffffff8152a815>] ? page_fault+0x25/0x30

Nov  7 14:52:17 Largerock-squid kernel: Mem-Info:

Nov  7 14:52:17 Largerock-squid kernel: Node 0 DMA per-cpu:

Nov  7 14:52:17 Largerock-squid kernel: CPU    0: hi:    0, btch:   1 usd:   0

Nov  7 14:52:17 Largerock-squid kernel: CPU    1: hi:    0, btch:   1 usd:   0

Nov  7 14:52:17 Largerock-squid kernel: CPU    2: hi:    0, btch:   1 usd:   0

Nov  7 14:52:17 Largerock-squid kernel: CPU    3: hi:    0, btch:   1 usd:   0

Nov  7 14:52:17 Largerock-squid kernel: CPU    4: hi:    0, btch:   1 usd:   0

Nov  7 14:52:17 Largerock-squid kernel: CPU    5: hi:    0, btch:   1 usd:   0

Nov  7 14:52:17 Largerock-squid kernel: CPU    6: hi:    0, btch:   1 usd:   0

Nov  7 14:52:17 Largerock-squid kernel: CPU    7: hi:    0, btch:   1 usd:   0

Nov  7 14:52:17 Largerock-squid kernel: Node 0 DMA32 per-cpu:

Nov  7 14:52:17 Largerock-squid kernel: CPU    0: hi:  186, btch:  31 usd:  77

Nov  7 14:52:17 Largerock-squid kernel: CPU    1: hi:  186, btch:  31 usd:  34

Nov  7 14:52:17 Largerock-squid kernel: CPU    2: hi:  186, btch:  31 usd:  31

Nov  7 14:52:17 Largerock-squid kernel: CPU    3: hi:  186, btch:  31 usd:  33

Nov  7 14:52:17 Largerock-squid kernel: CPU    4: hi:  186, btch:  31 usd:  24

Nov  7 14:52:17 Largerock-squid kernel: CPU    5: hi:  186, btch:  31 usd:  24

Nov  7 14:52:17 Largerock-squid kernel: CPU    6: hi:  186, btch:  31 usd:  22

Nov  7 14:52:17 Largerock-squid kernel: CPU    7: hi:  186, btch:  31 usd:  17

Nov  7 14:52:17 Largerock-squid kernel: Node 0 Normal per-cpu:

Nov  7 14:52:17 Largerock-squid kernel: CPU    0: hi:  186, btch:  31 usd: 148

Nov  7 14:52:17 Largerock-squid kernel: CPU    1: hi:  186, btch:  31 usd:  59

Nov  7 14:52:17 Largerock-squid kernel: CPU    2: hi:  186, btch:  31 usd:  64

Nov  7 14:52:17 Largerock-squid kernel: CPU    3: hi:  186, btch:  31 usd:  34

Nov  7 14:52:17 Largerock-squid kernel: CPU    4: hi:  186, btch:  31 usd:  21

Nov  7 14:52:17 Largerock-squid kernel: CPU    5: hi:  186, btch:  31 usd:  22

Nov  7 14:52:17 Largerock-squid kernel: CPU    6: hi:  186, btch:  31 usd:  50

Nov  7 14:52:17 Largerock-squid kernel: CPU    7: hi:  186, btch:  31 usd:  55

Nov  7 14:52:17 Largerock-squid kernel: active_anon:3847274 inactive_anon:2365500 isolated_anon:0

Nov  7 14:52:17 Largerock-squid kernel: active_file:77 inactive_file:0 isolated_file:32

Nov  7 14:52:17 Largerock-squid kernel: unevictable:0 dirty:1 writeback:0 unstable:0

Nov  7 14:52:17 Largerock-squid kernel: free:41667 slab_reclaimable:28547 slab_unreclaimable:280933

Nov  7 14:52:17 Largerock-squid kernel: mapped:2365635 shmem:2365544 pagetables:48036 bounce:0

Nov  7 14:52:17 Largerock-squid kernel: Node 0 DMA free:15548kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15156kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Nov  7 14:52:17 Largerock-squid kernel: lowmem_reserve[]: 0 3000 26482 26482

Nov  7 14:52:17 Largerock-squid kernel: Node 0 DMA32 free:101504kB min:7652kB low:9564kB high:11476kB active_anon:1681480kB inactive_anon:47812kB active_file:8kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:47836kB shmem:47812kB slab_reclaimable:18296kB slab_unreclaimable:591000kB kernel_stack:56kB pagetables:3520kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? yes

Nov  7 14:52:17 Largerock-squid kernel: lowmem_reserve[]: 0 0 23482 23482

Nov  7 14:52:17 Largerock-squid kernel: Node 0 Normal free:49616kB min:59892kB low:74864kB high:89836kB active_anon:13707616kB inactive_anon:9414188kB active_file:300kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:24046080kB mlocked:0kB dirty:4kB writeback:0kB mapped:9414704kB shmem:9414364kB slab_reclaimable:95892kB slab_unreclaimable:533024kB kernel_stack:1648kB pagetables:188624kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:458 all_unreclaimable? yes

Nov  7 14:52:17 Largerock-squid kernel: lowmem_reserve[]: 0 0 0 0

Nov  7 14:52:17 Largerock-squid kernel: Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 2*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15548kB

Nov  7 14:52:17 Largerock-squid kernel: Node 0 DMA32: 23974*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 102048kB

Nov  7 14:52:17 Largerock-squid kernel: Node 0 Normal: 11163*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 48748kB

Nov  7 14:52:17 Largerock-squid kernel: 2365580 total pagecache pages

Nov  7 14:52:17 Largerock-squid kernel: 0 pages in swap cache

Nov  7 14:52:17 Largerock-squid kernel: Swap cache stats: add 0, delete 0, find 0/0

Nov  7 14:52:17 Largerock-squid kernel: Free swap  = 0kB

Nov  7 14:52:17 Largerock-squid kernel: Total swap = 0kB

Nov  7 14:52:17 Largerock-squid kernel: 6881264 pages RAM

Nov  7 14:52:17 Largerock-squid kernel: 147804 pages reserved

Nov  7 14:52:17 Largerock-squid kernel: 4911130 pages shared

Nov  7 14:52:17 Largerock-squid kernel: 4272662 pages non-shared

Nov  7 14:52:17 Largerock-squid kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name

Nov  7 14:52:17 Largerock-squid kernel: [  500]     0   500     2737      179   5     -17         -1000 udevd

Nov  7 14:52:17 Largerock-squid kernel: [  921]     0   921     2662      107   6     -17         -1000 udevd

Nov  7 14:52:17 Largerock-squid kernel: [  922]     0   922     2662      107   7     -17         -1000 udevd

Nov  7 14:52:17 Largerock-squid kernel: [ 1230]     0  1230    23294       68   4     -17         -1000 auditd

Nov  7 14:52:17 Largerock-squid kernel: [ 1254]     0  1254     1540       24   4       0             0 portreserve

Nov  7 14:52:17 Largerock-squid kernel: [ 1261]     0  1261    63856      235   1       0             0 rsyslogd

Nov  7 14:52:17 Largerock-squid kernel: [ 1290]     0  1290     2738       62   6       0             0 irqbalance

Nov  7 14:52:17 Largerock-squid kernel: [ 1304]    32  1304     4744       62   6       0             0 rpcbind

Nov  7 14:52:17 Largerock-squid kernel: [ 1322]    29  1322     5837      114   1       0             0 rpc.statd

Nov  7 14:52:17 Largerock-squid kernel: [ 1432]    81  1432     5351       71   2       0             0 dbus-daemon

Nov  7 14:52:17 Largerock-squid kernel: [ 1448]     0  1448    47336      223   4       0             0 cupsd

Nov  7 14:52:17 Largerock-squid kernel: [ 1473]     0  1473     1020       27   5       0             0 acpid

Nov  7 14:52:17 Largerock-squid kernel: [ 1482]    68  1482     9476      234   1       0             0 hald

Nov  7 14:52:17 Largerock-squid kernel: [ 1483]     0  1483     5082       54   4       0             0 hald-runner

Nov  7 14:52:17 Largerock-squid kernel: [ 1512]     0  1512     5612       41   5       0             0 hald-addon-inpu

Nov  7 14:52:17 Largerock-squid kernel: [ 1530]    68  1530     4484       36   4       0             0 hald-addon-acpi

Nov  7 14:52:17 Largerock-squid kernel: [ 1551]     0  1551    96434      124   1       0             0 automount

Nov  7 14:52:17 Largerock-squid kernel: [ 1567]     0  1567     1570       32   4       0             0 mcelog

Nov  7 14:52:17 Largerock-squid kernel: [ 1576]     0  1576    49460      955   5       0             0 snmpd

Nov  7 14:52:17 Largerock-squid kernel: [ 1588]     0  1588    16655      177   4     -17         -1000 sshd

Nov  7 14:52:17 Largerock-squid kernel: [ 1664]     0  1664    20326      226   6       0             0 master

Nov  7 14:52:17 Largerock-squid kernel: [ 1677]    89  1677    20389      221   0       0             0 qmgr

Nov  7 14:52:17 Largerock-squid kernel: [ 1688]     0  1688    27580       35   1       0             0 abrtd

Nov  7 14:52:17 Largerock-squid kernel: [ 1703]     0  1703    29324      152   2       0             0 crond

Nov  7 14:52:17 Largerock-squid kernel: [ 1717]   500  1717  5166698    14730   4       0             0 squid

Nov  7 14:52:17 Largerock-squid kernel: [ 1719]   500  1719  5168582    21614   0       0             0 squid

Nov  7 14:52:17 Largerock-squid kernel: [ 1721]   500  1721  6119473  2180266   6       0             0 squid

Nov  7 14:52:17 Largerock-squid kernel: [ 1722]   500  1722  6136601  2165217   4       0             0 squid

Nov  7 14:52:17 Largerock-squid kernel: [ 1723]   500  1723  6252323  2575690   4       0             0 squid

Nov  7 14:52:17 Largerock-squid kernel: [ 1724]   500  1724  5950985  1737617   0       0             0 squid

Nov  7 14:52:17 Largerock-squid kernel: [ 1742]     0  1742     5385       46   1       0             0 atd

Nov  7 14:52:17 Largerock-squid kernel: [ 1754]     0  1754    15587       89   4       0             0 certmonger

Nov  7 14:52:17 Largerock-squid kernel: [ 1770]     0  1770     1016       18   4       0             0 mingetty

Nov  7 14:52:17 Largerock-squid kernel: [ 1772]     0  1772     1016       18   3       0             0 mingetty

Nov  7 14:52:17 Largerock-squid kernel: [ 1774]     0  1774     1016       19   6       0             0 mingetty

Nov  7 14:52:17 Largerock-squid kernel: [ 1776]     0  1776     1016       19   2       0             0 mingetty

Nov  7 14:52:17 Largerock-squid kernel: [ 1778]     0  1778     1016       17   6       0             0 mingetty

Nov  7 14:52:17 Largerock-squid kernel: [ 1780]     0  1780     1016       19   5       0             0 mingetty

Nov  7 14:52:17 Largerock-squid kernel: [ 7149]    89  7149    20346      220   4       0             0 pickup

Nov  7 14:52:17 Largerock-squid kernel: [ 7295]     0  7295    25095      264   6       0             0 sshd

Nov  7 14:52:17 Largerock-squid kernel: [ 7297]     0  7297    27085       90   7       0             0 bash

Nov  7 14:52:17 Largerock-squid kernel: [ 7316]     0  7316    28210      211   0       0             0 htop

Nov  7 14:52:17 Largerock-squid kernel: [ 7339]     0  7339    27085       91   5       0             0 bash

Nov  7 14:52:17 Largerock-squid kernel: [ 7360]     0  7360    25224       31   6       0             0 tailf

Nov  7 14:52:17 Largerock-squid kernel: [ 7361]     0  7361    27085       93   3       0             0 bash

Nov  7 14:52:17 Largerock-squid kernel: [ 7380]     0  7380    27057      220   4       0             0 nano

Nov  7 14:52:17 Largerock-squid kernel: [ 7383]     0  7383    27085       89   5       0             0 bash

Nov  7 14:52:17 Largerock-squid kernel: [ 7402]     0  7402    27590      226   0       0             0 iptraf

Nov  7 14:52:17 Largerock-squid kernel: [ 7412]     0  7412    27085       91   4       0             0 bash

Nov  7 14:52:17 Largerock-squid kernel: [ 7431]     0  7431    25224       30   5       0             0 tailf

Nov  7 14:52:17 Largerock-squid kernel: Out of memory: Kill process 1723 (squid) score 384 or sacrifice child

Nov  7 14:52:17 Largerock-squid kernel: Killed process 1723, UID 500, (squid) total-vm:25009292kB, anon-rss:4356424kB, file-rss:5946336kB

 

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users

[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux