Search squid archive

Re: Sudden but sustained high bandwidth usage

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



skyrocketing = using our maximum link download bandwidth.
This machine is only proxying. Not being a firewall, not a router, nor a gateway. It has access to the internet through our gateway/firewall (pfsense).
Lots of LAN clients are connected to the proxy, this is their only way to the internet. 1 interface, debian linux. EXT4 FS. CPU/MEM usage is always stable.
Clients use it explicitly or via wpad. Never transparently. Now I'm using 3 workers, because 1 is not enough and we have spare cores.
It's a VM machine with netapp storage. lots of raid disks.
SQUID was running perfectly without cache_dirs.

I think squid is downloading and redownloading the same files over and over again because: 1- these are segmented downloads and range_offset_limit is set to NONE for these files. 2- it can't store the downloaded files on the cache but I don't know why!

1457358960.737   8399 10.23.0.63 TCP_SWAPFAIL_MISS/206 1058138 GET http://au.download.windowsupdate.com/c/msdownload/update/software/uprl/2013/10/ie11-windows6.1-x64-en-us_ddec9ddc256ffa7d97831af148f6cc45130c6857.exe - HIER_DIRECT/201.30.251.43 application/octet-stream
1457358987.869  22464 10.88.10.5 TCP_SWAPFAIL_MISS/206 1416417 GET http://au.download.windowsupdate.com/d/msdownload/update/software/secu/2016/02/ie11-windows6.1-kb3141092-x64_0f7a98b9dc9f5c7ac73f6f543bf004a15e4d7be8.psf - HIER_DIRECT/201.30.251.26 application/octet-stream

Squid Object Cache: Version 3.5.15-20160301-r13999
Build Info:
Service Name: squid
Start Time:     Fri, 04 Mar 2016 21:15:47 GMT
Current Time:   Mon, 07 Mar 2016 18:14:37 GMT
Connection information for squid:
        Number of clients accessing cache:      4543
        Number of HTTP requests received:       7305505
        Number of ICP messages received:        0
        Number of ICP messages sent:    0
        Number of queued ICP replies:   0
        Number of HTCP messages received:       0
        Number of HTCP messages sent:   0
        Request failure ratio:   0.00
        Average HTTP requests per minute since start:   1765.1
        Average ICP messages per minute since start:    0.0
        Select loop called: 365044476 times, 11.146 ms avg
Cache information for squid:
        Hits as % of all requests:      5min: 22.0%, 60min: 21.2%
        Hits as % of bytes sent:        5min: 3.9%, 60min: 7.3%
        Memory hits as % of hit requests:       5min: 2.5%, 60min: 2.9%
        Disk hits as % of hit requests: 5min: 28.1%, 60min: 26.8%
        Storage Swap size:      179951936 KB
        Storage Swap capacity:  45.1% used, 54.9% free
        Storage Mem size:       4194176 KB
        Storage Mem capacity:   100.0% used,  0.0% free
        Mean Object Size:       46.07 KB
        Requests given to unlinkd:      0
Median Service Times (seconds)  5 min    60 min:
        HTTP Requests (All):   0.06514  0.07007
        Cache Misses:          0.09730  0.10075
        Cache Hits:            0.00055  0.00055
        Near Hits:             0.06521  0.06757
        Not-Modified Replies:  0.00055  0.00055
        DNS Lookups:           0.00019  0.00019
        ICP Queries:           0.00000  0.00000
Resource usage for squid:
        UP Time:        248329.183 seconds
        CPU Time:       21525.705 seconds
        CPU Usage:      8.67%
        CPU Usage, 5 minute avg:        52.35%
        CPU Usage, 60 minute avg:       47.90%
        Maximum Resident Size: 85809792 KB
        Page faults with physical i/o: 104
Memory accounted for:
        Total accounted:       164356 KB
        memPoolAlloc calls: 1801863396
        memPoolFree calls:  1811836496
File descriptor usage for squid:
        Maximum number of file descriptors:   81920
        Largest file desc currently in use:   6157
        Number of file desc currently in use: 8216
        Files queued for open:                   0
        Available number of file descriptors: 73704
        Reserved number of file descriptors:   500
        Store Disk files open:                  11
Internal Data Structures:
         25054 StoreEntries
           391 StoreEntries with MemObjects
         96086 Hot Object Cache Items
        3905959 on-disk objects

Best Regards,

-- 
Heiler Bemerguy - (91) 98151-4894
Assessor Técnico - CINBESA (91) 3184-1751

Em 07/03/2016 14:33, Eliezer Croitoru escreveu:
On 07/03/2016 16:29, Heiler Bemerguy wrote:
We're still getting all these SWAPFAIL and our link is
skyrocketing...... please help! I think it didn't happen on older
versions (.14 and below)

Hey,

What do you mean by skyrocketing?? like in the graph??
Also it is not clear to me something about the machine, is this machine a FW\ROUTER\GW?
If so is it for a lan?
How many interface this machine has?
Is it a pfsense? if so what version?
What is the FS used for the cache directories?
Did you also measured CPU when you see the spikes? if so what is it?
How clients access the proxy service? transparently or using a browser setings or WPAD with dhcp settings?
Also I have seen you are using 2 workers, is it because one worker doesn't seem to do the job?
Did you tried to change the values of:
cache_swap_low 98
cache_swap_high 99

from this high to lower numbers such as:
cache_swap_low 90
cache_swap_high 95

or even lower?
cache_swap_low 80
cache_swap_high 85

I am unsure about this since in the docs at:
http://www.squid-cache.org/Doc/config/cache_swap_low/
http://www.squid-cache.org/Doc/config/cache_swap_high/

the ROCK storage is not mentioned.

Also on what hardware are you running? what disks?

All the above are important and in your case it is possible that there is something wrong in how the network is planned or the software doing something wrong.

In scenarios like this I offer to verify two things:
- test what happens when you disable disk cache.(from CPU, bandwidth, DISK aspect)
- dump the cache manager info page to see basic statistics about the proxy traffic using: http:/cache_ip_or_visiblie_host_name:3128/squid-internal-mgr/info

For scenarios like this I started working on a logging\monitoring service\script that will run in the background of the machine and will dump content of some statistics to enable couple squid developers eyes to see these and then have a better understanding the nature of the issue.
For now the script\service is not ready and will not be able to help us so we need these dumps and information..

Eliezer
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users

[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux