Re: Squid 3.5.21 - High CPU (100%)

[Jasper Van Der Westhuizen <jvdwesthuiz@xxxxxxxxxxxxxx>]

> > Is there anything that you guys can suggest I do around the cache?
> > Should I try a different store type? A different filesystem type perhaps?
>
>
> If your store has a configuration knob that effectively limits disk
> writing rate, then use it to limit that rate to avoid overflowing the queue.
>
> You can also consider using rock store that has an explicit write rate
> limiting option (but comes with other problems that may or may not
> affect your setup).
>
> Adding more physical disk spindles helps, of course.

Thanks Alex. My next approach is to add more spindles. In fact I am currently testing with 4 cache dirs. All aufs currently. After adding these cache dirs, Squid started crashing every 20mins or so..

--cut-- (Cache.log)

FATAL: Received Segment Violation...dying.
2016/09/26 15:54:54 kid1| Closing HTTP port 0.0.0.0:8080
2016/09/26 15:54:54 kid1| storeDirWriteCleanLogs: Starting...
2016/09/26 15:54:54 kid1|     65536 entries written so far.
2016/09/26 15:54:54 kid1|   Finished.  Wrote 98159 entries.
2016/09/26 15:54:54 kid1|   Took 0.05 seconds (2044425.47 entries/sec).
CPU Usage: 1230.157 seconds = 166.490 user + 1063.666 sys
Maximum Resident Size: 3535840 KB
Page faults with physical i/o: 0
2016/09/26 15:54:57 kid1| Set Current Directory to /var/cache/squid/
2016/09/26 15:54:57 kid1| Starting Squid Cache version 3.5.21 for x86_64-pc-linux-gnu...
2016/09/26 15:54:57 kid1| Service Name: squid
2016/09/26 15:54:57 kid1| Process ID 16054
2016/09/26 15:54:57 kid1| Process Roles: worker
2016/09/26 15:54:57 kid1| With 65536 file descriptors available
2016/09/26 15:54:57 kid1| Initializing IP Cache...
2016/09/26 15:54:57 kid1| DNS Socket created at 0.0.0.0, FD 7
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding domain xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding nameserver xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding nameserver xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Adding nameserver xxxxxx from /etc/resolv.conf
2016/09/26 15:54:57 kid1| Logfile: opening log daemon:/var/log/squid//access.log
2016/09/26 15:54:57 kid1| Logfile Daemon: opening log /var/log/squid//access.log
2016/09/26 15:54:57 kid1| Store logging disabled
2016/09/26 15:54:57 kid1| Swap maxSize 41943040 + 4194304 KB, estimated 3549026 objects
2016/09/26 15:54:57 kid1| Target number of buckets: 177451
2016/09/26 15:54:57 kid1| Using 262144 Store buckets
2016/09/26 15:54:57 kid1| Max Mem  size: 4194304 KB
2016/09/26 15:54:57 kid1| Max Swap size: 41943040 KB
2016/09/26 15:54:57 kid1| Rebuilding storage in /var/cache/squid-c1/ (clean log)
2016/09/26 15:54:57 kid1| Rebuilding storage in /var/cache/squid-c2/ (clean log)
2016/09/26 15:54:57 kid1| Rebuilding storage in /var/cache/squid-c3/ (clean log)
2016/09/26 15:54:57 kid1| Rebuilding storage in /var/cache/squid-c4/ (clean log)
2016/09/26 15:54:57 kid1| Using Least Load store dir selection
2016/09/26 15:54:57 kid1| Set Current Directory to /var/cache/squid/
2016/09/26 15:54:57 kid1| Finished loading MIME types and icons.
2016/09/26 15:54:57 kid1| HTCP Disabled.
2016/09/26 15:54:57 kid1| Sending SNMP messages from 0.0.0.0:3401
2016/09/26 15:54:57 kid1| Configuring Parent xxxxxx8/80/0
2016/09/26 15:54:57 kid1| Configuring Parent xxxxxx/80/0
2016/09/26 15:54:57 kid1| Configuring Sibling xxxxxx/8080/3131
2016/09/26 15:54:57 kid1| Squid plugin modules loaded: 0
2016/09/26 15:54:57 kid1| Adaptation support is off.
2016/09/26 15:54:57 kid1| Accepting HTTP Socket connections at local=0.0.0.0:8080 remote=[::] FD 18 flags=9
2016/09/26 15:54:57 kid1| Accepting SNMP messages on 0.0.0.0:3401
2016/09/26 15:54:57 kid1| Store rebuilding is 33.00% complete
2016/09/26 15:54:57 kid1| Done reading /var/cache/squid-c3/ swaplog (8660 entries)
2016/09/26 15:54:57 kid1| Done reading /var/cache/squid-c1/ swaplog (12119 entries)
2016/09/26 15:54:58 kid1| Done reading /var/cache/squid-c2/ swaplog (30684 entries)
2016/09/26 15:54:58 kid1| Done reading /var/cache/squid-c4/ swaplog (46696 entries)
2016/09/26 15:54:58 kid1| Finished rebuilding storage from disk.
2016/09/26 15:54:58 kid1|     98159 Entries scanned
2016/09/26 15:54:58 kid1|         0 Invalid entries.
2016/09/26 15:54:58 kid1|         0 With invalid flags.
2016/09/26 15:54:58 kid1|     98159 Objects loaded.
2016/09/26 15:54:58 kid1|         0 Objects expired.
2016/09/26 15:54:58 kid1|         0 Objects cancelled.
2016/09/26 15:54:58 kid1|         0 Duplicate URLs purged.
2016/09/26 15:54:58 kid1|         0 Swapfile clashes avoided.
2016/09/26 15:54:58 kid1|   Took 0.61 seconds (162197.49 objects/sec).
2016/09/26 15:54:58 kid1| Beginning Validation Procedure
2016/09/26 15:54:58 kid1|   Completed Validation Procedure
2016/09/26 15:54:58 kid1|   Validated 98159 Entries
2016/09/26 15:54:58 kid1|   store_swap_size = 11495920.00 KB
2016/09/26 15:54:58 kid1| storeLateRelease: released 0 objects

--cut--

 I tried rock but was stuck with this error for a while and decided to go back to aufs until I can do some more reading up. 

FATAL: Rock cache_dir at /var/cache/squid-c3//rock failed to open db file: (2) No such file or directory

I am currently testing again after recompiling squid..  The rest of my proxy cluster has the "cache deny all" configuration directive.. So nothing is cached.

Disclaimer:
http://www.shopriteholdings.co.za/Pages/ShopriteE-mailDisclaimer.aspx

_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users