I am not sure if it is relevant, maybe it is:
I am developing an ICAP daemon and after the ICAP server sends a "100 continue"
Squid sends the object to the ICAP server in small chunks of varying sizes:
4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
Note that the interval of receiving the chunks is 1/1000th of a second.
It seems that Squid forwards the object to the ICAP server every time it receives
one or a few TCP packets.
I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening:
Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive.
Amos, is there a debug setting that can confirm/reject this suspicion?
Marcus
On 07/23/2015 04:25 AM, Jens Offenbach wrote:
A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very confusing results.
I cleared the cache:
rm -rf /var/cache/squid3/*
squid -z
squid
http_proxy=http://139.2.57.120:3128/ wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar
The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec.
During caching progress, we have 100 % CPU usage and a disk IO that is corresponds with the download rate.
For further investigations I give you my build properties:
squid -v
Squid Cache: Version 3.5.6
Service Name: squid
configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-e
s
i' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security'
Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr
Von: "Eliezer Croitoru" <eliezer@xxxxxxxxxxxx>
An: squid-users@xxxxxxxxxxxxxxxxxxxxx
Betreff: Re: Squid3: 100 % CPU load during object caching
On 22/07/2015 21:59, Eliezer Croitoru wrote:
Hey Jens,
I have tested the issue with LARGE ROCK and not AUFS or UFS.
Using squid or not my connection to the server is about 2.5 MBps (20Mbps).
Squid is sitting on an intel atom with SSD drive and on a HIT case the
download speed is more then doubled to 4.5 MBps(36Mbps).
I have not tried it with AUFS yet.
And I must admit that AUFS beats rock cache with speed.
I have tried rock with basic "cache_dir rock /var/spool/squid 8000" vs
"cache_dir aufs /var/spool/squid 8000 16 256" and the aufs cache HIT
results more then doubles 3 the speed rock gave with default settings.
So about 15MBps which is 120Mbps.
I do not seem to feel what Jens feels but the 100% CPU might be because
of spinning disk hangs while reading the file from disk.
Amos, I remember that there were some suggestions how to tune large rock.
Any hints?
I can test it and make it a suggestion for big files.
Eliezer
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users
