I have attached strace to Squid and waited until the download rate has decreased to 500 KB/sec.
I used "cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240".
Here is the download link:
http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace
I hope it can help you.
Gesendet: Donnerstag, 23. Juli 2015 um 13:29 Uhr
Von: "Marcus Kool" <marcus.kool@xxxxxxxxxxxxxxx>
An: "Jens Offenbach" <wolle5050@xxxxxx>, "Eliezer Croitoru" <eliezer@xxxxxxxxxxxx>, "Amos Jeffries" <squid3@xxxxxxxxxxxxx>, squid-users@xxxxxxxxxxxxxxxxxxxxx
Betreff: Re: Squid3: 100 % CPU load during object caching
Von: "Marcus Kool" <marcus.kool@xxxxxxxxxxxxxxx>
An: "Jens Offenbach" <wolle5050@xxxxxx>, "Eliezer Croitoru" <eliezer@xxxxxxxxxxxx>, "Amos Jeffries" <squid3@xxxxxxxxxxxxx>, squid-users@xxxxxxxxxxxxxxxxxxxxx
Betreff: Re: Squid3: 100 % CPU load during object caching
I am not sure if it is relevant, maybe it is:
I am developing an ICAP daemon and after the ICAP server sends a "100 continue"
Squid sends the object to the ICAP server in small chunks of varying sizes:
4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
Note that the interval of receiving the chunks is 1/1000th of a second.
It seems that Squid forwards the object to the ICAP server every time it receives
one or a few TCP packets.
I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening:
Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive.
Amos, is there a debug setting that can confirm/reject this suspicion?
Marcus
On 07/23/2015 04:25 AM, Jens Offenbach wrote:
> A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very confusing results.
>
> I cleared the cache:
> rm -rf /var/cache/squid3/*
> squid -z
> squid
> http_proxy=http://139.2.57.120:3128/ wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar
>
> The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec.
>
> During caching progress, we have 100 % CPU usage and a disk IO that is corresponds with the download rate.
>
> For further investigations I give you my build properties:
> squid -v
> Squid Cache: Version 3.5.6
> Service Name: squid
> configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-e
s
i' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security'
>
>
> Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr
> Von: "Eliezer Croitoru" <eliezer@xxxxxxxxxxxx>
> An: squid-users@xxxxxxxxxxxxxxxxxxxxx
> Betreff: Re: Squid3: 100 % CPU load during object caching
> On 22/07/2015 21:59, Eliezer Croitoru wrote:
>> Hey Jens,
>>
>> I have tested the issue with LARGE ROCK and not AUFS or UFS.
>> Using squid or not my connection to the server is about 2.5 MBps (20Mbps).
>> Squid is sitting on an intel atom with SSD drive and on a HIT case the
>> download speed is more then doubled to 4.5 MBps(36Mbps).
>> I have not tried it with AUFS yet.
>
>
> And I must admit that AUFS beats rock cache with speed.
> I have tried rock with basic "cache_dir rock /var/spool/squid 8000" vs
> "cache_dir aufs /var/spool/squid 8000 16 256" and the aufs cache HIT
> results more then doubles 3 the speed rock gave with default settings.
>
> So about 15MBps which is 120Mbps.
> I do not seem to feel what Jens feels but the 100% CPU might be because
> of spinning disk hangs while reading the file from disk.
>
> Amos, I remember that there were some suggestions how to tune large rock.
> Any hints?
> I can test it and make it a suggestion for big files.
>
> Eliezer
>
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> http://lists.squid-cache.org/listinfo/squid-users
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> http://lists.squid-cache.org/listinfo/squid-users
>
I am developing an ICAP daemon and after the ICAP server sends a "100 continue"
Squid sends the object to the ICAP server in small chunks of varying sizes:
4095, 5813, 1448, 4344, 1448, 1448, 2896, etc.
Note that the interval of receiving the chunks is 1/1000th of a second.
It seems that Squid forwards the object to the ICAP server every time it receives
one or a few TCP packets.
I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening:
Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive.
Amos, is there a debug setting that can confirm/reject this suspicion?
Marcus
On 07/23/2015 04:25 AM, Jens Offenbach wrote:
> A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very confusing results.
>
> I cleared the cache:
> rm -rf /var/cache/squid3/*
> squid -z
> squid
> http_proxy=http://139.2.57.120:3128/ wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar
>
> The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec.
>
> During caching progress, we have 100 % CPU usage and a disk IO that is corresponds with the download rate.
>
> For further investigations I give you my build properties:
> squid -v
> Squid Cache: Version 3.5.6
> Service Name: squid
> configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-e
s
i' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security'
>
>
> Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr
> Von: "Eliezer Croitoru" <eliezer@xxxxxxxxxxxx>
> An: squid-users@xxxxxxxxxxxxxxxxxxxxx
> Betreff: Re: Squid3: 100 % CPU load during object caching
> On 22/07/2015 21:59, Eliezer Croitoru wrote:
>> Hey Jens,
>>
>> I have tested the issue with LARGE ROCK and not AUFS or UFS.
>> Using squid or not my connection to the server is about 2.5 MBps (20Mbps).
>> Squid is sitting on an intel atom with SSD drive and on a HIT case the
>> download speed is more then doubled to 4.5 MBps(36Mbps).
>> I have not tried it with AUFS yet.
>
>
> And I must admit that AUFS beats rock cache with speed.
> I have tried rock with basic "cache_dir rock /var/spool/squid 8000" vs
> "cache_dir aufs /var/spool/squid 8000 16 256" and the aufs cache HIT
> results more then doubles 3 the speed rock gave with default settings.
>
> So about 15MBps which is 120Mbps.
> I do not seem to feel what Jens feels but the 100% CPU might be because
> of spinning disk hangs while reading the file from disk.
>
> Amos, I remember that there were some suggestions how to tune large rock.
> Any hints?
> I can test it and make it a suggestion for big files.
>
> Eliezer
>
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> http://lists.squid-cache.org/listinfo/squid-users
> _______________________________________________
> squid-users mailing list
> squid-users@xxxxxxxxxxxxxxxxxxxxx
> http://lists.squid-cache.org/listinfo/squid-users
>
_______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
