I have found something out... Hopefully, it helps to reproduce and solve the issue. I got it working with a good download rate, but very high CPU usage on Squid 3.3.8 and Squid 3.5.6. There seems to be problem with large files that get cached on disk in combination with memory caching. When I use these settings, memory usage of Squid grows step-by-step with 100% CPU usage and 500 KB/sec download rate: # MEMORY CACHE OPTIONS # ----------------------------------------------------------------------------- maximum_object_size_in_memory 1 GB memory_replacement_policy heap LFUDA cache_mem 4 GB # DISK CACHE OPTIONS # ----------------------------------------------------------------------------- maximum_object_size 10 GB cache_replacement_policy heap GDSF cache_dir aufs /var/cache/squid3 25600 16 256 I decided to turn off memory caching completely and used the following settings: # MEMORY CACHE OPTIONS # ----------------------------------------------------------------------------- maximum_object_size_in_memory 0 GB memory_replacement_policy heap LFUDA cache_mem 0 GB # DISK CACHE OPTIONS # ----------------------------------------------------------------------------- maximum_object_size 10 GB cache_replacement_policy heap GDSF cache_dir aufs /var/cache/squid3 25600 16 256 Now, I get stable and high download rates even on a cache miss. @Marcus: Could you please post your squid.config Gesendet: Freitag, 24. Juli 2015 um 08:25 Uhr Von: "Jens Offenbach" <wolle5050@xxxxxx> An: squid-users@xxxxxxxxxxxxxxxxxxxxx Betreff: Re: Squid3: 100 % CPU load during object caching I have made a quick test of Squid 3.3.8 on Ubuntu 15.04 and I get the same problem: 100 % CPU usage, 500 KB/sec download rate. Gesendet: Freitag, 24. Juli 2015 um 07:54 Uhr Von: "Jens Offenbach" <wolle5050@xxxxxx> An: "Marcus Kool" <marcus.kool@xxxxxxxxxxxxxxx>, "Eliezer Croitoru" <eliezer@xxxxxxxxxxxx>, "Amos Jeffries" <squid3@xxxxxxxxxxxxx>, squid-users@xxxxxxxxxxxxxxxxxxxxx Betreff: Re: Squid3: 100 % CPU load during object caching It is not easy for me, but I have tested Squid 3.3.8 from the Ubuntu packaging on a "real" physical infrastructure. I get the same results on the physical machine (1x Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz, 32 GB RAM, 1 TB disk) where Squid is running: 100 % CPU usage, 500 KB/sec download rate. All machines are idle and we have 1 GBit ethernet. The strace log from the physical test scenario can be found here, but I think it does not differ from the "virtual" test scenario: http://wikisend.com/download/293856/squid.strace2 @Marcus: Have you verified that the file does not fit into memory and gets cached on disk? On which OS is Squid running? What are your build options of Squid (squid -v)? Is it possible that the issue is not part of 3.4.12? Do we have a regression? @Amos, Eliezer Is someone able to reproduce the disk caching effect? Regards, Jens Gesendet: Donnerstag, 23. Juli 2015 um 20:08 Uhr Von: "Marcus Kool" <marcus.kool@xxxxxxxxxxxxxxx> An: "Jens Offenbach" <wolle5050@xxxxxx>, "Amos Jeffries" <squid3@xxxxxxxxxxxxx>, "Eliezer Croitoru" <eliezer@xxxxxxxxxxxx>, squid-users@xxxxxxxxxxxxxxxxxxxxx Betreff: Re: Aw: Re: Squid3: 100 % CPU load during object caching The strace output shows this loop: Squid reads 16K-1 bytes from FD 13 webserver Squid writes 4 times 4K to FD 17 /var/cache/squid3/00/00/00000000 Squid writes 4 times 4K to FD 12 browser But this loop does not explain the 100% CPU usage... Does Squid do a buffer reshuffle when it reads 16K-1 and writes 16K ? I did the download test with Squid 3.4.12 AUFS on an idle system with a 500 mbit connection and 1 CPU with 4 cores @ 3.7 GHz. The first download used 35% of 1 CPU core with a steady download speed of 62 MB/sec. The second (cached) download used 50% of 1 CPU core with a steady download speed of 87 MB/sec. I never looked at Squid CPU usage and do not know what is reasonable but it feels high. With respect to the 100% CPU issue of Jens, one factor is that Squid runs in a virtual machine. Squid in a virtual machine cannot be compared with a wget test since Squid allocates a lot of memory that the host must manage. This is a possible explanation for the fact that you see the performance going down and up. Can you do the same test on the host (i.e. not inside a VM). Marcus On 07/23/2015 10:39 AM, Jens Offenbach wrote: > I have attached strace to Squid and waited until the download rate has decreased to 500 KB/sec. > I used "cache_dir aufs /var/cache/squid3 88894 16 256 max-size=10737418240". > Here is the download link: > http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace[http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace][http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace[http://w1.wikisend.com/node-fs/download/6a004a416f65b4cdf7f8eff4ff961199/squid.strace]] > I hope it can help you. > *Gesendet:* Donnerstag, 23. Juli 2015 um 13:29 Uhr > *Von:* "Marcus Kool" <marcus.kool@xxxxxxxxxxxxxxx> > *An:* "Jens Offenbach" <wolle5050@xxxxxx>, "Eliezer Croitoru" <eliezer@xxxxxxxxxxxx>, "Amos Jeffries" <squid3@xxxxxxxxxxxxx>, squid-users@xxxxxxxxxxxxxxxxxxxxx > *Betreff:* Re: Squid3: 100 % CPU load during object caching > I am not sure if it is relevant, maybe it is: > > I am developing an ICAP daemon and after the ICAP server sends a "100 continue" > Squid sends the object to the ICAP server in small chunks of varying sizes: > 4095, 5813, 1448, 4344, 1448, 1448, 2896, etc. > Note that the interval of receiving the chunks is 1/1000th of a second. > It seems that Squid forwards the object to the ICAP server every time it receives > one or a few TCP packets. > > I have a suspicion that in the scenario of 100% CPU, large #write calls and low throughput a similar thing is happening: > Squid physically stores a small part of the object many times, i.e. every time one or a few TCP packets arrive. > > Amos, is there a debug setting that can confirm/reject this suspicion? > > Marcus > > > On 07/23/2015 04:25 AM, Jens Offenbach wrote: > > A test with ROCK "cache_dir rock /var/cache/squid3 51200" gives very confusing results. > > > > I cleared the cache: > > rm -rf /var/cache/squid3/* > > squid -z > > squid > > http_proxy=http://139.2.57.120:3128/[http://139.2.57.120:3128/][http://139.2.57.120:3128/[http://139.2.57.120:3128/]][http://139.2.57.120:3128/[http://139.2.57.120:3128/][http://139.2.57.120:3128/[http://139.2.57.120:3128/]]] wget http://test-server/freesurfer-Linux-centos6_x86_64-stable-pub-v5.3.0.tar > > > > The download starts with 10 MB/sec and stays constant for 1 minutes, then it drops gradually to 1 MB/sec and stays there for some time. After 5 minutes the download rate returns back to 10 MB/sec > very quickly and drops again step-by-step to 1 MB/sec. After 5-6 minutes the download rates rises again to 10 MB/sec and drops again gradually to 1 MB/sec. > > > > During caching progress, we have 100 % CPU usage and a disk IO that is corresponds with the download rate. > > > > For further investigations I give you my build properties: > > squid -v > > Squid Cache: Version 3.5.6 > > Service Name: squid > > configure options: '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=/include' '--mandir=/share/man' '--infodir=/share/info' '--sysconfdir=/etc' '--localstatedir=/var' > '--libexecdir=/lib/squid3' '--srcdir=.' '--disable-maintainer-mode' '--disable-dependency-tracking' '--disable-silent-rules' '--datadir=/usr/share/squid3' '--sysconfdir=/etc/squid3' > '--mandir=/usr/share/man' '--enable-inline' '--enable-async-io=8' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-delay-pools' '--enable-cache-digests' > '--enable-underscores' '--enable-icap-client' '--enable-follow-x-forwarded-for' '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' > '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,smb_lm' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,unix_group,wbinfo_group' > '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-e > s > i' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/cache/squid3' '--with-logdir=/var/log/squid3' '--with-pidfile=/var/run/squid3.pid' > '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fPIE -fstack-protector > --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall' 'LDFLAGS=-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' 'CPPFLAGS=-D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fPIE > -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security' > > > > > > Gesendet: Mittwoch, 22. Juli 2015 um 21:47 Uhr > > Von: "Eliezer Croitoru" <eliezer@xxxxxxxxxxxx> > > An: squid-users@xxxxxxxxxxxxxxxxxxxxx > > Betreff: Re: Squid3: 100 % CPU load during object caching > > On 22/07/2015 21:59, Eliezer Croitoru wrote: > >> Hey Jens, > >> > >> I have tested the issue with LARGE ROCK and not AUFS or UFS. > >> Using squid or not my connection to the server is about 2.5 MBps (20Mbps). > >> Squid is sitting on an intel atom with SSD drive and on a HIT case the > >> download speed is more then doubled to 4.5 MBps(36Mbps). > >> I have not tried it with AUFS yet. > > > > > > And I must admit that AUFS beats rock cache with speed. > > I have tried rock with basic "cache_dir rock /var/spool/squid 8000" vs > > "cache_dir aufs /var/spool/squid 8000 16 256" and the aufs cache HIT > > results more then doubles 3 the speed rock gave with default settings. > > > > So about 15MBps which is 120Mbps. > > I do not seem to feel what Jens feels but the 100% CPU might be because > > of spinning disk hangs while reading the file from disk. > > > > Amos, I remember that there were some suggestions how to tune large rock. > > Any hints? > > I can test it and make it a suggestion for big files. > > > > Eliezer > > > > _______________________________________________ > > squid-users mailing list > > squid-users@xxxxxxxxxxxxxxxxxxxxx > > http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users][http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users]][http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users][http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users]]] > > _______________________________________________ > > squid-users mailing list > > squid-users@xxxxxxxxxxxxxxxxxxxxx > > http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users][http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users]][http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users][http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users]]] > > _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users][http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users]] _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users[http://lists.squid-cache.org/listinfo/squid-users] _______________________________________________ squid-users mailing list squid-users@xxxxxxxxxxxxxxxxxxxxx http://lists.squid-cache.org/listinfo/squid-users
