OK I have kept your suggestion in my mind, but right now I'm not in a position to buy two HDD's. May be I can afford to buy 15 days later. For the time being, my prime problem is the loading of two major sites from where my users download mp3. Those are
www.music.com.bd and www.djmaza.com
Don't know the reason, but music.com.bd loads very slow. And in firebug i see that the problem persists while loading 3 ads from ads.clicksor.com and some facebook widgets. Can you please check and try to load these two domains if you're running a Squid 3.1.X version and see if everything is alright from your end.
Regards,
Saiful
----------------------------------------
> Date: Mon, 24 Jan 2011 11:41:04 +0200
> From: eliezer@xxxxxxxxxxxxxxxxxx
> To: squid-users@xxxxxxxxxxxxxxx
> Subject: Re: Some pages loading very slow in 3.1.10 Stable
>
> are you kdding me?
>
>
> 500 clients?
>
> if most of the clients are just doing almost nothing just downloading
> one page of 2 MB..
>
> how much is your HD I\O ?? in this case?(in speed MBps) ?
>
> so first.. change the UFS to AUFS you dont need to do anything to the
> cache it self cause it the same system just with Async options on.
> and just buy two more of these 500GB drives and put all three of them in
> raid 0 or 5.
> or first try 0 on two and then add another one to then if it goes smoothly.
> this will give you a lot more speed.
> i dont now the cause but it looks like or connectivity/dns or I\O problem.
>
> i'v tried to use iptraf but ifstat just gives you the numbers of every
> Interface you have in aginst out traffic meter in a simple way.
>
>
> On 24/01/2011 11:02, Saiful Alam wrote:
>
> >
> >>>> TRIED AUFS, but didn't get better performance, while
> >>>>
> > researching in the web, I read everywhere that AUFS is better than UFS
> > in terms of performance,but I don't know why I get bad performance with
> > this.
> > Processor is Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz
> > Motherboard is Intel® Desktop Board DH55TC
> >
> >
> >
> > RAM is 2x4GB=8GB DDR3
> > DISK 1 = 250GB Hitachi SATA HDP72502 {[USED ONLY FOR UBUNTU SYSTEM& BOOT]}
> > DISK 2 = 500GB Hitachi SATA HDS72105 {[USED FOR CACHE DIRECTORIES ONLY, MOUNTED ON /MEDIA/CACHE FILESYSTEM EXT4]}
> >
> > Normally my Disk I/O never goes more than 15% and I would say the average is about 3-4%.
> > For bandwidth monitoring I usually see iptraf which is also good, but surely I'll try ifstat next time.
> > At peak hours (which is between 10pm - 2am GMT +0600), we have around 500 clients connected (approx)
> >
> > I have tried apt-get install squid3 (which is the default 3.1.6 in apt repository) and found the performance of 3.1.10 (my custom configuration) is better than the 3.1.6.
> >
> > Regards,
> > Saiful
> >
> >
> > ----------------------------------------
> >
> >> Date: Mon, 24 Jan 2011 10:36:14 +0200
> >> From: eliezer@xxxxxxxxxxxxxxxxxx
> >> To: squid-users@xxxxxxxxxxxxxxx
> >> Subject: Re: Some pages loading very slow in 3.1.10 Stable
> >>
> >> It's a small peace and most of the answers are not really suppose to there.
> >>
> >>
> >> the first thing is that your cache is not just cache it's a store house..
> >>
> >> it;'s not bad but you can try to change the ufs to aufs..
> >>
> >> can get better performance.
> >>
> >> what are the specification of the machine?
> >>
> >> core i7 ?.... ram? what disk? array of disks?
> >>
> >> did you tried to ping from the machine or WGET?
> >>
> >> if it's debian you can install ifstat that can give you real-rime
> >> bandwidth usage and it might be cause of something else that is not
> >> related to squid..
> >>
> >> if you have even like 25 clients downloading obsessively mp3 files for
> >> like 10 or more minutes in this time your I\O of your hard drive will rise..
> >>
> >> also..
> >>
> >> you can monitor the access to the squid folders onfly while you have the
> >> problem and to understand what is causing it..
> >>
> >> if it's CPU load or DISK I\O load .. or other stuff.
> >>
> >> by the way.. can you try the ubuntu squid3 stable?
> >>
> >> im using squid3 stable on ubuntu 10.04 on an Intel Atom D450 machine
> >> with cache of 40GB and it's taking the load very nicely.
> >>
> >>
> >>
> >>
> >> On 24/01/2011 09:12, Saiful Alam wrote:
> >>
> >>
> >>> Some results of TCPDUMP in -vv mode.
> >>>
> >>> 13:12:04.191180 IP (tos 0x0, ttl 127, id 2750, offset 0, flags [DF], proto TCP (6), length 40)
> >>> 172.16.80.2.1155> 77.67.29.42.www: Flags [.], cksum 0x6de4 (correct), seq 1127903567, ack 4192021369, win 64700, length 0
> >>> 13:12:04.192822 IP (tos 0x0, ttl 64, id 4692, offset 0, flags [DF], proto TCP (6), length 823)
> >>> www-12-02-snc5.facebook.com.www> 10.16.63.123.3714: Flags [P.], cksum 0xc789 (correct), seq 4235393899:4235394682, ack 3661185770, win 14600, length 783
> >>> 13:12:04.831628 IP (tos 0x0, ttl 127, id 29487, offset 0, flags [DF], proto TCP (6), length 40)
> >>> 172.16.80.6.rmiregistry> cds174.lax.llnw.net.www: Flags [.], cksum 0xfa45 (correct), seq 0, ack 460801, win 65535, length 0
> >>> 13:12:05.355300 IP (tos 0x0, ttl 127, id 2265, offset 0, flags [DF], proto TCP (6), length 40)
> >>> 192.168.96.165.57362> 195.122.152.64.www: Flags [.], cksum 0xfc6c (correct), seq 1875916651, ack 4162436021, win 64240, length 0
> >>> 13:12:05.355322 IP (tos 0x0, ttl 64, id 18785, offset 0, flags [DF], proto TCP (6), length 1500)
> >>> 195.122.152.64.www> 192.168.96.165.57362: Flags [.], cksum 0x0f8f (correct), seq 4381:5841, ack 0, win 6432, length 1460
> >>> 13:12:05.355328 IP (tos 0x0, ttl 127, id 8325, offset 0, flags [DF], proto TCP (6), length 40)
> >>> 172.16.112.124.cvspserver> 8.27.4.126.www: Flags [.], cksum 0xfe6f (correct), seq 0, ack 662461, win 65535, length 0
> >>> 13:12:05.655764 IP (tos 0x0, ttl 127, id 5393, offset 0, flags [DF], proto TCP (6), length 40)
> >>> 172.16.64.10.2308> co63.reliablehosting.com.www: Flags [.], cksum 0x1b43 (correct), seq 63164814, ack 3610294926, win 65535, length 0
> >>> 13:12:05.655789 IP (tos 0x0, ttl 64, id 65532, offset 0, flags [DF], proto TCP (6), length 1500)
> >>> co63.reliablehosting.com.www> 172.16.64.10.2308: Flags [P.], cksum 0x4e5f (correct), seq 1461:2921, ack 0, win 17184, length 1460
> >>> 13:12:05.655798 IP (tos 0x0, ttl 64, id 65533, offset 0, flags [DF], proto TCP (6), length 1215)
> >>> co63.reliablehosting.com.www> 172.16.64.10.2308: Flags [P.], cksum 0xa23e (correct), seq 2921:4096, ack 0, win 17184, length 1175
> >>> 13:12:05.656066 IP (tos 0x0, ttl 126, id 42247, offset 0, flags [DF], proto TCP (6), length 73)
> >>> 172.16.30.6.1575> 79-110-88-28.gfsrv.net.15103: Flags [P.], cksum 0x207b (correct), seq 22088:22121, ack 55870, win 64475, length 33
> >>> 13:12:05.656081 IP (tos 0x0, ttl 126, id 42248, offset 0, flags [DF], proto TCP (6), length 51)
> >>> 172.16.30.6.1575> 79-110-88-28.gfsrv.net.15103: Flags [P.], cksum 0xd9a7 (correct), seq 22121:22132, ack 55870, win 64475, length 11
> >>> 13:12:05.949960 IP (tos 0x0, ttl 126, id 5909, offset 0, flags [DF], proto TCP (6), length 40)
> >>> 172.16.10.9.1408> GGCpmo1.pal.seabone.net.www: Flags [.], cksum 0xfe1a (correct), seq 3933402590, ack 133923280, win 65535, length 0
> >>> 13:12:05.949983 IP (tos 0x0, ttl 126, id 31989, offset 0, flags [DF], proto TCP (6), length 52)
> >>> 172.16.52.6.62360> c-n180-u0282-208.webazilla.com.www: Flags [.], cksum 0x0bbb (correct), seq 0, ack 1119485, win 64240, options [nop,nop,sack 1 {1120945:1129705}], length 0
> >>> 13:12:05.949997 IP (tos 0x0, ttl 127, id 1202, offset 0, flags [DF], proto TCP (6), length 40)
> >>> 172.16.200.42.1454> 146.82.204.207.www: Flags [.], cksum 0x5a1a (correct), seq 0, ack 1965427, win 65375, length 0
> >>> 13:12:05.950985 IP (tos 0x0, ttl 126, id 36156, offset 0, flags [DF], proto TCP (6), length 40)
> >>> 172.16.30.7.1795> cds17.sin.llnw.net.www: Flags [.], cksum 0x779d (correct), seq 4079975438, ack 3825116412, win 65535, length 0
> >>> 13:12:05.951132 IP (tos 0x0, ttl 126, id 5910, offset 0, flags [DF], proto TCP (6), length 996)
> >>> 172.16.10.9.1408> GGCpmo1.pal.seabone.net.www: Flags [P.], cksum 0xf7ba (correct), seq 0:956, ack 1, win 65535, length 956
> >>> 13:12:05.951152 IP (tos 0x0, ttl 64, id 20092, offset 0, flags [DF], proto TCP (6), length 40)
> >>> GGCpmo1.pal.seabone.net.www> 172.16.10.9.1408: Flags [.], cksum 0xdc7e (correct), seq 1, ack 956, win 7648, length 0
> >>> 13:12:05.955120 IP (tos 0x0, ttl 127, id 2284, offset 0, flags [DF], proto TCP (6), length 40)
> >>> 192.168.96.165.57362> 195.122.152.64.www: Flags [.], cksum 0xda34 (correct), seq 0, ack 8761, win 64240, length 0
> >>> 13:12:05.955148 IP (tos 0x0, ttl 64, id 18790, offset 0, flags [DF], proto TCP (6), length 972)
> >>> 195.122.152.64.www> 192.168.96.165.57362: Flags [P.], cksum 0x02b4 (correct), seq 8761:9693, ack 0, win 6432, length 932
> >>> 13:12:05.955605 IP (tos 0x0, ttl 126, id 20667, offset 0, flags [DF], proto TCP (6), length 40)
> >>> ^C 172.16.106.3.49956> 213.254.248.146.www: Flags [.], cksum 0x696e (correct), seq 348423304, ack 3320774375, win 16425, length 0
> >>>
> >>> 226 packets captured
> >>> 107740 packets received by filter
> >>> 107433 packets dropped by kernel
> >>>
> >>>
> >>> Regards,
> >>> Saiful
> >>>
> >>> ----------------------------------------
> >>>
> >>>
> >>>> From: saifulmr@xxxxxxxxxxx
> >>>> To: rmkml@xxxxxxx
> >>>> CC: squid-users@xxxxxxxxxxxxxxx
> >>>> Date: Mon, 24 Jan 2011 18:06:02 +1100
> >>>> Subject: RE: Some pages loading very slow in 3.1.10 Stable
> >>>>
> >>>>
> >>>> Thanks for your prompt reply. Well I am not much experienced admin in linux or in terms of squid, and therefore I havent installed wireshark/tshark/tcpdump in squid yet, but I will install it now to go in deep.
> >>>>
> >>>> My previous version of squid was 2.7 Stable downloaded from aptitude which was installed in a different machine, and not in this new machine. My current configuration in the current PC (Core i7) is a squid 3.1.10 custom compiled by me with only the ZPH and the Linux Net Filter enabled. I haven't used any other parameters while configuring it.
> >>>>
> >>>> Default Cache_Dir is splitted into three directories:
> >>>> UFS /media/cache/storage501 51400 16 256
> >>>> UFS /media/cache/storage502 51400 16 256
> >>>> UFS /media/cache/storage503 51400 16 256
> >>>>
> >>>> and cache_mem is 150MB
> >>>>
> >>>> Sometimes during peak hours while the network load is high, we do experience very slow browsing (in all websites) which is usually resolved after a quick restart of the squid process/service.
> >>>>
> >>>> Thanks& Best Regards,
> >>>> Shishir
> >>>>
> >>>> ----------------------------------------
> >>>>
> >>>>
> >>>>> Date: Mon, 24 Jan 2011 07:48:12 +0100
> >>>>> From: rmkml@xxxxxxx
> >>>>> To: saifulmr@xxxxxxxxxxx
> >>>>> CC: rmkml@xxxxxxx
> >>>>> Subject: Re: Some pages loading very slow in 3.1.10 Stable
> >>>>>
> >>>>> Hi Saiful,
> >>>>> On same plateform, do you have a old squid version working better? what is your previous squid version please?
> >>>>> can you start network capture like wireshark/tshark/tcpdump on your squid please? and run firebug for example and compare last and your previous squid version please?
> >>>>> can you compare squid cpu usage last and previous version please?
> >>>>> Regards
> >>>>> Rmkml
> >>>>>
> >>>>>
> >>>>> On Mon, 24 Jan 2011, Saiful Alam wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Hello Everyone,
> >>>>>> My Squid Configuration is pretty much default, except
> >>>>>> the fact, that I have added some refresh_patterns myself and collected
> >>>>>> from internet in order to get more hits. The server is a Squid3.1.10 (in
> >>>>>> 3128 intercept transparent) running on Ubuntu 10.10 and comprises of
> >>>>>> Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz. I would say that
> >>>>>> everything is fine, except some pages are loading very very slow. For
> >>>>>> example, www.music.com.bd / www.djmaza.com ... I will try to give more
> >>>>>> examples as I find more, but for the time being these two sites are very
> >>>>>> important for my clients. Don't understand actually whats wrong. If
> >>>>>> this has anything to do with the squid.conf file, I will attach it with
> >>>>>> the next reply. For the time being here's my refresh_patterns.
> >>>>>>
> >>>>>> refresh_pattern ^http://.*\.facebook.com.*\.(jpg|gif|png|swf|wav|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern ^http://.*\.fbcdn.net.*\.(jpg|gif|png|swf|wav|mp3|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern ^http://.*\.yimg.com.*\.(jpg|gif|png|swf|wav|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern ^http://.*\.ytimg.com.*\.(jpg|gif|png|swf|wav|mp3|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern ^http://.*\.gstatic.com.*\.(jpg|gif|png) 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern ^http://.*\.google.com.*\.(jpg|gif|png) 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern dsebd\.org.*\.(jpg|gif|png) 600 50% 3000
> >>>>>> refresh_pattern windowsupdate.com/.*\.(cab|exe) 4320 100% 43200 reload-into-ims
> >>>>>> refresh_pattern download.microsoft.com/.*\.(cab|exe) 4320 100% 43200 reload-into-ims
> >>>>>> refresh_pattern ^http://.*\.cnn\.com 1000 50% 4320 override-lastmod
> >>>>>> refresh_pattern ^http://news\.bbc\.co\.uk 1000 50% 4320 override-lastmod
> >>>>>> refresh_pattern microsoft 1080 150% 10080 override-lastmod
> >>>>>> refresh_pattern msn\.com 4320 150% 10080 override-lastmod
> >>>>>> refresh_pattern ^http://.*FIDO 360 1000% 480
> >>>>>> refresh_pattern \.r[0-9][0-0]$ 10080 150% 40320
> >>>>>> refresh_pattern ^http://.*\.asis$ 1440 50% 20160
> >>>>>> refresh_pattern -i \.art$ 10080 150% 43200
> >>>>>> refresh_pattern -i \.qtm$ 10080 150% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.viv$ 10080 150% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.pdf$ 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.jpg$ 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.png$ 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.gif$ 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.flv$ 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.ico$ 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.jpeg$ 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.x-flv$ 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.swf$ 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.wav$ 10080 300% 40320 reload-into-ims
> >>>>>> refresh_pattern -i \.mp3$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.mp4$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.zip$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.rar$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.exe$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.zip$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.doc$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.ppt$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.tiff$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.ram$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.rm$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.rpm$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.htm$ 1440 50% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.html$ 1440 50% 40320 override-expire override-lastmod reload-into-ims ignore-reload
> >>>>>> refresh_pattern -i \.txt$ 1440 100% 20160 reload-into-ims override-lastmod
> >>>>>> refresh_pattern -i \.arj$ 2880 200% 40320
> >>>>>> refresh_pattern -i \.Z$ 10080 200% 40320
> >>>>>> refresh_pattern ^ftp: 1440 50% 10080
> >>>>>> refresh_pattern ^gopher: 1440 10% 1440
> >>>>>> refresh_pattern . 0 40% 4320
> >>>>>>
> >>>>>> negative_ttl 1 minute
> >>>>>> positive_dns_ttl 15 hour
> >>>>>> negative_dns_ttl 20 second
> >>>>>>
> >>>>>> Thanks& Best Regards,
> >>>>>> Saiful
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >
