Re: Update Accelerator, Squid and Windows Update Caching

Regardt van de Vyver <squid@xxxxxxxxxxxxxxx> · Thu, 23 Oct 2008 15:26:59 +0200

Amos Jeffries wrote:
Richard Wall wrote:
On Fri, Oct 10, 2008 at 12:30 PM, Amos Jeffries 
<squid3@xxxxxxxxxxxxx> wrote:
Richard Wall wrote:
Hi,

I've been reading through the archive looking for information about
squid 2.6 and windows update caching. The FAQ mentions problems with
range offsets but it's not really clear which versions of Squid this
applies to.
All versions. The FAQ was the result of my experiments mid last 
year. With
some tweaks made early his year since Vista came out.
We haven't done a intensive experiments with Vista yet.

Hi Amos,

I'm still investigating Windows Update caching (with 2.6.STABLE17/18)

First of all, I have been doing some tests to try and find out the
problem with Squid and Content-Range requests.
 * I watch the squid logs as a vista box does its automatic updates
and I can see that *some* of its requests use ranges. (so far I have
only seen these when it requests .psf files...some of which seem to be
very large files...so the range request makes sense) See:
http://groups.google.hr/group/microsoft.public.windowsupdate/browse_thread/thread/af5db07dc2db9713 

# zcat squid.log.192.168.1.119.2008-10-16.gz | grep
"multipart/byteranges" | awk '{print $7}' | uniq | while read URL; do
echo $URL; wget --spider $URL 2>&1 | grep Length; done
http://www.download.windowsupdate.com/msdownload/update/software/secu/2008/10/windows6.0-kb956390-x86_2d03c4b14b5bad88510380c14acd2bffc26436a7.psf 

Length: 91,225,471 (87M) [application/octet-stream]
http://www.download.windowsupdate.com/msdownload/update/software/secu/2008/05/windows6.0-kb950762-x86_0cc2989b92bc968e143e1eeae8817f08907fd715.psf 

Length: 834,868 (815K) [application/octet-stream]
http://www.download.windowsupdate.com/msdownload/update/software/secu/2008/03/windows6.0-kb948590-x86_ed27763e42ee2e20e676d9f6aa13f18b84d7bc96.psf 

Length: 755,232 (738K) [application/octet-stream]
http://www.download.windowsupdate.com/msdownload/update/software/crup/2008/09/windows6.0-kb955302-x86_1e40fd3ae8f95723dbd76f837ba096adb25f3829.psf 

Length: 7,003,447 (6.7M) [application/octet-stream]
...

 * I have found that curl can make range requests so I've been using
it to test how Squid behaves....and it seems to do the right thing. eg
 - First ask for a range : The correct range is returned X-Cache: MISS
 - Repeat the range request :  The correct range is returned X-Cache: 
MISS
 - Request the entire file: The entire file is correctly returned 
X-Cache: MISS
 - Repeat the request: X-Cache: HIT
 - Repeat the previous range request: X-Cache: HIT
 - Request a different range: X-Cache: HIT

curl --range 1000-1002 --header Pragma: -v -x http://127.0.0.1:3128
http://www.download.windowsupdate.com/msdownload/update/software/secu/2008/05/windows6.0-kb950762-x86_0cc2989b92bc968e143e1eeae8817f08907fd715.psf 

/dev/null

Looking back through the archive I find this conversation from 2005:
http://www.squid-cache.org/mail-archive/squid-users/200504/0669.html

...but the behaviour there sounds like a result of setting:
range_offset_limit -1

Seems to me that Squid should do a good job of Windows Update caching.
There is another thread discussing how to override MS update cache
control headers:
http://www.squid-cache.org/mail-archive/squid-users/200508/0596.html

....but I don't see anything evil in the server response headers
today. I guess the client may be sending no-cache headers...I'll
double check that later.

Is there some other case that I'm missing?

As I said. I have not seen Vista in detail. I just had to turn off my 
old hack to get around the SP1 hanging. (that huge .psf perhapse?)
Never had to do anything with headers.

When I did my testing it was with outdated Win98-WinXP machines (often 
needing SP1 in XP's case).

The WU on them made an HTTPS request (seems to be auth-related even 
today) requested one or more update indexes fine. Then proceeded to 
random-access range requests out of the middle of the update *.cabs 
using dynamic urls at various update sites.
This was causing bandwidth blowout with all the MISS'es when I had 
several machines a week coming through.

I _think_ but have no confirmation, that the early patch-tuesday 
releases were done as large single .CAB files and a particular machine 
may only need updating from individual fixes inside them.

As your test showed, fetching the whole file squid can handle the 
ranges fine. It's when they are still in MISS state that ranges become 
trouble.

I'm going to experiment, but if anyone has any positive or
negative experience of Squid and windows update caching, I'd be really
interested to hear from you.

In case Squid cannot do windows update caching by its self, I'm also
looking at integrating Update Accelerator
(http://update-accelerator.advproxy.net/) script with standard squid
2.6 and wondered if anyone else had any experience of this.
The update accelerator script is just a perl wrapper around wget which
is configured as a Squid url_rewrite_program. It's not clear to me
what this script is doing that Squid wouldn't do by its self.
Strange indeed.

I got update accelerator working with Squid but I'm still not
convinced that it's necessary (see above).

-RichardW.

Hi Guys,

We've implemented our own 'flavour' of the cache accelerator for 
windows-updates and have been rather happy with it. Microsoft have 
gotten rather funky and are appending datetime stamps to the cab files 
in an attempt to reduce their cachability. If you strip the DT stamp and 
make a request the correct file is still fetched from the windows update 
sites.

I'm trying to move towards a combination of stroreurl_rewrite_program to 
try handle these files inside squid instead of externally.

If anyone feels they have windowsupdates working 100% without external 
rewriters their comments would be greatly appreciated.