Search squid archive

Re: Prefetching HTTP GET requests, but under HTTPS

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thank you Amos for the detailed reply!

I'm only going to prefetch specific pages (using regexp matching for
url patterns) that are pretty static and I'll check squid-prefetch.

Best,
Jianshi

On Tue, May 6, 2014 at 5:04 PM, Amos Jeffries <squid3@xxxxxxxxxxxxx> wrote:
> On 6/05/2014 5:53 p.m., Jianshi Huang wrote:
>> Hi,
>>
>> I need to build a prefetching proxy to speedup page loading/clicks and
>> I'm currently investigating Squid for prototype. Websites I want to
>> speedup are all under HTTPS.
>>
>> I briefly scanned Squid's document and google the keywords, looks like
>> I need to do the following setup:
>>
>> 1) Use SSL-Bump (and install Squid's cert in client's machine)
>
> Yes this is the only way to get around the HTTPS "problem".
>
>> 2) Two Squid setup, one runs prefetching script, another one does the caching.
>
> Any reason for that design?
>  Prefetching only needs three components: The cache (Squid), the logic
> deciding what to fetch (script), and possibly a database of past info to
> inform those decisions.
>
> Check out the squidclient tool we provide for making arbitrary web
> requests. It is the best tool around for scripting web requests, similar
> levels of control to libcurl which is (probably) the best for use in
> compiled code.
>
>
>> Does that make sense?
>
> Pre-fetching is a very old idea based around metrics from very old
> protocols such as HTTP/1.0 where the traffic was static, predictable and
> prefetching makes a fair bit of sense.
>
> However there are several popular features built into HTTP/1.1 protocol
> which greatly alter that balance. Dynamic content with variants makes it
> far less static. Response negotiation makes the responses far more
> unpredictable. Persistent connections greatly reduce the lag times.
> Revalidation reduces the bandwidth costs. Together these all make
> prefetching in HTTP/1.1 a much less beneficial operation than most of
> the literature makes it seem.
>
> Whether it makes sense depends entirly on where it is being installed,
> what the traffic is like, how and why the prefetching decisions are
> being made. Only you can really answer those and it may actually take
> doing to figure out whether it was a bad choice to begin with.
>
>
>> Is there a better solution?
>
> At the current point of Internet development I believe throwing efforts
> into assisting us with HTTP/2 development would be more beneficial. But
> I am somewhat biased being a member of the HTTPbis WG and seeking to get
> Squid HTTP/2 support off the ground.
>
>
>> Or has anybody done similar things?
>
> We get a fairly regular flow of questions from people wanting to do
> pre-fetching. They all hit that above issues eventually and drop out of
> sight.
>
> This thread summarizes teh standard problems and answers:
> http://arstechnica.com/civis/viewtopic.php?f=16&t=1204579
> (see fandingo's answer near the bottom)
>
> I am aware of this tool being used by more than a few dozen
> installations, although its popularity does seem to be in decline:
> https://packages.debian.org/unstable/web/squid-prefetch
>
>
>>
>> It would be great if someone could point out some
>> configuration/scripting files to me. Code speaks :)
>>
>
> Everythig in this regard is situation dependent. The above are likely to
> be the best you can find. People who actually get it going (apparently
> anyway) keep the secrets of how to avoid the HTTP/1.1 issues pretty close.
>
> HTH
> Amos
>



-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/




[Index of Archives]     [Linux Audio Users]     [Samba]     [Big List of Linux Books]     [Linux USB]     [Yosemite News]

  Powered by Linux