Hi Firan,
Sorry to hear you have had such a bad experience with Squid. Some
comments inline ...
On 2/09/2013 10:13 a.m., Golden Shadow wrote:
I've tried squid previously only in a lab environment. Over the last 3 months though, I had the chance to try squid in a real environment, in which the TPROXY squid I installed receives around 25000 http requests per minute.
Unfortunately, I've concluded that if someone was to install squid in a real environment, there would be no specific guide that he can follow to avoid all the problems that are waiting for him on the way. A guide about what is the best hardware to choose,
... but there is no "best". Squid is designed to work on a *very* wide
range of hardware (and virtual machine) systems in an even more varied
set of environments, many of which have not even been designed or
invented yet (thanks to POSIX).
If you want advice you have only to outline what environment you are
trying to work with and throw it out for the rest of the community to
respond to. If anyone out there has past experience you may hear back,
but be aware that even so any response is only _past_ experience and
usually a personal opinion. What works best for a coffee shop is not
anywhere near what works best for a Tier-1 gateway between countries,
which is different again from a mobile network gateway.
Also, the number of experienced admin capable of dealing with large
scale networks on a debugging level seem to be getting rarer. (If this
whole cloud fad actually meets its marketing goal we will probably end
up with only one world-wide admin who has nobody else around to ask
advice from. Suck to be learning that job.).
the recommended configure options (there is no good documentation for most of them),
... except the configuration manual
(http://www.squid-cache.org/Doc/config/). If there are issues with any
option please point it out. We are constantly improving it.
The authoritative reference guide of that is the rather long
squid.conf.documented which should have been installed with your proxy.
how to implement a TPROXY using squid,
... http://wiki.squid-cache.org/Features/Tproxy4#Squid_Configuration.
That one sub-section of the page (one single 'tproxy' flag even) really
is the whole Squid part required to get TPROXY running ona basic proxy
or cache. Everything else listed on that page and elsewhere is
system-specific details, policy choices, and troubleshooting help. We
cannot decide for you what you want done.
You are one of the trailblazers here. TPROXY to my knowledge has not
been used before by such a busy proxy as 25k req/min - that said
problems are not expected any more there than in other traffic types at
that rate. Once you get down to specific issues with the TPROXY
behaviour it is time to talk to the netfilter (or BSD) kernel people
where 99% of the code actually is. Squid only does a normal accept() and
setsockopt() system call, nothing particularly special.
having to tune number of allowed file descriptors on the server, how to handle SYN floods, page faults,
... there are many ways to do such things even within a single operating
system. How much documentation do you expect the Squid community to
write and publish?
Places like stackoverflow exist for that purpose. Better that we focus
on improving *Squid* and not re-writing documentation already maintained
and kept updated by others.
FYI: there are only two of us "developers" part-timing as librarians for
all the thousands of pages of already existing documentation. We rely
heavily on people just like yourself submitting change requests, or
donating your own time to write new more relevant documents.
poor support of SMP,
.. ahem. This is experimental. Want better support you have several options:
http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F
FWIW: The Measurement Factory developers have done an outstanding job of
getting it as far as they have in only 2 years (and the work there
continues!). Re-writing a large event-base asynchronous software
codebase to be multi-process aware and sharing state in real time has
thrown off many experienced developers and scuttled multiple past
efforts. Forgive them for not having done more than a few miracles at once.
... and yes the documentation there is barely half written because
things are changing under the documentation writers feet. The many hacks
required to simply get it running are being fixed and polished up as
fast as possible - even as workarounds are being figured out then broken
by the real fix.
caching youtube and dynamic contents,
... which have a rather lot of documentation actually
(http://wiki.squid-cache.org/ConfigExamples/DynamicContent). Given that
when it comes to Squid there is little difference between a "dynamic"
and a "static" URL. They are both URL with a resource which can be
cached or not cached according to the rules of HTTP.
The latest Squid releases (since um, Feb 2010) cache dynamic content by
default.
good examples of refresh_patterns to start with
Sigh ... we tried that. The patterns written in long ago and published
in the old texts were missing things like modern query-strings and
fragments[1]. Those old rules also included rules forcing long-term
caching of things like TIFF images which were popular back then but now
almost exclusively used for high-resolution geobase and medical scans
(not things you want cached and shared around publicly) - and OCD people
might point out those same uses have existed back then as well, so the
idea "images are static and can be cached" has always been false during
Squids' lifetime.
[1] Which reminds me I still have to find the time to go through the
entire wiki and codebase checking for example regex patterns and
ensuring they can cope with #-fragments which are now accepted as
standard in any URL. Thanks.
are just some examples of what I had to run through without having a good guide that would proactively sheds the light on those things.
You came here with specific problems and got specific asnwers to those
problems when one existed. Did you ask for newbie help? or where to find
the documentation manual or wiki FAQ? I dont remember so.
Posting on this list and surfing the web looking for solutions on sometimes obsolete articles and then following the trial and error way of resolving problems were the only things I had. I was able to overcome some of the problems I
faced, sometimes on my own and sometimes with the help of kind members on this list. Other problems are still unresolved and have been embarrassing me with the party I have installed this squid for!
My sympathies. Please take this as a life lesson and learn not to make
promises if you are newbie with so many features.
If you will continue to post about the problems I/we will continue to do
what we can to assist. But please understand that it is limited what we
can do, you will have to investigate as much as you can either way and
apply whatever fixes can be found only after the finding. If the
problems were easy they would likely have been fixed by now.
I'm very disappointed to see my squid malfunctioning and producing logs that I could not find any support for neither online nor on this list just after I tuned max-swap-rate and swap-timeout for my rock stores, which I did trying to fix another problem! After forgetting about using rock stores, I got very disappointed again to see my squid crashing each once in a while just after I removed the old aufs cache_dirs and replaced them with new aufs cache_dirs according to a "more recommended" way. I'm also very disappointed to see squid continuously restarting and core dumping because the swap.state header is inconsistent with the cache contents whenever squid is improperly shutdown!
Er. Nothing we can do about your memory. Sorry.
Keeping that in mind (sorry about the pun), the help here is mostly from
my or others memories too. So testing and verifying our advice is always
worth the time.
I don't think I have the right to criticize such a big project like squid with its huge lines of code and the huge efforts its developers have exerted on so far.
Sure you can. Most of the developers are people just like you, or
perhapse with a little more coding bent who hit a problem and sent the
main team a patch or bug report.
I'm just writing this to express myself and to say that I think versions of squid which are said to be "stable" are far from being really stable, at least for beginners like me. I've always loved open source software and always thought that open source software is much more stable than their closed source counterparts, but unfortunately I did not have the same impression with squid! There must be a problem, or problems, somewhere! It's either weak documentation or perhaps using squid in some environments would require deep insights into the code of squid or maybe a long experience with squid and having to go though many problems and get disappointed many times if you don't have good support!
Ahh ... stable is not exactly what you seem to think it means. "stable"
in release terms means un-changing. The "stable" Squid only change by
around 100 lines of code between releases, although it does vary upward
a bit sometimes.
We do try to make it match the other run-time meaning as well, but can
only ensure the known bugs are gone. Sorry if you are only now using it
in a way never tested for. All I can say is that behaviour testing is
done by community volunteers so please consider joining in to test the
upcoming versions before most of the development team move on to 3.5 and
the fix changes slow down to a crawl.
Formally we make no guarantees other than the releases will build on at
least a handful of popular operating systems.
I will give squid a little more time but I think I will give up soon and advise the party I installed squid for to go for another, commercial, cache proxy.
As alway the choice is yours. Thank you for at least mentioning your
problems though. All too many peope just say "it dont work" and walk away.
Amos
