Hi Amos, Kinkie and Eliezer @Kinkie: It's very interesting to me to know about the companies that provide commercial support for squid on:(http://www.squid-cache.org/Support/services.html). Thanks for your reply and for providing this link. @Eliezer: Thanks for your reply. I'm already using Centos but yeah, I think I need to consider faster disk drives, perhaps SSDs. @Amos: Thanks a lot for your detailed explanation, you made several things clearer to me. By "configure options", I meant the options that should be used with the configure script that is run before "make". Is there any good documentation about their meanings and what options to use with what values based on a certain environment and/or hardware? Take for example --enable-async-io=, should I use it? What is the best value to use? I don't think there is good documentation about those configure options. As for: http://wiki.squid-cache.org/Features/Tproxy4 Although it takes care of most things a beginner would need to implement a TPROXY, but it looks summarized and in my opinion does not remove the confusion caused by the many old and obsolete articles about how to implement a TPROXY. When I first started, I was puzzled whether I should compile the kernel from source to use the TPROXY patches or just use the kernel that comes with my Centos 6.4 kernel. Moreover, the following section really confused me: Use DIVERT to prevent existing connections going through TPROXY twice: iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT Based on my understanding of this iptables rule, I think it is intended for reply packets coming from web servers to squid (with the spoofed IP address), right?! Saying that its purpose is to "prevent existing connections going through TPROXY twice" really confused me and I still can't understand what this means! I know there is a lot of documentation online about tuning file descriptors and similar things, and yeah I think it's not the job of squid documentation to talk about how to do those things. What I meant is that it would perhaps be much easier for newbies to be notified that they may need to tune file descriptors, that they may get SYN floods, that they may have page faults, and all other things that are waiting for them down the way, all in a single document. This will also reduce the number of duplicate questions on this list, I guess. Thanks once again and I really appreciate your support. Best regards, Firas >Hi Firan, > Sorry to hear you have had such a bad experience with Squid. Some >comments inline ... >On 2/09/2013 10:13 a.m., Golden Shadow wrote: >> I've tried squid previously only in a lab environment. Over the last 3 months though, I had the chance to try squid in a real environment, in which the TPROXY squid I installed receives around 25000 http requests per minute. >> >> Unfortunately, I've concluded that if someone was to install squid in a real environment, there would be no specific guide that he can follow to avoid all the problems that are waiting for him on the way. A guide about what is the best hardware to choose, ... but there is no "best". Squid is designed to work on a *very* wide range of hardware (and virtual machine) systems in an even more varied set of environments, many of which have not even been designed or invented yet (thanks to POSIX). If you want advice you have only to outline what environment you are trying to work with and throw it out for the rest of the community to respond to. If anyone out there has past experience you may hear back, but be aware that even so any response is only _past_ experience and usually a personal opinion. What works best for a coffee shop is not anywhere near what works best for a Tier-1 gateway between countries, which is different again from a mobile network gateway. Also, the number of experienced admin capable of dealing with large scale networks on a debugging level seem to be getting rarer. (If this whole cloud fad actually meets its marketing goal we will probably end up with only one world-wide admin who has nobody else around to ask advice from. Suck to be learning that job.). > the recommended configure options (there is no good documentation for most of them), ... except the configuration manual (http://www.squid-cache.org/Doc/config/). If there are issues with any option please point it out. We are constantly improving it. The authoritative reference guide of that is the rather long squid.conf.documented which should have been installed with your proxy. > how to implement a TPROXY using squid, ... http://wiki.squid-cache.org/Features/Tproxy4#Squid_Configuration. That one sub-section of the page (one single 'tproxy' flag even) really is the whole Squid part required to get TPROXY running ona basic proxy or cache. Everything else listed on that page and elsewhere is system-specific details, policy choices, and troubleshooting help. We cannot decide for you what you want done. You are one of the trailblazers here. TPROXY to my knowledge has not been used before by such a busy proxy as 25k req/min - that said problems are not expected any more there than in other traffic types at that rate. Once you get down to specific issues with the TPROXY behaviour it is time to talk to the netfilter (or BSD) kernel people where 99% of the code actually is. Squid only does a normal accept() and setsockopt() system call, nothing particularly special. > having to tune number of allowed file descriptors on the server, how to handle SYN floods, page faults, ... there are many ways to do such things even within a single operating system. How much documentation do you expect the Squid community to write and publish? Places like stackoverflow exist for that purpose. Better that we focus on improving *Squid* and not re-writing documentation already maintained and kept updated by others. FYI: there are only two of us "developers" part-timing as librarians for all the thousands of pages of already existing documentation. We rely heavily on people just like yourself submitting change requests, or donating your own time to write new more relevant documents. > poor support of SMP, .. ahem. This is experimental. Want better support you have several options: http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F FWIW: The Measurement Factory developers have done an outstanding job of getting it as far as they have in only 2 years (and the work there continues!). Re-writing a large event-base asynchronous software codebase to be multi-process aware and sharing state in real time has thrown off many experienced developers and scuttled multiple past efforts. Forgive them for not having done more than a few miracles at once. ... and yes the documentation there is barely half written because things are changing under the documentation writers feet. The many hacks required to simply get it running are being fixed and polished up as fast as possible - even as workarounds are being figured out then broken by the real fix. > caching youtube and dynamic contents, ... which have a rather lot of documentation actually (http://wiki.squid-cache.org/ConfigExamples/DynamicContent). Given that when it comes to Squid there is little difference between a "dynamic" and a "static" URL. They are both URL with a resource which can be cached or not cached according to the rules of HTTP. The latest Squid releases (since um, Feb 2010) cache dynamic content by default. > good examples of refresh_patterns to start with Sigh ... we tried that. The patterns written in long ago and published in the old texts were missing things like modern query-strings and fragments[1]. Those old rules also included rules forcing long-term caching of things like TIFF images which were popular back then but now almost exclusively used for high-resolution geobase and medical scans (not things you want cached and shared around publicly) - and OCD people might point out those same uses have existed back then as well, so the idea "images are static and can be cached" has always been false during Squids' lifetime. [1] Which reminds me I still have to find the time to go through the entire wiki and codebase checking for example regex patterns and ensuring they can cope with #-fragments which are now accepted as standard in any URL. Thanks. > are just some examples of what I had to run through without having a good guide that would proactively sheds the light on those things. You came here with specific problems and got specific asnwers to those problems when one existed. Did you ask for newbie help? or where to find the documentation manual or wiki FAQ? I dont remember so. > Posting on this list and surfing the web looking for solutions on sometimes obsolete articles and then following the trial and error way of resolving problems were the only things I had. I was able to overcome some of the problems I > faced, sometimes on my own and sometimes with the help of kind members on this list. Other problems are still unresolved and have been embarrassing me with the party I have installed this squid for! My sympathies. Please take this as a life lesson and learn not to make promises if you are newbie with so many features. If you will continue to post about the problems I/we will continue to do what we can to assist. But please understand that it is limited what we can do, you will have to investigate as much as you can either way and apply whatever fixes can be found only after the finding. If the problems were easy they would likely have been fixed by now. > I'm very disappointed to see my squid malfunctioning and producing logs that I could not find any support for neither online nor on this list just after I tuned max-swap-rate and swap-timeout for my rock stores, which I did trying to fix another problem! After forgetting about using rock stores, I got very disappointed again to see my squid crashing each once in a while just after I removed the old aufs cache_dirs and replaced them with new aufs cache_dirs according to a "more recommended" way. I'm also very disappointed to see squid continuously restarting and core dumping because the swap.state header is inconsistent with the cache contents whenever squid is improperly shutdown! Er. Nothing we can do about your memory. Sorry. Keeping that in mind (sorry about the pun), the help here is mostly from my or others memories too. So testing and verifying our advice is always worth the time. > > I don't think I have the right to criticize such a big project like squid with its huge lines of code and the huge efforts its developers have exerted on so far. Sure you can. Most of the developers are people just like you, or perhapse with a little more coding bent who hit a problem and sent the main team a patch or bug report. > I'm just writing this to express myself and to say that I think versions of squid which are said to be "stable" are far from being really stable, at least for beginners like me. I've always loved open source software and always thought that open source software is much more stable than their closed source counterparts, but unfortunately I did not have the same impression with squid! There must be a problem, or problems, somewhere! It's either weak documentation or perhaps using squid in some environments would require deep insights into the code of squid or maybe a long experience with squid and having to go though many problems and get disappointed many times if you don't have good support! Ahh ... stable is not exactly what you seem to think it means. "stable" in release terms means un-changing. The "stable" Squid only change by around 100 lines of code between releases, although it does vary upward a bit sometimes. We do try to make it match the other run-time meaning as well, but can only ensure the known bugs are gone. Sorry if you are only now using it in a way never tested for. All I can say is that behaviour testing is done by community volunteers so please consider joining in to test the upcoming versions before most of the development team move on to 3.5 and the fix changes slow down to a crawl. Formally we make no guarantees other than the releases will build on at least a handful of popular operating systems. > I will give squid a little more time but I think I will give up soon and advise the party I installed squid for to go for another, commercial, cache proxy. As alway the choice is yours. Thank you for at least mentioning your problems though. All too many peope just say "it dont work" and walk away. Amos
