Strange Interaction between Squid and Facebook

Patrick Blair - Peapod <patrick.blair@xxxxxxxxx> · Thu, 29 Oct 2015 14:44:25 -0500

Hi All,

I apologize
 for the length of this post, but I'm really at my wits' end and am 
completely out of ideas as to how I might fix this or why this is 
happening.

Background and Architecture:
We are 
using Squid as our user internet access proxy, it is performing 
authentication via LDAP if a site other than what is on the allowed list
 of domains is accessed. We are running the proxy at our secondary 
datacenter and routing all user internet traffic out through the links 
on that end, so that user traffic does not "conflict" with our main 
website traffic at our primary datacenter, as we don't yet have separate
 circuits for each type of traffic. We have hundreds of users accessing 
the proxy on a daily basis.
We were running squid 3.3.12 on 
Solaris 10 11/06 SPARC, with a fairly basic configuration, I can provide
 it if necessary, but everything was working "fine" as far as we were 
able to tell (no complaints from the users). The only problem was that 
the physical hardware is very old and needed to be retired, as well as 
being a box we couldn't easily add more resources to.

So I 
set up a new proxy, running Centos 7.1 x86_64 on a VM (in VMware ESXi) 
with the latest Squid release (3.5.10) built from source, with a 
configuration along the lines of http://wiki.squid-cache.org/ConfigExamples/SmpCarpCluster. We moved to a VM so we would have greater flexibility and be able to add more resources easily if the server became stressed.
It
 worked very well in our testing, and there were no real issues that we 
were able to see, so we rolled it out to replace the old proxy.

Since
 then, everything has been working fine, apart from one site, Facebook, 
not loading correctly. It varies based on the particular browser 
accessing it, but some/most of the style sheets or content don't appear 
to load correctly, resulting in a very mangled look. Running the 
developer tools for each respective browser give us information that the
 connections to the stylesheets or content are aborted. Most of this 
content is hosted on fbcdn.net, which we've made sure is on our allowed domains list. 
What
 is interesting is that the first object (or multiple ones) is retrieved
 successfully through the proxy, but subsequent ones are denied. Again, 
this appears to vary depending on browser/os combinations, with Chrome 
on OSX and IE (on Windows) seeming to work the best.

We are NOT 
running a Tproxy or any other sort of intercepting proxy, all the 
clients are explicitly aware of the proxy's existence through a .pac 
configuration file pushed out through Group Policy.

I've tried 
disabling ssl_bump (which shouldn't be enabled anyway) for the facebook 
domains, setting cache deny for those domains, and setting always direct
 for those domains, none of which has had any effect.

I've also 
tried reverting to a more "simple" config, even the exact config that we
 were using on the "old" Squid that was working on Solaris, but that too
 fails. I've also changed from using Squid 3.5.10 to the version 
packaged by CentOS (squid-3.3.8-12.el7_0.x86_64), tried this with both 
configurations to no avail.

The only thing that has worked is 
setting up a "test" squid at our primary datacenter, same 
configurations, but this one does work. We've checked and verified that 
there are no custom routes or any other network configurations that vary
 between the servers, only IP addresses. Both are on unrestricted vlans 
that allow direct access to the internet. We are checking with our 
networking team to see if there is any custom routing that is in place 
on their end, but it's very doubtful that is the case.

I believe 
I've covered everything here, I can provide any other information or 
configurations if necessary (I didn't provide those here because of the 
length already). If anyone out there has encountered this issue, I would
 GREATLY appreciate any information or troubleshooting assistance you 
could provide.

Best Regards,

Pat Blair
Sr. Unix Administrator
Peapod, LLC
pblair@xxxxxxxxxx

This email and any attachments may contain information that is proprietary,
confidential and/or privileged and for the sole use of the intended recipients(s)
only.
If you are not the intended recipient, please notify the sender by return
email and delete all copies of this email and any attachments. Ahold and/or its
subsidiaries shall neither be liable for the inaccurate or incomplete transmission
of the information contained in this email or any attachments, nor for any delay
in its receipt. To the extent this email is intended to create any legal obligation,
the obligation shall bind only the contracting entity and not any other entity within
the Ahold Group.

This email and any attachments may contain information that is proprietary,
confidential and/or privileged and for the sole use of the intended recipients(s)
only.
If you are not the intended recipient, please notify the sender by return
email and delete all copies of this email and any attachments. Ahold and/or its
subsidiaries shall neither be liable for the inaccurate or incomplete transmission
of the information contained in this email or any attachments, nor for any delay
in its receipt. To the extent this email is intended to create any legal obligation,
the obligation shall bind only the contracting entity and not any other entity within
the Ahold Group.
_______________________________________________
squid-users mailing list
squid-users@xxxxxxxxxxxxxxxxxxxxx
http://lists.squid-cache.org/listinfo/squid-users