I've got more information (on the FreeBSD side):
The packets are coming in over the GRE interface, but seem to be
randomly disappearing after the IPFW forward operation (forwards to
localhost:3128).
Here's the ipfw config:
00150 fwd 127.0.0.1,3128 tcp from any to any dst-port 80 via gre0 in
00250 fwd 127.0.0.1,3128 ip from any to any via gre0 in
00350 fwd 127.0.0.1,3128 tcp from any to any dst-port 80 in via gre0
00450 fwd 127.0.0.1,3128 tcp from any to any dst-port 80 via gre1 in
00550 fwd 127.0.0.1,3128 ip from any to any via gre1 in
00650 fwd 127.0.0.1,3128 tcp from any to any dst-port 80 in via gre1
00750 fwd 127.0.0.1,3128 tcp from any to any dst-port 80 via gre2 in
00850 fwd 127.0.0.1,3128 ip from any to any via gre2 in
00950 fwd 127.0.0.1,3128 tcp from any to any dst-port 80 in via gre2
01050 fwd 127.0.0.1,3128 log logamount 5 tcp from any to any dst-port 80
via gre3 in
01150 fwd 127.0.0.1,3128 ip from any to any via gre3 in
01250 fwd 127.0.0.1,3128 tcp from any to any dst-port 80 in via gre3
01350 fwd 127.0.0.1,3128 log logamount 5 tcp from any to any dst-port 80 in
01450 allow ip from any to any
I've tested this with a single test machine, and there's lots of
browsing issues. Most sites return "document contains no data" errors.
From the ipfw log (this appears right before the browser error message):
Jan 24 12:27:08 squidc-1 kernel: ipfw: 1050 Forward to 127.0.0.1:3128
TCP 66.146.207.75:65250 63.245.209.11:80 in via gre3
I took packet dumps of the communication over the ethernet and gre3
interfaces, and the result is that the data comes in, gets forwarded,
and disappears (seems to never even reach squid). I even turned on full
debugging in Squid, and nothing appeared in the logs for erratic web
requests. There's also no consistency to which requests return errors;
it's completely random.
I can send you the packet dumps if you'd like.
Ryan Thoryk
Ryan Thoryk wrote:
Adrian Chadd wrote:
It might be an IOS release issue then. You may need to upgrade to some
more recent?
If you're really nice then I can load that IOS version on the 7200 I have
here. Let me know the output of "show ver" and I'll see what I can do.
Here's the detailed version strings (of the 4th router):
IOS (tm) 7200 Software (C7200-P-M), Version 12.2(27), RELEASE SOFTWARE
(fc3)
ROM: System Bootstrap, Version 11.1(13)CA, EARLY DEPLOYMENT RELEASE
SOFTWARE (fc1)
BOOTLDR: 7200 Software (C7200-BOOT-M), Version 12.0(10)S, EARLY
DEPLOYMENT RELEASE SOFTWARE (fc1)
The 2nd squid machine gets redirects from a cisco switch, and is
still running without any noticeable problems. Also the reports I
currently have show that the problems were happening with users on
the 4th router, but I have no way of verifying that it was just the 4th.
Which platform/IOS version? That switch will be doing L2 redirect.
The switch is a 3550, and will be replaced soon:
Cisco IOS Software, C3550 Software (C3550-IPSERVICESK9-M), Version
12.2(25)SEE1, RELEASE SOFTWARE (fc1)
ROM: Bootstrap program is C3550 boot loader
"Some people report problems with WCCP and IOS 12.x. They see
truncated or fragmented GRE packets arriving at the cache. Apparently
it works if you disable Cisco Express Forwarding for the interface"
Well, its for some IOS releases. The trick here is to realise the GRE MTU
will be smaller than the ethernet MTU, so you need to make certain that
all packets that'll come to you over GRE will fit unfragmented inside it.
You do this by ensuring the MSS the Squid box negotiates on its incoming
and outgoing TCP connections is less than the default (1460); something
like 1360 would definitely bypass the GRE packet size issue.
Hmm interesting. Tomorrow I'll try to get a web browser running on one
of the networks behind the 4th router, and try some random web browsing
before and after those changes. I increased the default MSS on the
squid box to 1460 from (I think) 500 (or whatever the FreeBSD 6.2
default is) in that previous try. The both the em0 interface and the
router's ethernet interface have an MTU of 1500. Also I just realized
we were trying to previously adjust the tunnel MTU according to the MTU
of an unused tunnel interface on the first router.
Its also potentially a topology related issue. Its hard to tell without
a diagram and set of configs. :)
Well if you need more info, then I'll see what I can get.
Ryan Thoryk
Adrian
