"A Tale of TTL Troubles"
I was hired to implement VPN for a subnet. The owner has a /27 at his
home site, and he wanted to have the machines there answering BOTH on
those IP addresses and some addresses at a remote colocation provider.
Make sense? Not to me either. :( I think he's trying to fool his
customers into thinking he has a physical presence in the colocation
city. But it's his money, and a very interesting issue.
I set up a simple openvpn peer-mode tunnel. It mostly seems to work,
except return packets back through the VPN are giving me trouble.
Sorry, I'm not allowed to post details, so IP addresses are munged. :(
HOME_SITE_NET=x.x.x.0/27 [ 0-31 ]
HOME_DEF_GATEWAY=x.x.x.1 (I have no access to this router.)
HOME_VPN_ENDPOINT=x.x.x.4, a/k/a y.y.y.38
REMOTE_SITE_NET=y.y.y.0/24
(out of this I was told to bind .38 on the VPN peer and route .39
through .60 via the VPN. Sigh ... CIDR masks would have been nice.)
REMOTE_DEF_GATEWAY=y.y.y.1 (I have no access to this router either.)
REMOTE_VPN_ENDPOINT=y.y.y.37, a/k/a 192.168.255.37
(The latter is only used for the tunnel traffic.)
Routing on HOME_VPN_ENDPOINT :
[root@localhost ~]# ip route list
y.y.y.39 dev eth0 scope link
192.168.255.37 dev tun0 proto kernel scope link src y.y.y.38
y.y.y.40/29 dev eth0 scope link
y.y.y.48/28 dev eth0 scope link
x.x.x.0/27 dev eth0 proto kernel scope link src x.x.x.4
169.254.0.0/16 dev eth0 scope link
default via x.x.x.1 dev eth0
[root@localhost ~]# ip route list table colo
default via 192.168.255.37 dev tun0 src y.y.y.38
[root@localhost ~]# ip rule list
0: from all lookup local
32763: from y.y.y.48/28 lookup colo
32764: from y.y.y.40/29 lookup colo
32765: from y.y.y.39 lookup colo
32766: from all lookup main
32767: from all lookup default
The 3 "dev eth0 scope link" routes are what was needed to cover .39
through .60. It goes over, to .63, oh well.
One of the machines is x.x.x.6 and y.y.y.39, .49, .59, and .60 on the
VPN. It's my guinea pig, hereinafter called "www". CentOS release 4.4
(Final) 2.6.9-42.ELsmp on each of www and HOME_VPN_ENDPOINT; I think
REMOTE_VPN_ENDPOINT is similar, but I don't think it's the problem.
[root@www ~]# ip route list
y.y.y.0/27 dev eth0 proto kernel scope link src y.y.y.6
169.254.0.0/16 dev eth0 scope link
default via y.y.y.1 dev eth0
[root@www ~]# ip rule list
0: from all lookup local
32765: from x.x.x.32/27 lookup vpn
32766: from all lookup main
32767: from all lookup default
[root@www ~]# ip route list table vpn
default via y.y.y.4 dev eth0
I tested the routing by using my own IP in an iptables -j LOG rule,
followed by an nmap -sP of the range.
[root@localhost ~]# iptables-save
# Generated by iptables-save v1.2.11 on Mon Dec 11 09:10:27 2006
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:LogVpn - [0:0]
-A FORWARD -d y.y.y.32/255.255.255.224 -j LogVpn
-A FORWARD -s y.y.y.32/255.255.255.224 -j LogVpn
-A LogVpn -s my.IP.add.ress -j LOG --log-prefix "5Vpn IN: "
-A LogVpn -d my.IP.add.ress -j LOG --log-prefix "5Vpn OUT: "
COMMIT
# Completed on Mon Dec 11 09:10:27 2006
Results of the scan: all but y.y.y.39 were up. Logs: um, rather than ask
you to pick through all that, I'll summarise:
[root@localhost ~]# dmesg | grep ^5Vpn
...
.39 worked perfectly. The ping and HTTP requests came IN=tun0 and
OUT=eth0; the replies came IN=eth0 and OUT=tun0. The inbound ICMP had
TTL=39. The inbound HTTP had TTL=33, a difference of 6.
.49 seemed to have a routing loop, and the "www" host apparently didn't
answer the ARP queries. We only got "5Vpn IN:" packets, none "OUT:".
Each packet passed through 3 times, with ICMP TTL values of 34, 22 and
10, and HTTP TTL values of 26, 14, and 2. Interestingly, each time the
IN and OUT interface was eth0. And coincidentally (?) it's 13 hops from
me to www's VPN IP, and 11 hops from www to REMOTE_VPN_ENDPOINT going
through HOME_DEF_GATEWAY.
.59 and .60 both worked, but the return packets must have gone through
HOME_DEF_GATEWAY, because both IN=eth0 and OUT=eth0. for each of the
"IN:" and "OUT:" packets. Inbound TTL values were 38 and 47 for ICMP,
and 34 and 40 for HTTP, on each of .59 and .60 respectively.
Meanwhile, over on www I set up similar log rules, and the results for
this scan pretty well aligned with the logs on HOME_VPN_ENDPOINT. Same
TTLs on inbound as noted above. Nothing was logged for .49 in or out,
again leading me to think that www ignored the ARP queries.
I wait awhile, try again, and I get different results. But never does
everything work right.
This is nuts!! Looks like sometimes www won't answer ARP for its
locally-bound IP addresses, and the rules on HOME_VPN_ENDPOINT are
being ignored. And how the VPN packets could possibly come in on eth0
is beyond me.
What am I missing here? What can I try?
One idea I had, was possibly to do proxy ARP on the HOME site too. Tell
the HOME machines to use 192.168.255.37 as the default route in their
vpn tables. That takes HOME_VPN_ENDPOINT routing rules out of the
equation.
Trying that now, similar intermittent results. ARRRRRRRRRGH!!
--
Offlist mail to this address is discarded unless
"/dev/rob0" or "not-spam" is in Subject: header
_______________________________________________
LARTC mailing list
LARTC@xxxxxxxxxxxxxxx
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
