I'm using iproute2-ss010824 and a 2.4.20 kernel.
Quite a subtle issue here, so I can imagine it has not been spotted before.
The setup:
I have set up a machine as a gateway. The 'external' interface uses a
dummy IP address (eg. 10.0.0.2) and the internal side is a normal
address. (it's more complex in real life using Zebra and such, but this is
the basic setup that shows the problem)
To make sure I can start connections to the outside world from this
machine I must make sure that the source address of the packets is from
the internal interface with the 'real' address range, or otherwise packets
won't come back as they would originate from the 10.0.0.x range.
I could not use iptables to translate/NAT the source address (other issues
there), but the 'scope' option in iproute2 seemed the best answer and indeed
it seems to work fine.
I have now set up the internal interface with a 'global' scope and
the external with a 'link' scope:
1: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 02:02:05:62:00:23 brd ff:ff:ff:ff:ff:ff
inet 17.70.0.1/28 brd 17.70.0.15 scope global eth0
2: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:30:48:27:25:15 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.2/24 brd brd 10.0.0.255 scope link eth1
So according to the docs now it should select the 10.0.0.2 address
for directy connected machines and use the 17.70.0.1 address for 'remote'
destinations.
This basically works fine. When I connect to a directly-connected 10.0.0.xxx
address it indeed chooses the 10.0.0.2 source address (link-address), but when
connecting to a remote host (via a GW learned by Zebra) it uses the 17.70.0.1
source address.
Great! Problem solved! Well.. Almost..
There seems to be one snag: Incorrect ARP source address.
If there is no ARP entry for the gateway yet (no traffic has gone out, routes
learned from another BGP peer) and I try to reach a remote address immediately
then the ARP request that goes out on the 10.0.0.0 network for the
correct gateway does *not* contain the 10.0.0.2 source address, but
instead 17.70.0.1.
Well.. That obviously does not work as this IP address does not occur on
this LAN and as a reasult most other routers will (correctly) ignore this.
If I try to connect to the correct gateway on a 10.0.0.x adress directly
then it does work as it will use the correct 10.0.0.2 source for it's
ARP request.
It seems that the ARP code also chooses the 'global' scope address for the
ARP request, while it should really always choose the 'link' address
of this interface as the source of the broadcast.
I have now temporarily fixed this by either adding some static ARP entries
or ARP-table filtering using iptables, but I feel that's only a temporary
measure.
Have I overlooked something in my setup or should I start poking in the
kernel ARP code?
Thanx!
Bye, Arno.
