Here is the evidence for my conclusions, quoted from our Bugzilla database.
------- Additional Comment #11 From Patrick Turley 2004-09-16 19:43
Our configuration uses the ip command to attach IP addresses to an
interface. Using these methods, it experience a severe limit on the
total number of addresses.
Tim has reported attaching thousands of addresses to his system test
boxes. There is also plenty of evidence on the Internet of other people
doing the same. All of the examples I looked at in detail were using the
ifconfig command.
In an attempt to discover if the ip command was the real problem, I
hunted down which ifconfig command would accomplish the same effect as
the ip command. Here are the ip commands required to attach a network to
a given device:
ip addr add <networkaddress>/<prefixlength> brd + dev <device>
ip link set <device> up
The equivalent ifconfig command is:
ifconfig <device>:<aliasnumber> <address> netmask <netmask> up
I replaced the ip commands with the ifconfig command, ran the test and
got precisely the same results. The ifconfig command didn't seem to
improve anything.
------- Additional Comment #12 From Patrick Turley 2004-09-16 20:04
I used a script I downloaded from the Internet that uses ifconfig to
attach many IP addresses to a single interface. I used this script to
attach 1000 IP addresses to eth0 and saw no ill effects.
I used the low-level ip commands to examine the state of the network
more closely:
[root@rnsa root]# ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc htb qlen 1000
link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.0.1/24 brd 192.168.0.255 scope global eth0:0
inet 13.0.0.1/24 brd 13.255.255.255 scope global eth0:1
inet 13.0.1.1/24 brd 13.255.255.255 scope global eth0:251
inet 13.0.2.1/24 brd 13.255.255.255 scope global eth0:501
inet 13.0.3.1/24 brd 13.255.255.255 scope global eth0:751
inet 13.0.0.2/24 brd 13.255.255.255 scope global secondary eth0:2
inet 13.0.0.3/24 brd 13.255.255.255 scope global secondary eth0:3
inet 13.0.0.4/24 brd 13.255.255.255 scope global secondary eth0:4
inet 13.0.0.5/24 brd 13.255.255.255 scope global secondary eth0:5
.
.
.
inet 13.0.3.246/24 brd 13.255.255.255 scope global secondary eth0:996
inet 13.0.3.247/24 brd 13.255.255.255 scope global secondary eth0:997
inet 13.0.3.248/24 brd 13.255.255.255 scope global secondary eth0:998
inet 13.0.3.249/24 brd 13.255.255.255 scope global secondary eth0:999
inet 13.0.3.250/24 brd 13.255.255.255 scope global secondary eth0:1000
Notice that most of these addresses are tagged as "secondary". From
http://www.linux-ip.net/html/linux-ip.html#tools-ip-address-add
we see that:
When you add an address within the same CIDR network as another
address on the same interface, the second address becomes a
secondary address, meaning that if the first address is removed,
the second address will also be purged from the interface.
This is consistent with the output of ip addr show. The script looped
through 1000 address, which is enough to span 4 class C networks. So,
the first address in each such network became a primary address, and all
the other addresses within those networks became secondaries. This
finally gives a concrete explanation for why we see many addresses on an
interface disappear when we only delete one.
From
http://www.linux-ip.net/gl/ip-cref/node33.html
we learn that a little more about secondary addresses:
1 secondary
-- the address is not used when selecting the default source
address of outgoing packets (Cf. Appendix A, p.[*].). An IP
address becomes secondary if another address with the same
prefix bits already exists. The first address is primary. It is
the leader of the group of all secondary addresses. When the
leader is deleted, all secondaries are purged too.
I also had a look at the route table:
[root@rnsa root]# ip route show
13.0.2.0/24 dev eth0 proto kernel scope link src 13.0.2.1
13.0.3.0/24 dev eth0 proto kernel scope link src 13.0.3.1
13.0.0.0/24 dev eth0 proto kernel scope link src 13.0.0.1
13.0.1.0/24 dev eth0 proto kernel scope link src 13.0.1.1
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.1
10.1.0.0/16 dev eth1 proto kernel scope link src 10.1.1.18
127.0.0.0/8 dev lo scope link
default via 10.1.254.254 dev eth1
This confirms what was said immediately above - none of the secondary
addresses are represented here - only the primary ones. This implies
that packets destined for one of the newly-created networks will only
take on a primary address as their source address.
------- Additional Comment #13 From Patrick Turley 2004-09-16 20:19
In our configuration, no two networks are ever allowed to overlap. That
means that *all* the addresses are primary. Take a look at the network
information for a system that has 100 VLANs and networks:
[root@rnsa root]# ip addr show
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc htb qlen 1000
link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
inet 192.168.0.1/24 brd 192.168.0.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc htb qlen 1000
link/ether 00:09:6b:66:09:07 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.18/16 brd 10.1.255.255 scope global eth1
1304: vlan1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
inet 10.0.1.1/24 brd 10.0.1.255 scope global vlan1
1305: vlan2: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
inet 10.0.2.1/24 brd 10.0.2.255 scope global vlan2
1306: vlan3: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
inet 10.0.3.1/24 brd 10.0.3.255 scope global vlan3
.
.
.
1401: vlan98: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
inet 10.0.98.1/24 brd 10.0.98.255 scope global vlan98
1402: vlan99: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
inet 10.0.99.1/24 brd 10.0.99.255 scope global vlan99
1403: vlan100: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
inet 10.0.100.1/24 brd 10.0.100.255 scope global vlan100
Since each address appears on a separate interface, they are all
primary. However, since they are all on disjoint networks, they would
still all be primary even if they all appeared on the same interface
(which is the original configuration that spawned this problem report).
Have a look at the routing table as well:
[root@rnsa root]# ip route show
10.0.20.0/24 dev vlan20 proto kernel scope link src 10.0.20.1
10.0.21.0/24 dev vlan21 proto kernel scope link src 10.0.21.1
10.0.22.0/24 dev vlan22 proto kernel scope link src 10.0.22.1
.
.
.
10.0.73.0/24 dev vlan73 proto kernel scope link src 10.0.73.1
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.1
10.0.100.0/24 dev vlan100 proto kernel scope link src 10.0.100.1
10.0.99.0/24 dev vlan99 proto kernel scope link src 10.0.99.1
10.0.98.0/24 dev vlan98 proto kernel scope link src 10.0.98.1
10.0.97.0/24 dev vlan97 proto kernel scope link src 10.0.97.1
10.0.96.0/24 dev vlan96 proto kernel scope link src 10.0.96.1
10.1.0.0/16 dev eth1 proto kernel scope link src 10.1.1.18
127.0.0.0/8 dev lo scope link
default via 10.1.254.254 dev eth1
There's an entry in the routing table for every VLAN (as there must be).
And, even if all these networks appeared on the same interface, there
would still need to be a routing rule for each.
------- Additional Comment #14 From Patrick Turley 2004-09-16 20:25
Everything I've learned so far leads me to conclude that the limit we're
running up against is the number of *primary* addresses. The problem
might be having too many primary addresses on a single interface, or it
might be the total number of primary addresses in the system.
Since each primary address requires a route, my first theory was that we
were putting too many entries in the routing table. I've since
discovered that the default size of the routing table is 32K, and
anecdotal evidence on the Internet suggests that a Linux router can deal
with this number comfortably.
Note, however, that N primary addresses implies at least N routing table
entries - but the reverse is not true. It's not likely that anyone else
is trying to configure more than a handful of addresses. The reason for
this apparent limit on the number of primary addresses is still a bit of
a mystery.
_______________________________________________
LARTC mailing list / LARTC@xxxxxxxxxxxxxxx
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
