Hello,
I was
recently tasked with moving existing network configuration for a machine
and some nspawn containers from iupdown to networkd.
The situation looks as follows:
A
single VPS has 3 IPs. One 37.x.x.x/22, and two 91.x.x.x/32. The 37-ip
is to be routed to the main server, whereas the two 91-ips should be
routed directly to nspawn containers running on the server. The server
uses systemd 247 and the container uses systemd 252, both Debian.
I created a bridge netdev like so:
[NetDev]
Name=br0
Type=bridge
# Matches physical network card
MACAddress=AA:BB:CC:DD:EE:FF
Bound the physical ethernet to it like so:
[Match]
Name=ens3
[Network]
Bridge=br0
And set up the main IP for the bridge like so:
[Match]
Name=br0
[Network]
DNS=...
DNS=...
Address=37.x.x.x/22
Gateway=37.x.x.1
The nspawn containers are added to the bridge via
[Network]
Bridge=br0
Up
until this point everything works. However, configuring networking
between the host and containers proved quite difficult and I'm unsure
whether I'm doing something wrong or networkd is.
What I tried was the following, inside the container:
[Match]
Virtualization=container
Name=host0
[Address]
Address=91.x.x.x/32
[Route]
Gateway=37.x.x.x
GatewayOnLink=true
However,
this did not create any usable routes to the host, nor did it throw any
errors in the journal. Pinging the host does not work.
Manually creating the routes with ip route did work:
ip r add 37.x.x.x dev host0 onlink
ip r add default dev host0 via 37.x.x.x
I tried a variety of different combinations of options in the .network file, Scope, Type, etc...
The only thing that successfully created any routes was the following:
[Match]
Virtualization=container
Name=host0
[Address]
Address=91.x.x.x/32
Peer=37.x.x.x/32
[Network]
Gateway=37.x.x.x
This
strikes me as odd because nowhere in the documentation, nor in any
online searching could I find this described as necessary (beyond the
manpage mentioning that Peer= exists)
On the
host side, I thought the bridge device, acting on Layer 2, would
automatically figure out routes to the containers (via ARP), or that
nspawn and networkd would interact in some way to add routes. However,
this didn't seem to happen, so I also had to add the following to the
bridge's .network file:
[Route]
Source=37.x.x.x
Destination=91.x.x.A
[Route]
Source=37.x.x.x
Destination=91.x.x.B
With
all of this, everything works fine now. However, since the routes, both
host-to-container and container-to-host, differ somewhat from the old
(also working) setup, and some of the steps necessary I could not find
described anywhere, I am left wondering if I fundamentally misunderstand
something about how Linux networking works, or if perhaps networkd is
behaving oddly because of the IP addresses being considered in different
networks.
I would love to find a conclusive
answer to this, especially because I want to make sure I understood the
fundamental concepts involved correctly.
