throw routes are getting removed when networkd is restarted

[Robert Dahlem <Robert.Dahlem@xxxxxxx>]

Hi,

I'm running on Debian Bullseye, systemd 247.

StrongSwan 5.9.1 (an IPsec implementation) establishes throw routes in
table 220 when activating the bypass-lan plugin.

Basically that means: you have a VPN tunnel giving you a prioritized
default route through the VPN gateway but you can still reach systems in
local networks. It looks like this:

# ip a
...
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP group default qlen 1000
    ...
    inet 192.168.1.160/24 brd 192.168.1.255 scope global dynamic ens18
    inet 172.29.254.11/32 scope global ens18
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
state UP group default qlen 1000
    ...
    inet 192.168.180.2/24 brd 192.168.180.255 scope global ens19
4: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UP group default qlen 1000
    inet 10.10.10.1/24 brd 10.10.10.255 scope global vmbr1
    ...
# ip rule
0:      from all lookup local
220:    from all lookup 220
32766:  from all lookup main
32767:  from all lookup default
# ip route sh table 220
default via 192.168.1.1 dev ens18 proto static src 172.29.254.11
throw 10.10.10.0/24 proto static
throw 192.168.1.0/24 proto static
throw 192.168.180.0/24 proto static

Any outgoing traffic goes through table 220 where the default route
points to the VPN tunnel. Without the throw routes traffic for local
networks would be sent through the VPN tunnel too.

Now the problem: when I restart networkd, the throw routes get removed:

# systemctl restart systemd-networkd
# ip route sh table 220
default via 192.168.1.1 dev ens18 proto static src 172.29.254.11

Of course now I can no longer reach the local networks.

I run networkd with "Environment=SYSTEMD_LOG_LEVEL=debug", so I get this
in the log:

# grep throw /var/log/syslog  | cut -d " " -f 6- | grep -v lo: \
| sed 's!src: n/a, gw: n/a, prefsrc: n/a, scope: global, !!'
Remembering route: dst: 192.168.180.0/24, table: 220, proto: static,
type: throw
Remembering route: dst: 192.168.1.0/24, table: 220, proto: static, type:
throw
Remembering route: dst: 10.10.10.0/24, table: 220, proto: static, type:
throw
Removing route: dst: 192.168.180.0/24, table: 220, proto: static, type:
throw
Removing route: dst: 10.10.10.0/24, table: 220, proto: static, type: throw
Removing route: dst: 192.168.1.0/24, table: 220, proto: static, type: throw
Removing route: dst: 192.168.180.0/24, table: 220, proto: static, type:
throw
Removing route: dst: 10.10.10.0/24, table: 220, proto: static, type: throw
Removing route: dst: 192.168.1.0/24, table: 220, proto: static, type: throw
Removing route: dst: 192.168.180.0/24, table: 220, proto: static, type:
throw
Removing route: dst: 10.10.10.0/24, table: 220, proto: static, type: throw
Removing route: dst: 192.168.1.0/24, table: 220, proto: static, type: throw
Forgetting route: dst: 192.168.180.0/24, table: 220, proto: static,
type: throw
Forgetting route: dst: 10.10.10.0/24, table: 220, proto: static, type: throw
Forgetting route: dst: 192.168.1.0/24, table: 220, proto: static, type:
throw

At first, networkd remembers the throw routes, then it removes and
forgets them. Why is that and how can I prevent it from doing so?

(Actually, the problem is a bit more complex and has to do with
disappearing throw routes when interfaces come up "late", i.e. WIFI
interfaces. I tried to show the behavior in a simple test case.)

Regards,
Robert