Antw: [EXT] Upgraded multiple systems to systemd 249.3 and all had eth1 not started / configured

["Ulrich Windl" <Ulrich.Windl@xxxxxxxxxxxxxxxxxxxx>]

Hi!

Actually I wonder (even before the upgrade) what that nonsense is about:
eth1: Interface name change detected, eth1 has been renamed to eth0.

Regards,
Ulrich

>>> Amish <anon.amish@xxxxxxxxx> schrieb am 16.08.2021 um 12:39 in Nachricht
<83597df0-fef1-049e-3707-2aa2a3819fb9@xxxxxxxxx>:
> Hello everyone,
> 
> So I have around seven Arch Linux based systems.
> 
> All systems have 2 or 3 network cards.
> 
> eth0 is LAN side (192.168.x.x/24 range). eth1 and eth2 has WAN (public 
> internet) connectivity.
> 
> Today I upgraded all systems to systemd 249.3 and Linux kernel 
> 5.13.10.arch1-1.
> 
> Everything worked fine before upgrading. And to my horror I started 
> getting complaints from all sites that internet is not working.
> 
> *Then I realized that all machines have interface eth1 down.*
> 
> Here is the journal log showing error after upgrading (journalctl -b 0 
> -u systemd-networkd)
> 
> Aug 16 09:30:18 kk systemd[1]: Starting Network Configuration...
> Aug 16 09:30:18 kk systemd-networkd[429]: lo: Link UP
> Aug 16 09:30:18 kk systemd-networkd[429]: lo: Gained carrier
> Aug 16 09:30:18 kk systemd-networkd[429]: Enumeration completed
> Aug 16 09:30:18 kk systemd[1]: Started Network Configuration.
> *Aug 16 09:30:18 kk systemd-networkd[429]: eth1: Interface name change 
> detected, renamed to eth0.**
> **Aug 16 09:30:18 kk systemd-networkd[429]: Could not process link 
> message: File exists**
> **Aug 16 09:30:18 kk systemd-networkd[429]: eth0: Failed**
> *Aug 16 09:30:18 kk systemd-networkd[429]: eth2: Interface name change 
> detected, renamed to eth1.
> Aug 16 09:30:18 kk systemd-networkd[429]: eth0: Interface name change 
> detected, renamed to tmpeth1.
> Aug 16 09:30:18 kk systemd-networkd[429]: eth1: Interface name change 
> detected, renamed to tmpeth2.
> Aug 16 09:30:18 kk systemd-networkd[429]: eth0: Interface name change 
> detected, renamed to tmpeth0.
> Aug 16 09:30:18 kk systemd-networkd[429]: tmpeth0: Interface name change 
> detected, renamed to eth0.
> Aug 16 09:30:18 kk systemd-networkd[429]: tmpeth1: Interface name change 
> detected, renamed to eth1.
> Aug 16 09:30:18 kk systemd-networkd[429]: tmpeth2: Interface name change 
> detected, renamed to eth2.
> Aug 16 09:30:19 kk systemd-networkd[429]: eth0: Link UP
> Aug 16 09:30:19 kk systemd-networkd[429]: eth2: Link UP
> Aug 16 09:30:19 kk systemd-networkd[429]: eth2: Gained carrier
> Aug 16 09:30:22 kk systemd-networkd[429]: eth0: Gained carrier
> 
> Explanation about tmpeth* naming is below but that is probably not 
> related to this issue. So can be ignored.
> 
> Notice how there is an error about renaming eth1 to eth0. I dont know 
> what is doing this renaming of eth1 to eth0. This renaming didn't happen 
> before upgrade. (see journal log below)
> 
> And also notice that there is no line stating eth1: Link UP.
> 
> 
> *Here is the journal log when systems worked perfectly.**
> *
> 
> Aug 13 09:17:20 kk systemd[1]: Starting Network Service...
> Aug 13 09:17:21 kk systemd-networkd[421]: Enumeration completed
> Aug 13 09:17:21 kk systemd[1]: Started Network Service.
> Aug 13 09:17:21 kk systemd-networkd[421]: eth0: Interface name change 
> detected, eth0 has been renamed to tmpeth0.
> Aug 13 09:17:21 kk systemd-networkd[421]: eth2: Interface name change 
> detected, eth2 has been renamed to eth0.
> Aug 13 09:17:21 kk systemd-networkd[421]: eth0: Interface name change 
> detected, eth0 has been renamed to tmpeth2.
> Aug 13 09:17:21 kk systemd-networkd[421]: eth1: Interface name change 
> detected, eth1 has been renamed to eth0.
> Aug 13 09:17:21 kk systemd-networkd[421]: eth0: Interface name change 
> detected, eth0 has been renamed to tmpeth1.
> Aug 13 09:17:21 kk systemd-networkd[421]: tmpeth1: Interface name change 
> detected, tmpeth1 has been renamed to eth1.
> Aug 13 09:17:21 kk systemd-networkd[421]: tmpeth2: Interface name change 
> detected, tmpeth2 has been renamed to eth2.
> Aug 13 09:17:21 kk systemd-networkd[421]: tmpeth0: Interface name change 
> detected, tmpeth0 has been renamed to eth0.
> Aug 13 09:17:21 kk systemd-networkd[421]: eth1: Link UP
> Aug 13 09:17:21 kk systemd-networkd[421]: eth2: Link UP
> Aug 13 09:17:21 kk systemd-networkd[421]: eth2: Gained carrier
> Aug 13 09:17:21 kk systemd-networkd[421]: eth0: Link UP
> Aug 13 09:17:26 kk systemd-networkd[421]: eth0: Gained carrier
> 
> Notice how there was no attempt to rename eth1 to eth0 at the beginning 
> (i.e. when everything worked fine)
> 
> And notice how all interfaces showed Link UP.
> 
> 
> So something changed either in systemd or in Linux kernel.
> 
> Any idea what is wrong where? And which process is trying to rename eth1 
> to eth0 at system startup?
> 
> All systems are production systems and after today's long downtime, I 
> can not downgrade any system to check what is wrong as management would 
> be on fire if there is another downtime.
> 
> Thank you in advance,
> 
> Amish
> 
> 
> PS:
> 
> Little about tmpeth* naming.
> 
> Some old scripts that we have expect interface names starting with eth. 
> But those names are not predictable.
> 
> So to get predictable names starting with eth*, first I temporarily 
> rename all interface with tmpeth*. This is done via udev rules.
> 
> SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="XX:XX:XX:XX:XX:XX", 
> NAME="tmpeth0"
> SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="XX:XX:XX:XX:XX:YY", 
> NAME="tmpeth1"
> SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="XX:XX:XX:XX:XX:ZZ", 
> NAME="tmpeth2"
> 
> Then I have a small service (script) which runs before 
> network-pre.target to convert these names back to eth*
> 
> #search for network interface with name starting from "tmpeth" and 
> rename them to "eth"
> /usr/bin/find /sys/class/net -maxdepth 1 -name "tmpeth[0-9]" -type l 
> -printf "%f\n" | while read tmpiface; do /usr/bin/ip link set dev 
> "$tmpiface" name "$(echo $tmpiface | sed s/tmpeth/eth/)"; done
> 
> This ensures that I have predictable names starting with eth*. And it is 
> working fine from 2-3 years. Even with current issue, name assignment is 
> working fine.