Yet another update, I was able to get it working .. but feel like it is a hack so comments welcome ... see below:
On Wed, Nov 17, 2021 at 12:26 AM Brian Hutchinson <b.hutchman@xxxxxxxxx> wrote:
Update belowOn Tue, Nov 16, 2021 at 2:27 PM Brian Hutchinson <b.hutchman@xxxxxxxxx> wrote:Hi Mikulėnas,On Tue, Nov 16, 2021, 3:12 AM Mantas Mikulėnas <grawity@xxxxxxxxx> wrote:Most of this looks like it could be done with systemd-networkd to create a bond .netdev, with a small oneshot service for i2c. (What's the exact criteria for when it should be run? Does it depend on bond0 being there, does it need to be last, etc?)It can be last in the startup chain I guess, don't know what other systemd things that might need the network to be up before the last unit file runs.I start linuxptp too so I would have the unit file that starts ptp4l start after the bond was created etc.Same thing for the i2c command to enable the switch.Regards,BrianOn Tue, Nov 16, 2021, 02:58 Brian Hutchinson <b.hutchman@xxxxxxxxx> wrote:Hi,I'm on a IMX8 platform and have a Microchip KSZ9567 Ethernet switch. I can use IP commands to manually bring lan1 and lan2 interfaces up and then create a redundant/failover bond ... but I'm having difficulty figuring out how to do this the "systemd" way.My first attempt was to just have systemd run a script of all the commands I do manually but during system startup there appears to be race conditions so I have to set my service type to "Idle" and sometimes even that doesn't work. So I want to exploit any systemd support for DSA and bonding.Here is script my manual steps which is what I want systemd to ultimately do:#!/bin/bash
# Create a redundant bond between ksz9567 DSA lan1 and lan2 interfaces
# Load bonding kernel module
modprobe bonding
# Bring up CPU interface (cpu to switch port 7 - the RGMII link)
ip link set eth0 up
# Create a bond
echo +bond0 > /sys/class/net/bonding_masters
# Set mode to active-backup (redundancy failover)
echo active-backup > /sys/class/net/bond0/bonding/mode
# Set time it takes (in ms) for slave to move when a link goes down
echo 1000 > /sys/class/net/bond0/bonding/miimon
# Add slaves to bond
echo +lan1 > /sys/class/net/bond0/bonding/slaves
echo +lan2 > /sys/class/net/bond0/bonding/slaves
# Set IP and netmask of the bond
ip addr add 192.168.0.4/24 dev bond0
# And bring bond up. Pings and network connectivity should work now
ip link set bond0 up
# For a board that doesn't have Ethernet switch hardware strapped to enable at boot .. enable it now
i2cset -f -y 0 0x5f 0x03 0x00 0x01 iThanks for any information, pointers etc.Regards,BrianSo not sure I'm doing this right. eth0 needs to be up before lan1 and lan2 can be added to the bond. Not quite sure how to do that with systemd but I made the following files and see some progress but ping doesn't work so appears I have no network connectivity:root@imx8mmevk:/etc/systemd/network# cat 10-bond1.netdev
[NetDev]
Name=bond1
Kind=bond
[Bond]
Mode=active-backup
PrimaryReselectPolicy=failure
MIIMonitorSec=2sroot@imx8mmevk:/etc/systemd/network# cat 10-bond1.network
[Match]
Name=bond1
[Network]
Address=192.168.0.4/24root@imx8mmevk:/etc/systemd/network# cat 20-lan1.network
[Match]
Name=lan1
[Network]
Bond=bond1
PrimarySlave=trueroot@imx8mmevk:/etc/systemd/network# cat 30-lan2.network
[Match]
Name=lan2
[Network]
Bond=bondip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1506 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
3: lan1@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
4: lan2@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether f0:1f:af:6b:b2:17 brd ff:ff:ff:ff:ff:ff
5: bond1: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether be:87:0a:9b:13:03 brd ff:ff:ff:ff:ff:ffcat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v5.10.69
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: None
MII Status: down
MII Polling Interval (ms): 2000
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0At this point there should be info on lan1 and lan2 status but don't see it.... so of course I can't ping.Next I did systemctl restart systemd-networkd and saw the following:systemctl restart systemd-networkd
root@imx8mmevk:~# [ 33.816313] device eth0 entered promiscuous mode
[ 33.821026] audit: type=1700 audit(1636550966.339:2): dev=eth0 prom=256 old_prom=0 auid=4294967295 uid=995 gid=994 ses=4294967295
[ 33.867164] ksz9477-switch 0-005f lan2: configuring for phy/gmii link mode
[ 33.875066] bond1: (slave lan2): Enslaving as a backup interface with a down link
[ 33.919055] ksz9477-switch 0-005f lan1: configuring for phy/gmii link mode
[ 33.926683] bond1: (slave lan1): Enslaving as a backup interface with a down link
[ 38.066148] ksz9477-switch 0-005f lan1: Link is Up - 1Gbps/Full - flow control rx/tx
[ 39.472022] bond1: (slave lan1): link status definitely up, 1000 Mbps full duplex
[ 39.479537] bond1: (slave lan1): making interface the new active one
[ 39.486154] bond1: active interface up!
[ 39.490071] IPv6: ADDRCONF(NETDEV_CHANGE): bond1: link becomes readyAt which point I can ping. Feels like there still might be some kind of race condition as things won't work unless I manually issue systemctl restart systemd-networkd command after logging in.In kernel dmesg logs I see:[ 4.165940] bond1: (slave lan2): Opening slave failed
[ 4.196834] bond1: (slave lan1): Opening slave failed
[ 4.315588] Generic PHY fixed-0:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=fixed-0:00, irq=POLL)
[ 4.326181] fec 30be0000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[ 4.561000] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes readyLoaded: loaded (/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
Active: [[0;1;32mactive (running)[[0m since Sun 2020-09-20 10:43:59 UTC; 1 years 1 months ago
TriggeredBy: [[0;1;32m*[[0m systemd-networkd.socket
Docs: man:systemd-networkd.service(8)
Main PID: 251 (systemd-network)
Status: "Processing requests..."
Tasks: 1 (limit: 1574)
Memory: 1.5M
CGroup: /system.slice/systemd-networkd.service
`-251 /lib/systemd/systemd-networkd
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: [[0;1;31m[[0;1;39m[[0;1;31mlan2: Could not join netdev: Network is down[[0m
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mlan2: Failed[[0m
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: [[0;1;31m[[0;1;39m[[0;1;31mlan1: Could not join netdev: Network is down[[0m
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mlan1: Failed[[0m
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: eth0: IPv6 successfully enabled
Nov 10 13:28:56 imx8mmevk systemd-networkd[251]: eth0: Link UP
Nov 10 13:28:57 imx8mmevk systemd-networkd[251]: eth0: Gained carrier
Nov 10 13:28:57 imx8mmevk systemd-networkd[251]: bond1: IPv6 successfully enabled
Nov 10 13:28:57 imx8mmevk systemd-networkd[251]: bond1: Link UP
Nov 10 13:28:58 imx8mmevk systemd-networkd[251]: eth0: Gained IPv6LL
... so looks like the bond stuff is trying to happen before eth0 (my DSA HOST/CPU interface to switch) is up. How can I make eth0 up with systemd? eth0 just needs to be up ... no IP etc., and the bond1 gets the IP etc.For now I'm doing the i2c command to enable my switch in u-boot but still need to incorporate that into systemd somehow.Any ideas as to what I'm doing wrong? I think at a minimum I need to bring eth0 up before the bonding stuff happens but not quite sure what that would look like using systemd.Regards,Brian
I tried and tried to get eth0 to come up before the bond was brought up. I had everything named in lexical order but didn't appear to matter. I added a eth0.network file and in it specified
ActivationPolicy=always-up and other things but could not get eth0 to come up.It was obvious the bond was trying to be established before eth0 was up and since this is using DSA that just won't work. Dmesg logs would show slaves being added before eth0 was up as in copy/paste from previous email.
So I added a service to bring eth0 up:
cat eth0-up.service
[Unit]
Description=Bring eth0 up before bonding
Before=network-pre.target
Wants=network-pre.target
[Service]
ExecStart=/usr/local/bin/eth0-up.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
[Unit]
Description=Bring eth0 up before bonding
Before=network-pre.target
Wants=network-pre.target
[Service]
ExecStart=/usr/local/bin/eth0-up.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
cat /usr/local/bin/eth0-up.sh
#!/bin/bash
ip link set eth0 up
#!/bin/bash
ip link set eth0 up
... I feel like this is a hack, that systemd can probably do this but either I can't figure out how or there is a problem with the code.
Again, advice and info welcome.
Regards,
Brian
