Re: Systemd setup DSA interfaces in port mode and bond them together?

[Alvin Šipraga <ALSI@xxxxxxxxxxxxxxx>]

On 11/18/21 23:25, Brian Hutchinson wrote:
> Hi Alvin,
> 
> 
> On Thu, Nov 18, 2021 at 4:20 PM Alvin Šipraga <ALSI@xxxxxxxxxxxxxxx 
> <mailto:ALSI@xxxxxxxxxxxxxxx>> wrote:
> 
>     Hi Brian,
> 
>     On 11/18/21 01:20, Brian Hutchinson wrote:
>       > Yet another update, I was able to get it working .. but feel
>     like it is
>       > a hack so comments welcome ... see below:
>       >
> 
>     <snip>
> 
>       >
>       > I tried and tried to get eth0 to come up before the bond was brought
>       > up.  I had everything named in lexical order but didn't appear to
>       > matter.  I added a eth0.network file and in it specified
>       > |ActivationPolicy=|always-up and other things but could not get
>     eth0 to
>       > come up.
> 
>     What kernel version are you using? Since Linux 5.11 we have the
> 
> 
> I'm using linux-fslc-imx 5.10.69
> 
>     following two changes ([1] and [2]) which should automatically bring
>     up/down the master (eth0) whenever user ports (lan1, lan2) are brought
>     up/down. Please confirm whether or not you are using 5.11 or later.
> 
> 
> I don't think that will work either.  eth0 has to be up and stay up or 
> DSA driver won't work at all.  eth0 has to be up or the slaves can't be 
> added to the bond.

This use-case is also addressed in the newer kernels (>=5.11), see the 
below commit. It is not only when a user port is brought up (as I 
summarized it), but rather when it is opened.

commit 9d5ef190e5615a7b63af89f88c4106a5bc127974
Author: Vladimir Oltean <vladimir.oltean@xxxxxxx>
Date:   Fri Feb 5 15:37:10 2021 +0200

     net: dsa: automatically bring up DSA master when opening user port

     DSA wants the master interface to be open before the user port is 
due to
     historical reasons. The promiscuity of interfaces that are down used to
     have issues, as referenced Lennert Buytenhek in commit df02c6ff2e39
     ("dsa: fix master interface allmulti/promisc handling").

     The bugfix mentioned there, commit b6c40d68ff64 ("net: only invoke
     dev->change_rx_flags when device is UP"), was basically a "don't do
     that" approach to working around the promiscuity while down issue.

     Further work done by Vlad Yasevich in commit d2615bf45069 ("net: core:
     Always propagate flag changes to interfaces") has resolved the
     underlying issue, and it is strictly up to the DSA and 8021q drivers
     now, it is no longer mandated by the networking core that the master
     interface must be up when changing its promiscuity.

     From DSA's point of view, deciding to error out in dsa_slave_open
     because the master isn't up is
     (a) a bad user experience and
     (b) knocking at an open door.
     Even if there still was an issue with promiscuity while down, DSA could
     still just open the master and avoid it.

     Doing it this way has the additional benefit that user space can now
     remove DSA-specific workarounds, like systemd-networkd with 
BindCarrier:
     https://github.com/systemd/systemd/issues/7478

     And we can finally remove one of the 2 bullets in the "Common pitfalls
     using DSA setups" chapter.

     Tested with two cascaded DSA switches:

     $ ip link set sw0p2 up
     fsl_enetc 0000:00:00.2 eno2: configuring for fixed/internal link mode
     fsl_enetc 0000:00:00.2 eno2: Link is Up - 1Gbps/Full - flow control 
rx/tx
     mscc_felix 0000:00:00.5 swp0: configuring for fixed/sgmii link mode
     mscc_felix 0000:00:00.5 swp0: Link is Up - 1Gbps/Full - flow 
control off
     8021q: adding VLAN 0 to HW filter on device swp0
     sja1105 spi2.0 sw0p2: configuring for phy/rgmii-id link mode
     IPv6: ADDRCONF(NETDEV_CHANGE): eno2: link becomes ready
     IPv6: ADDRCONF(NETDEV_CHANGE): swp0: link becomes ready

     Signed-off-by: Vladimir Oltean <vladimir.oltean@xxxxxxx>
     Reviewed-by: Andrew Lunn <andrew@xxxxxxx>
     Reviewed-by: Florian Fainelli <f.fainelli@xxxxxxxxx>
     Signed-off-by: Jakub Kicinski <kuba@xxxxxxxxxx>


> 
> If you look back up the thread where I'm doing the commands manually 
> (I'll re-copy below) ... the first thing is to bring eth0 up.  If that 
> doesn't happen, nothing else works.

I forgot about the bond in my reply. Does it work if you do 
BindCarrier=eth0 from the bond's .network file? You might be out of luck 
depending on how networkd implements BindCarrier=, but hopefully 
something like that works. Remember also that you need a .network file 
covering eth0, even though it's just a conduit and you don't want to 
give it an IP. You can do that by omitting the [Network] section and 
just having a [Match] on eth0.

> 
> These are the manual steps that work.  This is what I'm trying to 
> automate the "systemd way".  I read that "bond0" is somehow reserved so 
> my systemd.netdev and .network files use bond1.
> 
> #!/bin/bash
> 
> # Create a redundant bond between ksz9567 DSA lan1 and lan2 interfaces
> 
> # Load bonding kernel module
> modprobe bonding
> 
> # Bring up CPU interface (cpu to switch port 7 - the RGMII link)
> ip link set eth0 up
> 
> # Create a bond
> echo +bond0 > /sys/class/net/bonding_masters
> 
> # Set mode to active-backup (redundancy failover)
> echo active-backup > /sys/class/net/bond0/bonding/mode
> 
> # Set time it takes (in ms) for slave to move when a link goes down
> echo 1000 > /sys/class/net/bond0/bonding/miimon
> 
> # Add slaves to bond
> 
> echo +lan1 > /sys/class/net/bond0/bonding/slaves
> echo +lan2 > /sys/class/net/bond0/bonding/slaves
> 
> # Set IP and netmask of the bond
> ip addr add 192.168.0.4/24 
> <http://192.168.0.4/24 
> dev bond0
> 
> # And bring bond up.  Pings and network connectivity should work now
> ip link set bond0 up
> 
> # For a board that doesn't have Ethernet switch hardware strapped to 
> enable at boot .. enable it now
> i2cset -f -y 0 0x5f 0x03 0x00 0x01 i
> 
> 
>     If you are using an older kernel, and systemd is version 243 or later,
> 
> 
> I checked and I'm using 244.5
> 
>     you can use the BindCarrier= setting (see [1]) in the [Network] section
>     of your .network file(s) for lan1 and lan2. Something like this:
> 
>              [Match]
>              Name=lan*
> 
>              [Network]
>              BindCarrier=eth0
> 
> 
> I added the BindCarrier=eth0 in both of my DSA lan1 and lan2 .network 
> files.  I got the same result as before ... dmesg reports lan1 and lan2 
> slaves are brought up before eth0 ... and so they fail.  The only thing 
> so far that works is to bring eth0 up with a network-pre.target 
> service.  And I'm doing the i2c command to enable my switch in 
> a|After=network-online.target service.
> |
> 
> [    4.367802] bond1: (slave lan2): Opening slave failed
> [    4.467561] bond1: (slave lan1): Opening slave failed
> [    4.640914] fec 30be0000.ethernet eth0: Link is Up - 1Gbps/Full - 
> flow control off
> [    5.104983] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> 
> Regards,
> 
> Brian