On Mon, 2022-06-27 at 10:10 +0200, Zbigniew Jędrzejewski-Szmek wrote:
> On Sun, Jun 26, 2022 at 12:36:14AM +0530, Vipul Siddharth wrote:
> > This document represents a proposed Change. As part of the Changes
> > process, proposals are publicly announced in order to receive
> > community feedback. This proposal will only be implemented if
> > approved
> > by the Fedora Engineering Steering Committee.
> >
> >
> > == Summary ==
> >
> > The `systemd-udev` package installs
> > `"/usr/lib/systemd/network/99-default.link"`,
> > which sets `Link.MACAddressPolicy=persistent`. This proposal is to
> > change it to set `Link.MACAddressPolicy=none` to stop changing the
> > MAC address.
> > This is particularly important for bridge and bond devices.
> >
> > This change can either only apply to bridge/bond devices, or to
> > various software devices. That is to be discussed.
>
> Hi!
>
> I already participated in the upstream discussion, so what I write
> here will be a restatement to some extent, but with a look from the
> side of Fedora specifically.
>
> The proposal has two variants: 1. just changing the policy to
> MACAddressPolicy=none
> or 2. limiting the change to bridge and bond devices.
>
> Re variant 1: MACAddressPolicy=persistent applies to all devices that
> don't have
> a hardware address. The proposal as written (blanket
> MACAddressPolicy=none)
> would change behaviour for all kinds of devices, incl. e.g. software
> devices like veths, and cheap hardware devices without a fixed MAC.
the proposal was only about software devices.
The difference is that software devices are explicitly added by some
tool (if we ignore module parameters like "max_bonds" -- which on
fedora/systemd is already 0 by default).
A hardware device gets automatically added when loading the kernel.
More importantly, udev already does renaming, so any user is well
advised to wait for udev to settles. At which point, they can also wait
for the MAC address policy.
> The proposal doesn't provide any justification for this (except for
> simplicity of implementation) and this variant seems pretty bad and
> I'm strongly opposed.
The first point: the race. And that udev touches interfaces created by
somebody, without being explicitly told to do so.
Yes, udev always does something to the device, like attaching
attributes such as ID_NET_NAMING_SCHEME. But aside the MAC address
policy, it doesn't do anything relevant where a race would matter.
>
> Re variant 2: the proposal limited to brige/bond devices seems much
> more
> reasonable. In particular, the case described below of a server
> (virtualized
> or not) in a big datacenter is the one case where the benefits of
> MACAddressPolicy=none are clearly visible. I still don't think it's
> worth
> changing the default, but here the cost:benefit ratio is much closer.
>
> > == Benefit to Fedora ==
> >
> > Pros:
> >
> > - Consistent behavior with RHEL8 and RHEL9.
> >
> > - Avoid race of udev and the tool that creates the interface.
>
> The race will happen if the creation is done in a specific way. But
> at
> the same time, even the Change proposal describes how to avoid the
> race ('ip link add address aa:aa:aa:aa:aa:aa type bridge'). So the
> situation
> can be summarized as "we have a bunch of 'big' tools that create
> devices like NetworkManager or systemd-networkd, which already know
> or
> can be easily fixed to avoid the race, and a manual tool which can be
> invoked
> in a way that avoids the race". Instead of changing the default in
> udev we
> could educate people how to invoke it better.
I don't think that big projects matter much. They can learn this. For
example, docker had a problem with this ([1]), which presumably got
fixed long ago.
[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/15#note_162509
The problem are smaller projects and user scripts/tools.
For example, NetworkManager-ci contains hundreds of scripts for
testing, and -- while we should know better -- we don't wait for udev
after creating a veth interface for testing. That is no problem for us,
we just set MACAddressPolicy=none. But it makes me wonder, just how
many "naive" scripts are out there that don't get this right.
You say, "the race will happen if the creation is done in a specific
way". The problem is that the specific way is the most straight forward
one.
Also, to educate people how to use iproute2 (?) better, seems not
clearly best. Is there really a problem on the user's side when they
are creating an interface and don't expect udev to interfere?
> > - Bridge and bond interfaces can get the MAC addresses from their
> > first port.
> >
> > In the case of `MACAddressPolicy=none` for a bond (or bridge) the
> > bond will
> > get a MAC related to one of its physical NIC devices. In the case
> > of
> > provisioning
> > new systems (especially in a large datacenter) information about
> > the server
> > can be used to configure the network environment (DHCP, Firewall,
> > etc) before
> > the server is ever even powered on. This does mean that you may
> > have to update
> > your network environment configuration if you replace a NIC (con),
> > but that you
> > won't have to update your network environment configuration if you
> > re-install
> > your server (pro), which is what you'd have to do now with
> > `MACAddressPolicy=persistent`.
>
> Yep. This is *the* case.
>
> > Cons:
> >
> > - Deviate from upstream systemd.
>
> It is also important to mention that Fedora will "deviate" from
> itself
> (it's former self). We would be changing a default in place since
> ~2013 [1].
>
> [1] https://github.com/systemd/systemd/commit/16b9b87aee
>
> > It is desirable that RHEL and Fedora behaves similar. A possible
> > outcome
> > could be the current behavior stays and RHEL 10 would change
> > behavior. On the
> > other hand, different distributions (or even Fedora spins) have
> > different
> > uses and needs. Deviating might be fine. In the same vein, there is
> > also
> > a desire to stay close to upstream systemd behavior. But the uses
> > of systemd
> > project go beyond Fedora/RHEL, so deviating here may also be fine.
>
> So:
> - Variant 1 is not good, variant 2 makes more sense.
> - The motivating case for v.2 is the "big datacenter" case and race
> when 'ip' is invoked.
> - We could improve the tools: 'ip link' could be taught to wait until
> udev has stopped processing the device, users can be taught to use
> better invocations.
see below.
> - For "small" users (individual machine admins who just install some
> server and configure it after turning it on and/or swap network
> cards between machines), having stable MAC addresses that doesn't
> depend on the order of device discovery or removal of a single
> network card seems better.
That claim is most relevant, but seems not well justified. It comes
from a desire for order, where things are stable.
I am not so convinced though, given that the solution requires to
involve udev. A counter point would be that many users might not care
for this feature, and those who do, can enable it just fine.
> - For "big" users (the datacenter case), changing the policy make
> sense,
> but at the same time, those folks can just insert a policy
> override,
> they're most likely using some ansible/puppet/cheffy thingy.
> - RHEL is more of the "big" case, Fedora is more the "small" case.
> Sometimes RHEL and Fedora have different defaults, sometimes RHEL
> takes years to follow what Fedora does.
I personally care mostly about RHEL here and to find out what to do. So
if we agree that RHEL is allowed to have a different downstream
behavior than Fedora, I'd be satisfied!
>
> So overall, I think this proposal would make most sense when limited
> to specific types of hardware (bridge and bond?), and also to
> specific
> spins: it's a better fit for CoreOS than Workstation…
Makes sense.
I think CoreOS would be tempted to set MACAddressPolicy=none. Thus, a
major Fedora user would go against the distro default. That raises a
question about why Workstation behaves different. But if CoreOS (or
other spins) are allowed to do that, I think Dusty would also be
satisfied.
>
> But I haven't seen discussion of other approaches: would making 'ip'
> better be an option? 'udev' already locks some devices when
> processing
> using a simple BSD lock. Should we do the same for network devices
> and just teach 'ip' to do the same? (This would be much simpler and
> maybe more appealing that teaching it to wait for udev to report
> that it's finished).
iproute2 is the obvious candidate, but there can be any tools that use
the kernel API some other way (e.g. pyroute2). It wouldn't solve the
problem for them making the "solution" less universal and attractive.
I would be surprised if iproute2 would accept patches to integrate with
udev. It seems like a low-level tool, and I could understand if the
answer is to use the tool correctly. Unfortunately, the correct way is
not immediately obvious.
Thank you for all your feedback!!
Thomas
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
