Am 2022-09-06 01:54, schrieb Vladimir Oltean:
On Tue, Sep 06, 2022 at 12:17:29AM +0200, Michael Walle wrote:
First, let me say, I'm fine with this patch. But I'm not sure,
how many MAC addresses are actually reserved on your
RDB/QDS boards?
AFAIK, the Reference Design Boards are sold with an unprogrammed I2C
EEPROM, but with a sticker containing 5 MAC addresses on the bottom of
the board. It doesn't have a clear correspondence between MAC addresses
and their intended use, although I suspect that one MAC address is
intended for each RJ45 port (although that isn't how I use them).
For the QIXIS Development Boards, I have no clue, it's probably even
nonsensical to talk about MAC address reservations since there is just
one onboard Ethernet port (RGMII) and the rest is routed via SERDES to
PCIe slots, to pluggable riser cards, from which Linux/U-Boot don't
bother
too much to read back any info, even though I can't exclude something
like an EEPROM may be available on those cards too. In any case, I
think
QDS boards don't leave the lab, so it doesn't matter too much.
The way I use the MAC addresses from the sticker of my RDBs, on a day
to
day basis, is:
ethaddr (eno0) - #1
eth1addr (eno2) - #2
eth2addr (swp0) - #2
eth3addr (swp1) - #2
eth4addr (swp2) - #2
eth5addr (swp3) - #2
Ah, I never thought of handing out the same MAC address.
And now I'm adding these new env variables:
eth6addr (swp4) - #2
eth7addr (swp5) - #2
eth8addr (eno3) - #3
So I still have 2 more unique MAC addresses to burn through.
I guess, they being evaluation boards you don't care? ;)
I do care a bit, but not that much.
On the Kontron sl28 boards we reserve just 8 and that is
already a lot for a board with max 6 out facing ports. 4 of
these ports used to be a switch, so in theory it should work
/used/ to be a switch? What happened to them? Details? Or you mean
"4 ports are used as a switch"?
I shouldn't probably write mails right before going to sleep.
Yes it should read "the 4 ports (swp0..swp3) are usually configured
to as a switch."
with 3 MAC addresses, right?
Which 3 MAC addresses would those be? Not sure I'm following. enetc #0,
enetc #1, enetc #2? That could work, multiple DSA user ports can share
the same MAC address (inherited from the DSA master or not) and things
would work just fine unless you connect them to each other.
enetc #0, #1 and swp0. As you mentioned, swp1..3 should inherit the
address from swp0 then if swp0 is added as the first device, right?
So why would enetc#2 (or #3) need a non-random mac address? I must
be missing something here.
Or even just 2 if there is no need to terminate any traffic on the
switch interfaces.
And here, which 2? enetc #0 and enetc #1?
Yes. The switch would just be a dumb ethernet switch.
Anyway, do we really need so many addresses?
idk, who's "we" and what does "need" mean? (serious questions)
We as in the users of the ls1028a SoC. And as I said, I thought
of *unique* MAC addresses.
I'm not sure I can give you any answer to this question. As an engineer
working with the kernel, I need to roll the LS1028A Ethernet around on
all its sides. The Linux RDB/QDS support will inevitably reflect what
we
need to test. Everybody else will have a fixed configuration, and the
user reviews will vary from 'internet works! 5 stars!' to 'internet
doesn't work! 1 star!'.
To offer that quality of service for all front-facing ports, you don't
need much. I know of a 12 port industrial switch that entered
production
with 1 MAC address, the "termination" address. It's fine, when it's
marketed as a switch, people come to expect that and don't wonder too
much.
What are the configurations here? For what is the address of the
internal ports used?
By internal ports you mean swp4/swp5, or eno2/eno3?
eno2/eno3.
If eno2/eno3, then a
configuration where having MAC addresses on these interfaces is useful
to me is running some of the kselftests on the LS1028A-RDB, which does
not have enough external enetc ports for 2 loopback pairs, so I do
this, thereby having 1 external loopback through a cable, and 1
internal
loopback in the SoC:
./psfp.sh eno0 swp0 swp4 eno2
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/drivers/net/ocelot/psfp.sh
Speaking of kselftests, it actually doesn't matter that much what the
MAC addresses *are*, since we don't enter any network, just loop back
traffic. In fact we have an environment variable STABLE_MAC_ADDRS,
which
when set, configures the ports to use some predetermined MAC addresses.
There are other configurations where it is useful for eno2 to see DSA
untagged traffic. These are downstream 802.1CB (where this hardware can
offload redundant streams in the forwarding plane, but not in the
termination plane, so we use eno2 as forwarding plane, for
termination),
I'm not that familiar with 802.1CB. Is this MAC address visible outside
of the switch or can it be a random one?
DPDK on eno2 (which mainline Linux doesn't care about), and vfio-pci +
QEMU, where DSA switch control still belongs to the Linux host, but the
guest has 'internet'.
For me, all of that is kind of a trade off. If you want to use
virtual interfaces, you might need to borrow a MAC address from
one of the switch ports (where you have 3 unique addresses left
if you combine all 4 ports to one bridge).
Let's say we are in the "port extender mode" and use the
second internal port as an actual switch port, that would
then be:
2x external enetc
1x internal enetc
4x external switch ports in port extender mode
Which makes 7 addresses. The internal enetc port doesn't
really make sense in a port extender mode, because there
is no switching going on.
It can make sense. You can run ptp4l -i eno2, and ptp4l -i swp4, as
separate processes, and you can get high quality synchronization
between
/dev/ptp0 (enetc) and /dev/ptp1 (felix) over internal Ethernet (there
isn't any other mechanism in the SoC to keep them in sync if that is
needed for some use case like a boundary_clock_jbod between eno0 + eno1
+ swp0-swp3).
Ok, could make sense.
So uhm, 6 addresses are the maximum?
No, the maximum is given by the number of ports, PFs and VFs. But
that's
a high number. It's the theoretical maximum. Then there's the practical
maximum, which is given by what kind of embedded system is built with
it.
I think that the more general-purpose the system is, the more garden
variety the networking use cases will be. I also think it would be very
absurd for everybody to reserve a number of MAC addresses equal to the
number of possibilities in which the LS1028A can expose IP termination
points, if they're likely to never need them.
I think we are on the same track here. I was ignoring any VFs for now.
So I guess, what I'm still missing here is why enet#2 and enet#3 (or
even swp4 and swp5) would need a non-random MAC address. Except from
your example above. Considering the usecase where swp0..3 is one bridge
with eno2 and eno3 being the CPU ports. Then I'd only need a unique
MAC address for eno0, eno1 and swp0, correct?
-michael
This is the MAC address distribution for now on the
sl28 boards:
https://lore.kernel.org/linux-devicetree/20220901221857.2600340-19-michael@xxxxxxxx/
Please tell me if I'm missing something here.
My 2 cents, if you don't need anything special like in-SoC PTP,
802.1CB,
virtualization, and don't habitually connect ports of the same ports to
each other or do some other sorts of redundant networking without
VLANs,
then there isn't too much wrong with one MAC address per RJ45 port, but
best discuss with those who are actually marketing the devices.
