From: Edward Cree <ecree.xilinx@xxxxxxxxx>
There's no clear explanation of what VF Representors are for, their
semantics, etc., outside of vendor docs and random conference slides.
Add a document explaining Representors and defining what drivers that
implement them are expected to do.
Signed-off-by: Edward Cree <ecree.xilinx@xxxxxxxxx>
---
This documents representors as I understand them, but I suspect others
(including other vendors) might disagree (particularly with the "what
functions should have a rep" section). I'm hoping that through review
of this doc we can converge on a consensus.
Documentation/networking/index.rst | 1 +
Documentation/networking/representors.rst | 219 ++++++++++++++++++++++
Documentation/networking/switchdev.rst | 1 +
3 files changed, 221 insertions(+)
create mode 100644 Documentation/networking/representors.rst
diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst
index 03b215bddde8..c37ea2b54c29 100644
--- a/Documentation/networking/index.rst
+++ b/Documentation/networking/index.rst
@@ -93,6 +93,7 @@ Contents:
radiotap-headers
rds
regulatory
+ representors
rxrpc
sctp
secid
diff --git a/Documentation/networking/representors.rst b/Documentation/networking/representors.rst
new file mode 100644
index 000000000000..4d28731a5b5b
--- /dev/null
+++ b/Documentation/networking/representors.rst
@@ -0,0 +1,219 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=============================
+Network Function Representors
+=============================
+
+This document describes the semantics and usage of representor netdevices, as
+used to control internal switching on SmartNICs. For the closely-related port
+representors on physical (multi-port) switches, see
+:ref:`Documentation/networking/switchdev.rst <switchdev>`.
+
+Motivation
+----------
+
+Since the mid-2010s, network cards have started offering more complex
+virtualisation capabilities than the legacy SR-IOV approach (with its simple
+MAC/VLAN-based switching model) can support. This led to a desire to offload
+software-defined networks (such as OpenVSwitch) to these NICs to specify the
+network connectivity of each function. The resulting designs are variously
+called SmartNICs or DPUs.
+
+Network function representors provide the mechanism by which network functions
+on an internal switch are managed. They are used both to configure the
+corresponding function ('representee') and to handle slow-path traffic to and
+from the representee for which no fast-path switching rule is matched.
+
+That is, a representor is both a control plane object (representing the function
+in administrative commands) and a data plane object (one end of a virtual pipe).
+As a virtual link endpoint, the representor can be configured like any other
+netdevice; in some cases (e.g. link state) the representee will follow the
+representor's configuration, while in others there are separate APIs to
+configure the representee.
+
+What does a representor do?
+---------------------------
+
+A representor has three main rôles.
+
+1. It is used to configure the representee's virtual MAC, e.g. link up/down,
+ MTU, etc. For instance, bringing the representor administratively UP should
+ cause the representee to see a link up / carrier on event.
+2. It provides the slow path for traffic which does not hit any offloaded
+ fast-path rules in the virtual switch. Packets transmitted on the
+ representor netdevice should be delivered to the representee; packets
+ transmitted to the representee which fail to match any switching rule should
+ be received on the representor netdevice. (That is, there is a virtual pipe
+ connecting the representor to the representee, similar in concept to a veth
+ pair.)
+ This allows software switch implementations (such as OpenVSwitch or a Linux
+ bridge) to forward packets between representees and the rest of the network.
+3. It acts as a handle by which switching rules (such as TC filters) can refer
+ to the representee, allowing these rules to be offloaded.
+
+The combination of 2) and 3) means that the behaviour (apart from performance)
+should be the same whether a TC filter is offloaded or not. E.g. a TC rule
+on a VF representor applies in software to packets received on that representor
+netdevice, while in hardware offload it would apply to packets transmitted by
+the representee VF. Conversely, a mirred egress redirect to a VF representor
+corresponds in hardware to delivery directly to the representee VF.
+
+What functions should have a representor?
+-----------------------------------------
+
+Essentially, for each virtual port on the device's internal switch, there
+should be a representor.
+The only exceptions are the management PF (whose port is used for traffic to
+and from all other representors) and perhaps the physical network port (for
+which the management PF may act as a kind of port representor. Devices that
+combine multiple physical ports and SR-IOV capability may need to have port
+representors in addition to PF/VF representors).
+
+Thus, the following should all have representors:
+
+ - VFs belonging to the management PF.
+ - Other PFs on the PCIe controller, and any VFs belonging to them.
+ - PFs and VFs on other PCIe controllers on the device (e.g. for any embedded
+ System-on-Chip within the SmartNIC).
+ - PFs and VFs with other personalities, including network block devices (such
+ as a vDPA virtio-blk PF backed by remote/distributed storage).
+ - Subfunctions (SFs) belonging to any of the above PFs or VFs, if they have
+ their own port on the switch (as opposed to using their parent PF's port).
+ - Any accelerators or plugins on the device whose interface to the network is
+ through a virtual switch port, even if they do not have a corresponding PCIe
+ PF or VF.
+
+This allows the entire switching behaviour of the NIC to be controlled through
+representor TC rules.
+
+An example of a PCIe function that should *not* have a representor is, on an
+FPGA-based NIC, a PF which is only used to deploy a new bitstream to the FPGA,
+and which cannot create RX and TX queues. Since such a PF does not have network
+access through the internal switch, not even indirectly via a distributed
+storage endpoint, there is no switch virtual port for the representor to
+configure or to be the other end of the virtual pipe.
+
+How are representors created?
+-----------------------------
+
+The driver instance attached to the management PF should enumerate the virtual
+ports on the switch, and for each representee, create a pure-software netdevice
+which has some form of in-kernel reference to the PF's own netdevice or driver
+private data (``netdev_priv()``).
+If switch ports can dynamically appear/disappear, the PF driver should create
+and destroy representors appropriately.
+The operations of the representor netdevice will generally involve acting
+through the management PF. For example, ``ndo_start_xmit()`` might send the
+packet, specially marked for delivery to the representee, through a TX queue
+attached to the management PF.
+
+How are representors identified?
+--------------------------------
+
+The representor netdevice should *not* directly refer to a PCIe device (e.g.
+through ``net_dev->dev.parent`` / ``SET_NETDEV_DEV()``), either of the
+representee or of the management PF.
+Instead, it should implement the ``ndo_get_port_parent_id()`` and
+``ndo_get_phys_port_name()`` netdevice ops (corresponding to the
+``phys_switch_id`` and ``phys_port_name`` sysfs nodes).
+``ndo_get_port_parent_id()`` should return a string identical to that returned
+by the management PF's ``ndo_get_phys_port_id()`` (typically the MAC address of
+the physical port), while ``ndo_get_phys_port_name()`` should return a string
+describing the representee's relation to the management PF.
+
+For instance, if the management PF has a ``phys_port_name`` of ``p0`` (physical
+port 0), then the representor for the third VF on the second PF should typically
+be ``p0pf1vf2`` (i.e. "port 0, PF 1, VF 2"). More generally, the
+``phys_port_name`` for a PCIe function should be the concatenation of one or
+more of:
+
+ - ``p<N>``, physical port number *N*.
+ - ``if<N>``, PCIe controller number *N*. The semantics of these numbers are
+ vendor-defined, and controller 0 need not correspond to the controller on
+ which the management PF resides.
+ - ``pf<N>``, PCIe physical function index *N*.
+ - ``vf<N>``, PCIe virtual function index *N*.
+ - ``sf<N>``, Subfunction index *N*.
+
+It is expected that userland will use this information (e.g. through udev rules)
+to construct an appropriately informative name or alias for the netdevice. For
+instance if the management PF is ``eth4`` then our representor with a
+``phys_port_name`` of ``p0pf1vf2`` might be renamed ``eth4pf1vf2rep``.
+
+There are as yet no established conventions for naming representors which do not
+correspond to PCIe functions (e.g. accelerators and plugins).
+
+How do representors interact with TC rules?
+-------------------------------------------
+
+Any TC rule on a representor applies (in software TC) to packets received by
+that representor netdevice. Thus, if the delivery part of the rule corresponds
+to another port on the virtual switch, the driver may choose to offload it to
+hardware, applying it to packets transmitted by the representee.
+
+Similarly, since a TC mirred egress action targeting the representor would (in
+software) send the packet through the representor (and thus indirectly deliver
+it to the representee), hardware offload should interpret this as delivery to
+the representee.
+
+As a simple example, if ``eth0`` is the management PF's netdevice and ``eth1``
+is a VF representor, the following rules::
+
+ tc filter add dev eth1 parent ffff: protocol ipv4 flower \
+ action mirred egress redirect dev eth0
+ tc filter add dev eth0 parent ffff: protocol ipv4 flower \
+ action mirred egress mirror dev eth1
+
+would mean that all IPv4 packets from the VF are sent out the physical port, and
+all IPv4 packets received on the physical port are delivered to the VF in
+addition to the management PF.
+
+Of course the rules can (if supported by the NIC) include packet-modifying
+actions (e.g. VLAN push/pop), which should be performed by the virtual switch.
+
+Tunnel encapsulation and decapsulation are rather more complicated, as they
+involve a third netdevice (a tunnel netdev operating in metadata mode, such as
+a VxLAN device created with ``ip link add vxlan0 type vxlan external``) and
+require an IP address to be bound to the underlay device (e.g. management PF or
+port representor). TC rules such as::
+
+ tc filter add dev eth1 parent ffff: flower \
+ action tunnel_key set id $VNI src_ip $LOCAL_IP dst_ip $REMOTE_IP \
+ dst_port 4789 \
+ action mirred egress redirect dev vxlan0
+ tc filter add dev vxlan0 parent ffff: flower enc_src_ip $REMOTE_IP \
+ enc_dst_ip $LOCAL_IP enc_key_id $VNI enc_dst_port 4789 \
+ action tunnel_key unset action mirred egress redirect dev eth1
+
+where ``LOCAL_IP`` is an IP address bound to ``eth0``, and ``REMOTE_IP`` is
+another IP address on the same subnet, mean that packets sent by the VF should
+be VxLAN encapsulated and sent out the physical port (the driver has to deduce
+this by a route lookup of ``LOCAL_IP`` leading to ``eth0``, and also perform an
+ARP/neighbour table lookup to find the MAC addresses to use in the outer
+Ethernet frame), while UDP packets received on the physical port with UDP port
+4789 should be parsed as VxLAN and, if their VSID matches ``$VNI``, decapsulated
+and forwarded to the VF.
+
+If this all seems complicated, just remember the 'golden rule' of TC offload:
+the hardware should ensure the same final results as if the packets were
+processed through the slow path, traversed software TC and were transmitted or
+received through the representor netdevices.
+
+Configuring the representee's MAC
+---------------------------------
+
+The representee's link state is controlled through the representor. Setting the
+representor administratively UP or DOWN should cause carrier ON or OFF at the
+representee.
+
+Setting an MTU on the representor should cause that same MTU to be reported to
+the representee.
+(On hardware that allows configuring separate and distinct MTU and MRU values,
+the representor MTU should correspond to the representee's MRU and vice-versa.)
+
+Currently there is no way to use the representor to set the station permanent
+MAC address of the representee; other methods available to do this include:
+
+ - legacy SR-IOV (``ip link set DEVICE vf NUM mac LLADDR``)
+ - devlink port function (see **devlink-port(8)** and
+ :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`)
diff --git a/Documentation/networking/switchdev.rst b/Documentation/networking/switchdev.rst
index f1f4e6a85a29..21e80c8e661b 100644
--- a/Documentation/networking/switchdev.rst
+++ b/Documentation/networking/switchdev.rst
@@ -1,5 +1,6 @@
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>
+.. _switchdev:
===============================================
Ethernet switch device driver model (switchdev)
