On Tue, Jan 19, 2021 at 6:57 PM <mwilck@xxxxxxxx> wrote:
>
> From: Martin Wilck <mwilck@xxxxxxxx>
>
> This reverts commit b2d2440430c0fdd5e0cad3efd6d1c9e3d3d02e5b.
>
> It's true that creating rxe on top of 802.1q interfaces doesn't work.
> Thus, commit fd49ddaf7e26 ("RDMA/rxe: prevent rxe creation on top of vlan interface")
> was absolutely correct.
>
> But b2d2440430c0 was incorrect assuming that with this change,
> RDMA and VLAN don't work togehter at all. It just has to be
> set up differently. Rather than creating rxe on top of the VLAN
> interface, rxe must be created on top of the physical interface.
> RDMA then works just fine through VLAN interfaces on top of that
> physical interface, via the "upper device" logic.
I read this commit log for several times. I can not get you.
Can you show me by an example?
Zhu Yanjun
>
> I've tested this mainly with NVMe over RDMA and rping, but I don't
> see why it wouldn't work just as well for other protocols. If there
> are real issues, I'd like to know.
>
> b2d2440430c0 broke this setup deliberately and should thus be
> reverted.
>
> Fixes: b2d2440430c0 ("RDMA/rxe: Remove VLAN code leftovers from RXE")
>
> Cc: Zhu Yanjun <zyjzyj2000@xxxxxxxxx>
> Cc: Mohammad Heib <goody698@xxxxxxxxx>
> Cc: Vijay Immanuel <vijayi@xxxxxxxxxxxxxxxxx>
> Cc: Nicolas Morey-Chaisemartin <nmoreychaisemartin@xxxxxxxx>
> Signed-off-by: Martin Wilck <mwilck@xxxxxxxx>
>
> ---
> Note: I'm currently not subscribed to linux-rdma.
>
> ---
> drivers/infiniband/sw/rxe/rxe_net.c | 18 ++++++++++++++++++
> drivers/infiniband/sw/rxe/rxe_resp.c | 5 +++++
> 2 files changed, 23 insertions(+)
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c
> index c4b06ced30a7..34bef7d8e6b4 100644
> --- a/drivers/infiniband/sw/rxe/rxe_net.c
> +++ b/drivers/infiniband/sw/rxe/rxe_net.c
> @@ -8,6 +8,7 @@
> #include <linux/if_arp.h>
> #include <linux/netdevice.h>
> #include <linux/if.h>
> +#include <linux/if_vlan.h>
> #include <net/udp_tunnel.h>
> #include <net/sch_generic.h>
> #include <linux/netfilter.h>
> @@ -19,6 +20,18 @@
>
> static struct rxe_recv_sockets recv_sockets;
>
> +struct device *rxe_dma_device(struct rxe_dev *rxe)
> +{
> + struct net_device *ndev;
> +
> + ndev = rxe->ndev;
> +
> + if (is_vlan_dev(ndev))
> + ndev = vlan_dev_real_dev(ndev);
> +
> + return ndev->dev.parent;
> +}
> +
> int rxe_mcast_add(struct rxe_dev *rxe, union ib_gid *mgid)
> {
> int err;
> @@ -153,9 +166,14 @@ static int rxe_udp_encap_recv(struct sock *sk, struct sk_buff *skb)
> {
> struct udphdr *udph;
> struct net_device *ndev = skb->dev;
> + struct net_device *rdev = ndev;
> struct rxe_dev *rxe = rxe_get_dev_from_net(ndev);
> struct rxe_pkt_info *pkt = SKB_TO_PKT(skb);
>
> + if (!rxe && is_vlan_dev(rdev)) {
> + rdev = vlan_dev_real_dev(ndev);
> + rxe = rxe_get_dev_from_net(rdev);
> + }
> if (!rxe)
> goto drop;
>
> diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
> index 5a098083a9d2..c7e3b6a4af38 100644
> --- a/drivers/infiniband/sw/rxe/rxe_resp.c
> +++ b/drivers/infiniband/sw/rxe/rxe_resp.c
> @@ -872,6 +872,11 @@ static enum resp_states do_complete(struct rxe_qp *qp,
> else
> wc->network_hdr_type = RDMA_NETWORK_IPV6;
>
> + if (is_vlan_dev(skb->dev)) {
> + wc->wc_flags |= IB_WC_WITH_VLAN;
> + wc->vlan_id = vlan_dev_vlan_id(skb->dev);
> + }
> +
> if (pkt->mask & RXE_IMMDT_MASK) {
> wc->wc_flags |= IB_WC_WITH_IMM;
> wc->ex.imm_data = immdt_imm(pkt);
> --
> 2.29.2
>
