On Mon, Sep 28, 2020 at 07:36:02PM -0300, Jason Gunthorpe wrote:
> On Mon, Sep 28, 2020 at 11:26:31PM +0300, Kamal Heib wrote:
> > Before this patch, the rtnl_link_ops are set only for ipoib network
> > devices that are created via the rtnl_link_ops->newlink() callback, this
> > patch fixes that by setting the rtnl_link_ops for all ipoib network
> > devices. Also, implement the dellink() callback to block users from
> > trying to remove the base ipoib network device while allowing it only
> > for child interfaces.
>
> Why?
>
This is needed to avoid the inconsistent user experience for PKeys that
is created via netlink VS PKeys that is created via sysfs and the based
ipoib interface, as you can see below the ipoib attributes are reported
only for PKeys that is created via netlink in the 'ip -d link show'
output:
PKey created via netlink (pkey, mode, and umcast attributes are present):
$ ip link add link mlx5_ib0 name mlx5_ib0.8001 type ipoib pkey 0x8001
$ ip -d link show dev mlx5_ib0.8001
28: mlx5_ib0.8001@mlx5_ib0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN mode DEFAULT group default qlen 256
link/infiniband 00:00:1f:e3:fe:80:00:00:00:00:00:00:24:8a:07:03:00:a3:19:64 brd 00:ff:ff:ff:ff:12:40:1b:80:01:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65520
ipoib pkey 0x8001 mode datagram umcast 0000 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
While:
PKey created via sysfs (the attributes are not present):
$ ip -d link show dev mlx5_ib0.8002
20: mlx5_ib0.8002@mlx5_ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP mode DEFAULT group default qlen 256
link/infiniband 00:00:11:7b:fe:80:00:00:00:00:00:00:24:8a:07:03:00:a3:19:64 brd 00:ff:ff:ff:ff:12:40:1b:80:02:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65520 addrgenmode none numtxqueues 256 numrxqueues 32 gso_max_size 65536 gso_max_segs 65535
Same for the base interface:
$ ip -d link show dev mlx5_ib0
19: mlx5_ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP mode DEFAULT group default qlen 256
link/infiniband 00:00:11:79:fe:80:00:00:00:00:00:00:24:8a:07:03:00:a3:19:64 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65520 addrgenmode none numtxqueues 256 numrxqueues 32 gso_max_size 65536 gso_max_segs 65535
altname ibp7s0f0
After applying this patch:
$ ip link add link mlx5_ib0 name mlx5_ib0.8001 type ipoib pkey 0x8001
$ ip -d link show dev mlx5_ib0.8001
38: mlx5_ib0.8001@mlx5_ib0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN mode DEFAULT group default qlen 256
link/infiniband 00:00:2e:4e:fe:80:00:00:00:00:00:00:24:8a:07:03:00:a3:19:64 brd 00:ff:ff:ff:ff:12:40:1b:80:01:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65520
ipoib pkey 0x8001 mode datagram umcast 0000 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
$ ip -d link show dev mlx5_ib0.8002
30: mlx5_ib0.8002@mlx5_ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP mode DEFAULT group default qlen 256
link/infiniband 00:00:1f:e6:fe:80:00:00:00:00:00:00:24:8a:07:03:00:a3:19:64 brd 00:ff:ff:ff:ff:12:40:1b:80:02:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65520
ipoib pkey 0x8002 mode datagram umcast 0000 addrgenmode none numtxqueues 256 numrxqueues 32 gso_max_size 65536 gso_max_segs 65535
$ ip -d link show dev mlx5_ib0
29: mlx5_ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP mode DEFAULT group default qlen 256
link/infiniband 00:00:1f:e4:fe:80:00:00:00:00:00:00:24:8a:07:03:00:a3:19:64 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65520
ipoib pkey 0xffff mode datagram umcast 0000 addrgenmode none numtxqueues 256 numrxqueues 32 gso_max_size 65536 gso_max_segs 65535
Also modifying the the ipoib attributes will work only for PKeys that is
created via netlink (for example, setting the mode).
Thanks,
Kamal
> Jason
