On 2/24/2020 12:29 PM, Jason Gunthorpe wrote: > On Mon, Feb 24, 2020 at 12:52:06PM +0200, Leon Romanovsky wrote: >>> Are you asking why bonding should be implemented as dedicated >>> ulp/driver, and not as an extension by the vendor driver? >> >> No, I meant something different. You are proposing to combine IB >> devices, while keeping netdev devices separated. I'm asking if it is >> possible to combine netdev devices with already existing bond driver >> and simply create new ib device with bond netdev as an underlying >> provider. > > Isn't that basically what we do now in mlx5? > And its broken for few aspects that I described in Q&A question-1 in this thread previously. On top of that user has no ability to disable rdma bonding. User exactly asked us that they want to disable in some cases. (not on mailing list). So there are non-upstream hacks exists that is not applicable for this discussion. > Logically the ib_device is attached to the bond, it uses the bond for > IP addressing, etc. > > I'm not sure trying to have 3 ib_devices like netdev does is sane, > that is very, very complicated to get everything to work. The ib stuff > just isn't designed to be stacked like that. > I am not sure I understand all the complications you have thought through. I thought of few and put forward in the Q&A in the thread and we can improve the design as we go forward. Stacking rdma device on top of existing rdma device as an ib_client so that rdma bond device exactly aware of what is going on with slaves and bond netdev. On top of that I am enjoying below list delete corruption call trace for a while now coming from this lag code in a silly devel test of load unload driver 30+ times (not even running traffic). :-) RIP: 0010:__list_del_entry_valid+0x7c/0xa0 unregister_netdevice_notifier_dev_net+0x1f/0x70 mlx5_lag_remove+0x61/0xf0 [mlx5_core] mlx5e_detach_netdev+0x24/0x50 [mlx5_core] mlx5e_detach+0x36/0x40 [mlx5_core] mlx5e_remove+0x48/0x60 [mlx5_core] mlx5_remove_device+0xb0/0xc0 [mlx5_core] mlx5_unregister_interface+0x39/0x90 [mlx5_core] cleanup+0x5/0xdd1 [mlx5_core] And there is some LAG code in a hw driver that reschedules the work if it cannot acquire some mutex lock...
