On Mon, Sep 27, 2021 at 03:09:44PM +0300, Leon Romanovsky wrote: > On Sun, Sep 26, 2021 at 05:36:01PM +0000, Chuck Lever III wrote: > > Hi Leon- > > > > Thanks for the suggestion! More below. > > > > > On Sep 26, 2021, at 4:02 AM, Leon Romanovsky <leon@xxxxxxxxxx> wrote: > > > > > > On Fri, Sep 24, 2021 at 03:34:32PM +0000, bugzilla-daemon@xxxxxxxxxxxxxxxxxxx wrote: > > >> https://bugzilla.kernel.org/show_bug.cgi?id=214523 > > >> > > >> Bug ID: 214523 > > >> Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP > > >> updates during a reconnect > > >> Product: Drivers > > >> Version: 2.5 > > >> Kernel Version: 5.14 > > >> Hardware: All > > >> OS: Linux > > >> Tree: Mainline > > >> Status: NEW > > >> Severity: normal > > >> Priority: P1 > > >> Component: Infiniband/RDMA > > >> Assignee: drivers_infiniband-rdma@xxxxxxxxxxxxxxxxxxxx > > >> Reporter: kolga@xxxxxxxxxx > > >> Regression: No > > >> > > >> RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During > > >> the setup the code uses hard coded timeout/retry values. These values are used > > >> for when Connect Request is not being answered to to re-try the request. During > > >> the re-try attempts the ARP updates of the destination server are ignored. > > >> Current timeout values lead to 4+minutes long attempt at connecting to a server > > >> that no longer owns the IP since the ARP update happens. > > >> > > >> The ask is to make the timeout/retry values configurable via procfs or sysfs. > > >> This will allow for environments that use RoCE to reduce the timeouts to a more > > >> reasonable values and be able to react to the ARP updates faster. Other CMA > > >> users (eg IB or others) can continue to use existing values. > > > > I would rather not add a user-facing tunable. The fabric should > > be better at detecting addressing changes within a reasonable > > time. It would be helpful to provide a history of why the ARP > > timeout is so lax -- do certain ULPs rely on it being long? > > I don't know about ULPs and ARPs, but how to calculate TimeWait is > described in the spec. > > Regarding tunable, I agree. Because it needs to be per-connection, most > likely not many people in the world will success to configure it properly. Maybe we should be disconnecting the cm_id if a gratituous ARP changes the MAC address? The cm_id is surely broken after that event right? Jason
