On 9/27/2021 9:32 PM, Haakon Bugge wrote:
External email: Use caution opening links or attachments
On 27 Sep 2021, at 15:10, Jason Gunthorpe <jgg@xxxxxxxx> wrote:
On Mon, Sep 27, 2021 at 08:55:19PM +0800, Mark Zhang wrote:
On 9/27/2021 8:24 PM, Jason Gunthorpe wrote:
External email: Use caution opening links or attachments
On Mon, Sep 27, 2021 at 03:09:44PM +0300, Leon Romanovsky wrote:
On Sun, Sep 26, 2021 at 05:36:01PM +0000, Chuck Lever III wrote:
Hi Leon-
Thanks for the suggestion! More below.
On Sep 26, 2021, at 4:02 AM, Leon Romanovsky <leon@xxxxxxxxxx> wrote:
On Fri, Sep 24, 2021 at 03:34:32PM +0000, bugzilla-daemon@xxxxxxxxxxxxxxxxxxx wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=214523
Bug ID: 214523
Summary: RDMA Mellanox RoCE drivers are unresponsive to ARP
updates during a reconnect
Product: Drivers
Version: 2.5
Kernel Version: 5.14
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Infiniband/RDMA
Assignee: drivers_infiniband-rdma@xxxxxxxxxxxxxxxxxxxx
Reporter: kolga@xxxxxxxxxx
Regression: No
RoCE RDMA connection uses CMA protocol to establish an RDMA connection. During
the setup the code uses hard coded timeout/retry values. These values are used
for when Connect Request is not being answered to to re-try the request. During
the re-try attempts the ARP updates of the destination server are ignored.
Current timeout values lead to 4+minutes long attempt at connecting to a server
that no longer owns the IP since the ARP update happens.
The ask is to make the timeout/retry values configurable via procfs or sysfs.
This will allow for environments that use RoCE to reduce the timeouts to a more
reasonable values and be able to react to the ARP updates faster. Other CMA
users (eg IB or others) can continue to use existing values.
I would rather not add a user-facing tunable. The fabric should
be better at detecting addressing changes within a reasonable
time. It would be helpful to provide a history of why the ARP
timeout is so lax -- do certain ULPs rely on it being long?
I don't know about ULPs and ARPs, but how to calculate TimeWait is
described in the spec.
Regarding tunable, I agree. Because it needs to be per-connection, most
likely not many people in the world will success to configure it properly.
Maybe we should be disconnecting the cm_id if a gratituous ARP changes
the MAC address? The cm_id is surely broken after that event right?
Is there an event on gratuitous ARP? And we also need to notify user-space
application, right?
I think there is a net notifier for this?
NETEVENT_NEIGH_UPDATE may be?
How about do it like this:
1. In cma.c we do register_netevent_notifier();
2. On each NETEVENT_NEIGH_UPDATE event, in netevent_callback():
2.1. Allocate a work (as seems the cb is in interrupt context);
2.2. In the new work:
foreach(cm_dev) {
foreach(id_priv) {
if ((id_priv.dst_ip == event.ip) &&
(id_priv.dst_addr != event.ha)) {
/* Anything more to do? */
report_event(RDMA_CM_EVENT_ADDR_CHANGE);
}
}
}
And I have these questions:
1. Should we do it in cma.c or cm.c?
2. Should we do register only once, or per id? If we do register per id
then there maybe many ids;
3. If we do it in cm.c, then should we do more like ib_cancel_mad()?
Or report an event is enough?
4. Need to create a work on each ARP event, would it be a heavy load?
5. Do we need a new event, instead of RDMA_CM_EVENT_ADDR_CHANGE?
6. How about if peer is not in same subnet?
Thank you very much.
