> -----Original Message----- > From: Steve Wise [mailto:swise@xxxxxxxxxxxxxxxxxxxxx] > Sent: Thursday, July 17, 2014 2:56 PM > To: 'Hefty, Sean'; 'Shirley Ma'; 'Devesh Sharma'; 'Roland Dreier' > Cc: 'linux-rdma@xxxxxxxxxxxxxxx'; 'chuck.lever@xxxxxxxxxx' > Subject: RE: [for-next 1/2] xprtrdma: take reference of rdma provider module > > > > > -----Original Message----- > > From: Hefty, Sean [mailto:sean.hefty@xxxxxxxxx] > > Sent: Thursday, July 17, 2014 2:50 PM > > To: Steve Wise; 'Shirley Ma'; 'Devesh Sharma'; 'Roland Dreier' > > Cc: linux-rdma@xxxxxxxxxxxxxxx; chuck.lever@xxxxxxxxxx > > Subject: RE: [for-next 1/2] xprtrdma: take reference of rdma provider module > > > > > > So the rdma cm is expected to increase the driver reference count > > > (try_module_get) for > > > > each new cm id, then deference count (module_put) when cm id is > > > destroyed? > > > > > > > > > > No, I think he's saying the rdma-cm posts a RDMA_CM_DEVICE_REMOVAL event > > > to each > > > application with rdmacm objects allocated, and each application is expected > > > to destroy all > > > the objects it has allocated before returning from the event handler. > > > > This is almost correct. The applications do not have to destroy all the objects that it has > > allocated before returning from their event handler. E.g. an app can queue a work item > > that does the destruction. The rdmacm will block in its ib_client remove handler until all > > relevant rdma_cm_id's have been destroyed. > > > > Thanks for the clarification. > And looking at xprtrdma, it does handle the DEVICE_REMOVAL event in rpcrdma_conn_upcall(). It sets ep->rep_connected to -ENODEV, wakes everybody up, and calls rpcrdma_conn_func() for that endpoint, which schedules rep_connect_worker... and I gave up following the code path at this point... :) For this to all work correctly, it would need to destroy all the QPs, MRs, CQs, etc for that device _before_ destroying the rdma cm ids. Otherwise the provider module could be unloaded too soon... Steve. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
