> -----Original Message----- > From: linux-rdma-owner@xxxxxxxxxxxxxxx [mailto:linux-rdma- > owner@xxxxxxxxxxxxxxx] On Behalf Of Steve Wise > Sent: Friday, July 18, 2014 2:29 AM > To: 'Chuck Lever' > Cc: 'Hefty, Sean'; 'Shirley Ma'; Devesh Sharma; 'Roland Dreier'; linux- > rdma@xxxxxxxxxxxxxxx > Subject: RE: [for-next 1/2] xprtrdma: take reference of rdma provider > module > > > > > -----Original Message----- > > From: linux-rdma-owner@xxxxxxxxxxxxxxx > > [mailto:linux-rdma-owner@xxxxxxxxxxxxxxx] On Behalf Of Chuck Lever > > Sent: Thursday, July 17, 2014 3:42 PM > > To: Steve Wise > > Cc: Hefty, Sean; Shirley Ma; Devesh Sharma; Roland Dreier; > > linux-rdma@xxxxxxxxxxxxxxx > > Subject: Re: [for-next 1/2] xprtrdma: take reference of rdma provider > > module > > > > > > On Jul 17, 2014, at 4:08 PM, Steve Wise <swise@xxxxxxxxxxxxxxxxxxxxx> > wrote: > > > > > > > > > > >> -----Original Message----- > > >> From: Steve Wise [mailto:swise@xxxxxxxxxxxxxxxxxxxxx] > > >> Sent: Thursday, July 17, 2014 2:56 PM > > >> To: 'Hefty, Sean'; 'Shirley Ma'; 'Devesh Sharma'; 'Roland Dreier' > > >> Cc: 'linux-rdma@xxxxxxxxxxxxxxx'; 'chuck.lever@xxxxxxxxxx' > > >> Subject: RE: [for-next 1/2] xprtrdma: take reference of rdma > > >> provider module > > >> > > >> > > >> > > >>> -----Original Message----- > > >>> From: Hefty, Sean [mailto:sean.hefty@xxxxxxxxx] > > >>> Sent: Thursday, July 17, 2014 2:50 PM > > >>> To: Steve Wise; 'Shirley Ma'; 'Devesh Sharma'; 'Roland Dreier' > > >>> Cc: linux-rdma@xxxxxxxxxxxxxxx; chuck.lever@xxxxxxxxxx > > >>> Subject: RE: [for-next 1/2] xprtrdma: take reference of rdma > > >>> provider module > > >>> > > >>>>> So the rdma cm is expected to increase the driver reference > > >>>>> count > > >>>> (try_module_get) for > > >>>>> each new cm id, then deference count (module_put) when cm id is > > >>>> destroyed? > > >>>>> > > >>>> > > >>>> No, I think he's saying the rdma-cm posts a > > >>>> RDMA_CM_DEVICE_REMOVAL event to each application with > rdmacm > > >>>> objects allocated, and each application is expected to destroy > > >>>> all the objects it has allocated before returning from the event > > >>>> handler. > > >>> > > >>> This is almost correct. The applications do not have to destroy > > >>> all the objects > that > > > it has > > >>> allocated before returning from their event handler. E.g. an app > > >>> can queue a work > > > item > > >>> that does the destruction. The rdmacm will block in its ib_client > > >>> remove handler > > > until all > > >>> relevant rdma_cm_id's have been destroyed. > > >>> > > >> > > >> Thanks for the clarification. > > >> > > > > > > And looking at xprtrdma, it does handle the DEVICE_REMOVAL event in > > rpcrdma_conn_upcall(). > > > It sets ep->rep_connected to -ENODEV, wakes everybody up, and calls > > rpcrdma_conn_func() > > > for that endpoint, which schedules rep_connect_worker... and I gave > > > up following the > > code > > > path at this point... :) > > > > > > For this to all work correctly, it would need to destroy all the > > > QPs, MRs, CQs, etc > for > > > that device _before_ destroying the rdma cm ids. Otherwise the > > > provider module could > > be > > > unloaded too soon. > > > > We can't really deal with a CM_DEVICE_REMOVE event while there are > > active NFS mounts. > > > > System shutdown ordering should guarantee (one would hope) that NFS > > mount points are unmounted before the RDMA/IB core infrastructure is > > torn down. Ordering shouldn't matter as long all NFS activity has > > ceased before the CM tries to remove the device. > > > > So if something is hanging up the CM, there's something xprtrdma is > > not cleaning up properly. > > > > > Devesh, how are you reproducing this? Are you just rmmod'ing the ocrdma > module while there are active mounts? Yes, I am issuing rmmod while there is an active mount. In my case rmmod ocrdma remains blocked forever. Off-the-course of this discussion: Is there a reasoning behind not using ib_register_client()/ib_unregister_client() framework? > > > > > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the > body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at > http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
