> -----Original Message----- > From: Chuck Lever <chuck.lever@xxxxxxxxxx> > Sent: Friday, March 30, 2018 9:15 AM > To: Steve Wise <swise@xxxxxxxxxxxxxxxxxxxxx> > Cc: linux-rdma <linux-rdma@xxxxxxxxxxxxxxx> > Subject: Re: question re: CM_EVENT_DEVICE_REMOVAL > > > > > On Mar 30, 2018, at 9:22 AM, Steve Wise > <swise@xxxxxxxxxxxxxxxxxxxxx> wrote: > > > >>> On 3/29/2018 4:01 PM, Chuck Lever wrote: > >>>> Hi- > >>>> > >>>> I've been experimenting with the DEVICE_REMOVAL event on the > >>>> NFS/RDMA server, and noticed that there is a DEVICE_REMOVAL > >>>> event for IDs associated with a remote connection, but there > >>>> does not seem to be a a DEVICE_REMOVAL upcall for listener IDs. > >>>> > >>>> Is this correct API behavior? > >>>> > >>>> -- > >>>> Chuck Lever > >>>> > >>> > >>> I think you will see DEVICE_REMOVAL events only if the listen was done > >>> on a cm_id that was bound to a specific address and thus actually is > >>> associated with a device. Thus listening cm_ids bound to INADDR_ANY > >>> won't get DEVICE_REMOVAL events. The idea is that if the app listens > on > >>> INADDR_ANY, it doesn't need to know if a single device comes or goes... > >> > >> Thanks Steve. > >> > >> Then a listener ID has no hardware resources associated > >> with it? > > > > A listener ID bound to INADDR_ANY is not directly bound to any device, > so no hw resources. However, the CMA creates internal IDs that are bound > to each available device and setup to listen. So those internally-managed > IDs get destroyed or added as rdma devices are removed or inserted. > > > > EG: let's say you have a mlx5 and a cxgb4 device up and active. 3 IDs are > created when an application listens on INADDR_ANY: The ID created by the > ULP which is the parent ID, and 2 internal child IDs created by the CMA, one > listening on mlx5 and one listening on cxgb4. Now if cxgb4 is removed, the > CMA gets a DEVICE_REMOVAL ib_client event on that ID. Since the ID is > internal, the CMA doesn't pass a DEVICE_REMOVAL up to the ULP. However > that child/internal ID is destroyed since cxgb4 is going away. At that point > there are now 2 IDs: The ULP's ID and the mlx5 ID. If cxgb4 gets inserted > again, then we'll end up with 3 IDs again. > > > >> > >> Here's a dumb follow-up question: What about listeners on > >> specific addresses? > >> > > > > If the ULP binds to a particular address, then that ID is bound to the > underlying device, and I believe it will get the DEVICE_REMOVAL event in > that case. > > > > I encourage you to confirm this by observation. 😊 > > Are our kernel ULPs smart enough to deal with the difference? > I don't know for sure. iSER and NVMe/F drivers do handle DEVICE_REMOVAL though. For NVMe/F, the CMA device removal events proved difficult to handle correctly, so we ended up using the ib_client API and using that device removal path to free up all resources for that device. Looking at the code quickly, I see the target side uses a combination of both CMA and ib_client device removal events, and the host side just uses the ib_client removal event. Steve. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
