> On Mar 30, 2018, at 10:17 AM, Steve Wise <swise@xxxxxxxxxxxxxxxxxxxxx> wrote: > > > >> -----Original Message----- >> From: Chuck Lever <chuck.lever@xxxxxxxxxx> >> Sent: Friday, March 30, 2018 9:15 AM >> To: Steve Wise <swise@xxxxxxxxxxxxxxxxxxxxx> >> Cc: linux-rdma <linux-rdma@xxxxxxxxxxxxxxx> >> Subject: Re: question re: CM_EVENT_DEVICE_REMOVAL >> >> >> >>> On Mar 30, 2018, at 9:22 AM, Steve Wise >> <swise@xxxxxxxxxxxxxxxxxxxxx> wrote: >>> >>>>> On 3/29/2018 4:01 PM, Chuck Lever wrote: >>>>>> Hi- >>>>>> >>>>>> I've been experimenting with the DEVICE_REMOVAL event on the >>>>>> NFS/RDMA server, and noticed that there is a DEVICE_REMOVAL >>>>>> event for IDs associated with a remote connection, but there >>>>>> does not seem to be a a DEVICE_REMOVAL upcall for listener IDs. >>>>>> >>>>>> Is this correct API behavior? >>>>>> >>>>>> -- >>>>>> Chuck Lever >>>>>> >>>>> >>>>> I think you will see DEVICE_REMOVAL events only if the listen was done >>>>> on a cm_id that was bound to a specific address and thus actually is >>>>> associated with a device. Thus listening cm_ids bound to INADDR_ANY >>>>> won't get DEVICE_REMOVAL events. The idea is that if the app listens >> on >>>>> INADDR_ANY, it doesn't need to know if a single device comes or goes... >>>> >>>> Thanks Steve. >>>> >>>> Then a listener ID has no hardware resources associated >>>> with it? >>> >>> A listener ID bound to INADDR_ANY is not directly bound to any device, >> so no hw resources. However, the CMA creates internal IDs that are bound >> to each available device and setup to listen. So those internally-managed >> IDs get destroyed or added as rdma devices are removed or inserted. >>> >>> EG: let's say you have a mlx5 and a cxgb4 device up and active. 3 IDs are >> created when an application listens on INADDR_ANY: The ID created by the >> ULP which is the parent ID, and 2 internal child IDs created by the CMA, one >> listening on mlx5 and one listening on cxgb4. Now if cxgb4 is removed, the >> CMA gets a DEVICE_REMOVAL ib_client event on that ID. Since the ID is >> internal, the CMA doesn't pass a DEVICE_REMOVAL up to the ULP. However >> that child/internal ID is destroyed since cxgb4 is going away. At that point >> there are now 2 IDs: The ULP's ID and the mlx5 ID. If cxgb4 gets inserted >> again, then we'll end up with 3 IDs again. >>> >>>> >>>> Here's a dumb follow-up question: What about listeners on >>>> specific addresses? >>>> >>> >>> If the ULP binds to a particular address, then that ID is bound to the >> underlying device, and I believe it will get the DEVICE_REMOVAL event in >> that case. >>> >>> I encourage you to confirm this by observation. 😊 >> >> Are our kernel ULPs smart enough to deal with the difference? >> > > I don't know for sure. iSER and NVMe/F drivers do handle DEVICE_REMOVAL though. For NVMe/F, the CMA device removal events proved difficult to handle correctly, so we ended up using the ib_client API and using that device removal path to free up all resources for that device. Looking at the code quickly, I see the target side uses a combination of both CMA and ib_client device removal events, and the host side just uses the ib_client removal event. I have not had a chance to construct a test case here, but for the NFS/RDMA server, it turns out that all the listeners are INADDR_ANY (or IN6ADDR_ANY) in the current code base. However, NFS's RDMA listeners are always set up in init_net. I've got a patch to fix that up, but I'm wondering if the same arrangement of subordinate IDs is done with an INADDR_ANY listener in a non-init network namespace. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
