> On Mar 30, 2018, at 9:22 AM, Steve Wise <swise@xxxxxxxxxxxxxxxxxxxxx> wrote: > >>> On 3/29/2018 4:01 PM, Chuck Lever wrote: >>>> Hi- >>>> >>>> I've been experimenting with the DEVICE_REMOVAL event on the >>>> NFS/RDMA server, and noticed that there is a DEVICE_REMOVAL >>>> event for IDs associated with a remote connection, but there >>>> does not seem to be a a DEVICE_REMOVAL upcall for listener IDs. >>>> >>>> Is this correct API behavior? >>>> >>>> -- >>>> Chuck Lever >>>> >>> >>> I think you will see DEVICE_REMOVAL events only if the listen was done >>> on a cm_id that was bound to a specific address and thus actually is >>> associated with a device. Thus listening cm_ids bound to INADDR_ANY >>> won't get DEVICE_REMOVAL events. The idea is that if the app listens on >>> INADDR_ANY, it doesn't need to know if a single device comes or goes... >> >> Thanks Steve. >> >> Then a listener ID has no hardware resources associated >> with it? > > A listener ID bound to INADDR_ANY is not directly bound to any device, so no hw resources. However, the CMA creates internal IDs that are bound to each available device and setup to listen. So those internally-managed IDs get destroyed or added as rdma devices are removed or inserted. > > EG: let's say you have a mlx5 and a cxgb4 device up and active. 3 IDs are created when an application listens on INADDR_ANY: The ID created by the ULP which is the parent ID, and 2 internal child IDs created by the CMA, one listening on mlx5 and one listening on cxgb4. Now if cxgb4 is removed, the CMA gets a DEVICE_REMOVAL ib_client event on that ID. Since the ID is internal, the CMA doesn't pass a DEVICE_REMOVAL up to the ULP. However that child/internal ID is destroyed since cxgb4 is going away. At that point there are now 2 IDs: The ULP's ID and the mlx5 ID. If cxgb4 gets inserted again, then we'll end up with 3 IDs again. > >> >> Here's a dumb follow-up question: What about listeners on >> specific addresses? >> > > If the ULP binds to a particular address, then that ID is bound to the underlying device, and I believe it will get the DEVICE_REMOVAL event in that case. > > I encourage you to confirm this by observation. 😊 Are our kernel ULPs smart enough to deal with the difference? I don't think the transport architecture of the NFS server can. Once a listener transport is destroyed, the NFS service has to be restarted to get it back. That's disruptive for all users of that service, and it's different behavior than if the service was available via an INADDR_ANY listener. I'll see if I can manufacture a test case to confirm whether a listener on a specific address might get DEVICE_REMOVAL. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
