On Tue, Aug 13, 2024 at 09:26:55PM -0700, Christoph Hellwig wrote: > On Tue, Aug 13, 2024 at 01:05:02PM -0300, Jason Gunthorpe wrote: > > > Where do we block driver unbind with an open resource? > > > > I keep seeing it in different subsystems, safe driver unbind is really > > hard. :\ eg I think VFIO has some waits in it > > Well, waits for what? Looks like it waits for the fds to close at least. It has weird handshake where it notifies userspace that the device is closing and the userspace needs to close the fd. See vfio_unregister_group_dev() In the past VFIO couldn't do anything else because it had VMAs open that point to the device's mmio and it would be wrong to unbind the driver and leave those uncleaned. VFIO now knows how to zap VMA so maybe this could be improved, but it looks like a sizable uplift. > Usuaully an unbind has to wait for something, but waiting for > unbound references is always a bug. And I can't think of any > sensible subsystem doing this. I've seen several cases over the years.. It can be hard in many cases to deal with the issue. Like the VMA's above for instance. rdma also had to learn how to do zap before it could do fast unbind. Some places just have bugs where they don't wait and Weird Stuff happens. There is a strange misconception that module refcounts protect against unbind :\ Not saying this is good design, or desirable, just pointing out we seem to tolerate this in enough other places. In this case the only resolution I could think of would be to have the rdma stack somehow register a notifier on the pgmap. Then we could revoke the RDMA access synchronously during destruction and return those refcounts. That does seems a bit complicated though.. Jason
