On Fri, Dec 08, 2023 at 01:41:09PM -0400, Jason Gunthorpe wrote: > On Fri, Dec 08, 2023 at 05:38:51PM +0000, Jim Harris wrote: > > On Thu, Dec 07, 2023 at 07:48:10PM -0400, Jason Gunthorpe wrote: > > > > > > The mechanism of waiting in remove for userspace is inherently flawed, > > > it can never work fully correctly. :( I've hit this many times. > > > > > > Upon remove VFIO should immediately remove itself and leave behind a > > > non-functional file descriptor. Userspace should catch up eventually > > > and see it is toast. > > > > One nice aspect of the current design is that vfio will leave the BARs > > mapped until userspace releases the vfio handle. It avoids some rather > > nasty hacks for handling SIGBUS errors in the fast path (i.e. writing > > NVMe doorbells) where we cannot try to check for device removal on > > every MMIO write. Would your proposal immediately yank the BARs, without > > waiting for userspace to respond? This is mostly for my curiosity - SPDK > > already has these hacks implemented, so I don't think it would be > > affected by this kind of change in behavior. > > What we did in RDMA was map a dummy page to the BARs so the sigbus was > avoided. But in that case RDMA knows the BAR memory is used only for > doorbell write so this is a reasonable thing to do. Yeah, this is exactly what SPDK (and DPDK) does today.
