On Tue, Oct 26, 2021 at 08:52:10AM -0600, Alex Williamson wrote: > On Tue, 26 Oct 2021 09:13:53 -0300 > Jason Gunthorpe <jgg@xxxxxxxxxx> wrote: > > > On Tue, Oct 26, 2021 at 09:40:34AM +0100, Dr. David Alan Gilbert wrote: > > > * Jason Gunthorpe (jgg@xxxxxxxxxx) wrote: > > > > On Mon, Oct 25, 2021 at 07:47:29PM +0100, Dr. David Alan Gilbert wrote: > > > > > > > > > It may need some further refinement; for example in that quiesed state > > > > > do counters still tick? will a NIC still respond to packets that don't > > > > > get forwarded to the host? > > > > > > > > At least for the mlx5 NIC the two states are 'able to issue outbound > > > > DMA' and 'all internal memories and state are frozen and unchanging'. > > > > > > Yeh, so my point was just that if you're adding a new state to this > > > process, you need to define the details like that. > > > > We are not planning to propose any patches/uAPI specification for this > > problem until after the mlx5 vfio driver is merged.. > > I'm not super comfortable with that. If we're expecting to add a new > bit to define a quiescent state prior to clearing the running flag and > this is an optional device feature that userspace migration needs to be > aware of and it's really not clear from a hypervisor when p2p DMA might > be in use, I think that leaves userspace in a pickle how and when > they'd impose restrictions on assignment with multiple assigned > devices. It's likely that the majority of initial use cases wouldn't > need this feature, which would make it difficult to arbitrarily impose > later. Either supporting a freeze/quiesce split is an optional feature, and current mlx5_vfio is an example of a driver that does not implement it Or, freeze/quiesce is a mandatory HW feature and no VFIO driver will be merged that doesn't implement it - even if the HW cannot support it. Frankly, I think we will see HW that doesn't support this and VFIO will be required to have it as an optional feature What does userspace do? I don't know - but it does need to be aware if the optional HW feature is available and it does need to make choices if the VM will be migratable or not. I don't see any path where userspace truly cannot care unless HW support for this is mandatory. Jason
