> From: Jason Gunthorpe <jgg@xxxxxxxxxx> > Sent: Tuesday, January 11, 2022 1:53 AM > > > - It's necessary to support existing HW though it may only supports > > optional migration due to unbounded time of stopping DMA; > > At a minimum a device with optional migration needs to be reported to > userspace and qemu should not blindly adopt it without some opt-in > IMHO. yes > > > - We should influence IP designers to design HW to allow preempting > > in-fly requests and stop DMA quickly (also implying the capability of > > aborting/resuming in-fly PRI requests); > > Yes, I think we need a way to suspend the device then abort its PRIs > with some error. The ATS cache is not something that is migrated so > this seems reasonable. yes, any cache can be abandoned in the migration process. and such error must be recoverable otherwise it would break the guest functionality. The key is that PRI can happen at any time which implies that the device must be able to preempt and abort a transaction in very fine-grained granularity. There might be IP-specific factors affecting the final preemption granularity (e.g. complex GPUs will certainly have more challenges), which is what we need involve IP designers to fully understand. > > The only sketchy bit looks like how to resync the VM that still would > have a PRI in its queue and would still want to answer it. That answer > would have to be discarded.. Yes. This also suggests that iommu core needs allow the driver to cancel all previously-queued PRI requests for its device so the guest answer to that device can be ignored by the iommu core. > > > - Specific to the device state management uAPI, it should not assume > > a specific usage and instead allow the user to set a timeout value so > > transitioning to NDMA is failed if the operation cannot be completed > > within the specified timeout value. If the user doesn't set it, the > > migration driver could conservatively use a default timeout value to > > gate any potentially unbounded operation. > > This would need to go along with the flag above, as only optional > drivers should have something like this. > Agree. Thanks Kevin
