On Wed, Jan 26, 2022 at 01:17:26AM +0000, Tian, Kevin wrote: > > From: Jason Gunthorpe <jgg@xxxxxxxxxx> > > Sent: Tuesday, January 25, 2022 9:12 PM > > > > On Tue, Jan 25, 2022 at 03:55:31AM +0000, Tian, Kevin wrote: > > > > From: Jason Gunthorpe <jgg@xxxxxxxxxx> > > > > Sent: Saturday, January 15, 2022 3:35 AM > > > > + * > > > > + * The peer to peer (P2P) quiescent state is intended to be a quiescent > > > > + * state for the device for the purposes of managing multiple devices > > > > within > > > > + * a user context where peer-to-peer DMA between devices may be > > active. > > > > The > > > > + * PRE_COPY_P2P and RUNNING_P2P states must prevent the device > > from > > > > + * initiating any new P2P DMA transactions. If the device can identify > > P2P > > > > + * transactions then it can stop only P2P DMA, otherwise it must stop > > all > > > > + * DMA. The migration driver must complete any such outstanding > > > > operations > > > > + * prior to completing the FSM arc into either P2P state. > > > > + * > > > > > > Now NDMA is renamed to P2P... but we did discuss the potential > > > usage of using this state on devices which cannot stop DMA quickly > > > thus needs to drain pending page requests which further requires > > > running vCPUs if the fault is on guest I/O page table. > > > > I think this needs to be fleshed out more before we can add it, > > ideally along with a driver and some qemu implementation > > Yes. We have internal implementation but it has to be cleaned up > based on this new proposal. > > > > > It looks like the qemu part for this will not be so easy.. > > > > My point is that we know that usage in the radar (though it needs more > discussion with real example) then does it make sense to make the > current name more general? I'm not sure how many devices can figure > out P2P from normal DMAs. If most devices have to stop all DMAs to > meet the requirement, calling it a name about stopping all DMAs doesn't > hurt the current P2P requirement and is more extensible to cover other > stop-dma requirements. Except you are not talking about stopping all DMAs, you are talking about a state that might hang indefinately waiting for a vPRI to complete In my mind this is completely different, and may motivate another state in the graph PRE_COPY -> PRE_COPY_STOP_PRI -> PRE_COPY_STOP_P2P -> STOP_COPY As STOP_PRI can be defined as halting any new PRIs and always return immediately. STOP_P2P can hang if PRI's are open This affords a pretty clean approach for userspace to conclude the open PRIs or decide it has to give up the migration. Theoretical future devices that can support aborting PRI would not use this state and would have STOP_P2P as also being NO_PRI. On this device userspace would somehow abort the PRIs when it reaches STOP_COPY. Or at least that is one possibility. In any event, the v2 is built as Alex and Cornelia were suggesting with a minimal base feature set and two optional extensions for P2P and PRE_COPY. Adding a 3rd extension for vPRI is completely reasonable. Further, from what I can understand devices doing PRI are incompatible with the base feature set anyhow, as they can not support a RUNNING -> STOP_COPY transition without, minimally, completing all the open vPRIs. As VMMs implementing the base protocol should stop the vCPU and then move the device to STOP_COPY, it is inherently incompatible with what you are proposing. The new vPRI enabled protocol would have to superceed the base protocol and eliminate implicit transitions through the VPRI maintenance states as these are non-transparent. It is all stuff we can do in the FSM model, but it all needs a careful think and a FSM design. (there is also the interesting question how to even detect this as vPRI special cases should only even exist if the device was bound to a PRI capable io page table, so a single device may or may not use this depending, and at least right now things are assuming these flags are static at device setup time, so hurm) Jason
