On Tue, Sep 28, 2021 at 02:18:44PM -0600, Alex Williamson wrote:
> On Tue, 28 Sep 2021 16:35:50 -0300
> Jason Gunthorpe <jgg@xxxxxxxx> wrote:
>
> > On Tue, Sep 28, 2021 at 01:19:58PM -0600, Alex Williamson wrote:
> >
> > > In defining the device state, we tried to steer away from defining it
> > > in terms of the QEMU migration API, but rather as a set of controls
> > > that could be used to support that API to leave us some degree of
> > > independence that QEMU implementation might evolve.
> >
> > That is certainly a different perspective, it would have been
> > better to not express this idea as a FSM in that case...
> >
> > So each state in mlx5vf_pci_set_device_state() should call the correct
> > combination of (un)freeze, (un)quiesce and so on so each state
> > reflects a defined operation of the device?
>
> I'd expect so, for instance the implementation of entering the _STOP
> state presumes a previous state that where the device is apparently
> already quiesced. That doesn't support a direct _RUNNING -> _STOP
> transition where I argued in the linked threads that those states
> should be reachable from any other state. Thanks,
If we focus on mlx5 there are two device 'flags' to manage:
- Device cannot issue DMAs
- Device internal state cannot change (ie cannot receive DMAs)
This is necessary to co-ordinate across multiple devices that might be
doing peer to peer DMA between them. The whole multi-device complex
should be moved to "cannot issue DMA's" then the whole complex would
go to "state cannot change" and be serialized.
The expected sequence at the device is thus
Resuming
full stop -> does not issue DMAs -> full operation
Suspend
full operation -> does not issue DMAs -> full stop
Further the device has two actions
- Trigger serializating the device state
- Trigger de-serializing the device state
So, what is the behavior upon each state:
* 000b => Device Stopped, not saving or resuming
Does not issue DMAs
Internal state cannot change
* 001b => Device running, which is the default state
Neither flags
* 010b => Stop the device & save the device state, stop-and-copy state
Does not issue DMAs
Internal state cannot change
* 011b => Device running and save the device state, pre-copy state
Neither flags
(future, DMA tracking turned on)
* 100b => Device stopped and the device state is resuming
Does not issue DMAs
Internal state cannot change
* 110b => Error state
???
* 101b => Invalid state
* 111b => Invalid state
???
What should the ??'s be? It looks like mlx5 doesn't use these, so it
should just refuse to enter these states in the first place..
The two actions:
trigger serializing the device state
Done when asked to go to 010b ?
trigger de-serializing the device state
Done when transition from 100b -> 000b ?
There is a missing state "Stop Active Transactions" which would be
only "does not issue DMAs". I've seen a proposal to add that.
I'm happy enough with this and it seems clean and easy enough to
implement.
Jason
