Jason Gunthorpe <jgg@xxxxxxxxxx> writes: > Provide some more complete documentation for the migration region's > behavior, specifically focusing on the device_state bits and the whole > system view from a VMM. > > Signed-off-by: Jason Gunthorpe <jgg@xxxxxxxxxx> > --- > Documentation/driver-api/vfio.rst | 208 +++++++++++++++++++++++++++++- > 1 file changed, 207 insertions(+), 1 deletion(-) > > Alex/Cornelia, here is the first draft of the requested documentation I promised > > We think it includes all the feedback from hns, Intel and NVIDIA on this mechanism. > > Our thinking is that NDMA would be implemented like this: > > +#define VFIO_DEVICE_STATE_NDMA (1 << 3) > > And a .add_capability ops will be used to signal to userspace driver support: > > +#define VFIO_REGION_INFO_CAP_MIGRATION_NDMA 6 > > I've described DIRTY TRACKING as a seperate concept here. With the current > uAPI this would be controlled by VFIO_IOMMU_DIRTY_PAGES_FLAG_START, with our > change in direction this would be per-tracker control, but no semantic change. > > Upon some agreement we'll include this patch in the next iteration of the mlx5 driver > along with the NDMA bits. > > Thanks, > Jason > > diff --git a/Documentation/driver-api/vfio.rst b/Documentation/driver-api/vfio.rst > index c663b6f978255b..b28c6fb89ee92f 100644 > --- a/Documentation/driver-api/vfio.rst > +++ b/Documentation/driver-api/vfio.rst > @@ -242,7 +242,213 @@ group and can access them as follows:: > VFIO User API > ------------------------------------------------------------------------------- > > -Please see include/linux/vfio.h for complete API documentation. > +Please see include/uapi/linux/vfio.h for complete API documentation. > + > +------------------------------------------------------------------------------- > + > +VFIO migration driver API > +------------------------------------------------------------------------------- > + > +VFIO drivers that support migration implement a migration control register > +called device_state in the struct vfio_device_migration_info which is in its > +VFIO_REGION_TYPE_MIGRATION region. > + > +The device_state triggers device action both when bits are set/cleared and > +continuous behavior for each bit. For VMMs they can also control if the VCPUs in > +a VM are executing (VCPU RUNNING) and if the IOMMU is logging DMAs (DIRTY > +TRACKING). These two controls are not part of the device_state register, KVM > +will be used to control the VCPU and VFIO_IOMMU_DIRTY_PAGES_FLAG_START on the > +container controls dirty tracking. > + > +Along with the device_state the migration driver provides a data window which > +allows streaming migration data into or out of the device. > + > +A lot of flexibility is provided to userspace in how it operates these bits. The > +reference flow for saving device state in a live migration, with all features: > + > + RUNNING, VCPU_RUNNING > + Normal operating state > + RUNNING, DIRTY TRACKING, VCPU RUNNING > + Log DMAs > + Stream all memory So I'd recommend actually building the docs and looking at the result; this will not render the way you expect it to. I'd suggest using a literal block for preformatted sections like this. Thanks, jon
