On 12/7/2022 9:09 AM, Jason Gunthorpe wrote:
Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding. On Tue, Dec 06, 2022 at 05:07:01PM -0800, Brett Creeley wrote:+struct file * +pds_vfio_step_device_state_locked(struct pds_vfio_pci_device *pds_vfio, + enum vfio_device_mig_state next) +{ + enum vfio_device_mig_state cur = pds_vfio->state; + struct device *dev = &pds_vfio->pdev->dev; + unsigned long lm_action_start; + int err = 0; + + dev_dbg(dev, "%s => %s\n", + pds_vfio_lm_state(cur), pds_vfio_lm_state(next)); + + lm_action_start = jiffies; + if (cur == VFIO_DEVICE_STATE_STOP && next == VFIO_DEVICE_STATE_STOP_COPY) { + /* Device is already stopped + * create save device data file & get device state from firmware + */ + err = pds_vfio_get_save_file(pds_vfio); + if (err) + return ERR_PTR(err); + + /* Get device state */ + err = pds_vfio_get_lm_state_cmd(pds_vfio); + if (err) { + pds_vfio_put_save_file(pds_vfio); + return ERR_PTR(err); + } + + return pds_vfio->save_file->filep; + } + + if (cur == VFIO_DEVICE_STATE_STOP_COPY && next == VFIO_DEVICE_STATE_STOP) { + /* Device is already stopped + * delete the save device state file + */ + pds_vfio_put_save_file(pds_vfio); + pds_vfio_send_host_vf_lm_status_cmd(pds_vfio, + PDS_LM_STA_NONE); + return NULL; + } + + if (cur == VFIO_DEVICE_STATE_STOP && next == VFIO_DEVICE_STATE_RESUMING) { + /* create resume device data file */ + err = pds_vfio_get_restore_file(pds_vfio); + if (err) + return ERR_PTR(err); + + return pds_vfio->restore_file->filep; + } + + if (cur == VFIO_DEVICE_STATE_RESUMING && next == VFIO_DEVICE_STATE_STOP) { + /* Set device state */ + err = pds_vfio_set_lm_state_cmd(pds_vfio); + if (err) + return ERR_PTR(err); + + /* delete resume device data file */ + pds_vfio_put_restore_file(pds_vfio); + return NULL; + } + + if (cur == VFIO_DEVICE_STATE_RUNNING && next == VFIO_DEVICE_STATE_STOP) { + /* Device should be stopped + * no interrupts, dma or change in internal state + */ + err = pds_vfio_suspend_device_cmd(pds_vfio); + if (err) + return ERR_PTR(err); + + return NULL; + } + + if (cur == VFIO_DEVICE_STATE_STOP && next == VFIO_DEVICE_STATE_RUNNING) { + /* Device should be functional + * interrupts, dma, mmio or changes to internal state is allowed + */ + err = pds_vfio_resume_device_cmd(pds_vfio); + if (err) + return ERR_PTR(err); + + pds_vfio_send_host_vf_lm_status_cmd(pds_vfio, + PDS_LM_STA_NONE); + return NULL; + }Please implement the P2P states in your device. After long discussions we really want to see all VFIO migrations implementations support this. It is still not clear what qemu will do when it sees devices that do not support P2P, but it will not be nice.
Does that mean VFIO_MIGRATION_P2P is going to be required going forward or do we just need to handle the P2P transitions? Can you point me to where this is being discussed?
Also, since you are obviously using and testing the related qemu series, please participate in the review of that in the qemu list, or at least offer your support with testing.
ACK.
While HCH is objecting to this driver even existing I won't comment on specific details.. Though it is intesting this approach doesn't change NVMe at all so it does seem less objectionable to me than the Intel RFC.
That's understandable and thanks for the initial feedback. Yes, no NVMe changes required.
Jason -- You received this message because you are subscribed to the Google Groups "Pensando Drivers" group. To unsubscribe from this group and stop receiving emails from it, send an email to drivers+unsubscribe@xxxxxxxxxxx. To view this discussion on the web visit https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fa%2Fpensando.io%2Fd%2Fmsgid%2Fdrivers%2FY5DIvM1Ca0qLNzPt%2540ziepe.ca&data=05%7C01%7Cbrett.creeley%40amd.com%7Cb5b743b18f054684cc3108dad875c84f%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C638060297638789271%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3xXVIMNmYF7qFjhOiW9DDhbrzZklx%2FZ9xmEirgwodfw%3D&reserved=0.
