Re: [RFC] handling hostdev save/load net config for non SR-IOV devices

Laine Stump <laine@xxxxxxxxx> · Thu, 18 Jul 2019 11:29:06 -0400



    On 7/18/19 10:29 AM, Daniel Henrique
      Barboza wrote:

    
      Hi,

        
        I have a PoC that enables partial coldplug assignment of
        multifunction

        PCI devices with managed mode. At this moment, Libvirt can't
        handle

        this scenario - the code will detach only the hostdevs from the
        XML,

        when in fact the whole IOMMU needs to be detached. This can be

        verified by the fact that Libvirt handles the unmanaged scenario

        well, as long as the user detaches the whole IOMMU beforehand.

        
        I have played with 2 approaches. The one I am planning to
        contribute

        back is a change inside virHostdevGetPCIHostDeviceList(), that

        adds the extra PCI devices for detach/re-attach in case a PCI

        Multifunction device in managed mode is presented in the XML.

      
    If you're thinking of doing that automatically,
        then I should warn you that we had discussed that a long time
        ago, and decided that it was a bad idea to do it because it was
        likely someone would, e.g. try to assign an audio device to
        their guest that happened to be one function on a multifunction
        device that also contained a disk controller (or some other
        device) that the host needed for proper operation.
    

    It may be that in *your* particular case, you
        understand that the functions you don't want to assign to the
        guest are not otherwise used, and it's not dangerous to suddenly
        detach them from their host driver. But you can't assume that
        will always be the case.
    

    If you *really* can't accept just assigning all
        the devices in that IOMMU group to the guest (thus making them
        all explicitly listed in the config, and obvious to the
        administrator that they won't be available on the host) and
        simply not using them, then you either need to separately detach
        those particular functions from the host, or come up with a way
        of having the domain config explicitly list them as "detached
        from the host but not actually attached to the guest".
    

     Now, there's a catch. Inside both  virHostdevPreparePCIDevices()

          and virHostdevReAttachPCIDevices() there are code to
          save/restore

          the network configuration for SR-IOV devices. These
        functions iterates

        in the hostdevs list, instead of the pcidevs list I'm changing.
        The final

        result, given that the current conditions used for SR-IOV
        matches the

        conditions for multifunction PCI devices as well, is that not
        all virtual

        functions will get their network configuration saved/restored.

      
    If you're not going to use a device (which is
        implied by the fact that it's not in the hostdevs list) then
        nothing about its network config will change, so there is no
        reason to save/restore it.
    

        For example, a guest that uses 3 of 4 functions of a PCI
        MultiFunction

        card, let's say functions 0,1 and 3. The code will handle the
        detach

        of all the IOMMU, including the function 2 that isn't declared
        in the

        XML.
    

    Again, the above sentence implies that you're wanting to make
      this completely automatic, which we previously decided was
      something we didn't want to do. 

    
     However, since function 2 isn't a hostdev, its
        network config

        will not be restored after the VM shutdown.

      
    You're talking about something that will never
        occur - on every SRIOV network card I've ever seen each VF is in
        its own IOMMU group, and can be assigned to a guest independent
        of what's done with any other VF. I've never seen a case (except
        maybe once with a newly released motherboard that had broken
        IOMMU firmware(?)) where a VF was in the same IOMMU group as any
        other device.

      
        Now comes the question: how much effort should be spent into
        making

        the network config of all the functions be restored? Is this a
        blocker

        for the whole code to be accepted or, given it is proper
        documented

        somewhere, it can be done later on?
    

--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list