On Tue, 2023-10-31 at 12:59 -0700, Nirmal Patel wrote:
> On Tue, 2023-10-31 at 10:31 -0500, Bjorn Helgaas wrote:
> > On Mon, Oct 30, 2023 at 04:16:54PM -0400, Nirmal Patel wrote:
> > > VMD Hotplug should be enabled or disabled based on VMD rootports'
> > > Hotplug configuration in BIOS. is_hotplug_bridge is set on each
> > > VMD rootport based on Hotplug capable bit in SltCap in probe.c.
> > > Check is_hotplug_bridge and enable or disable native_pcie_hotplug
> > > based on that value.
> > >
> > > Currently VMD driver copies ACPI settings or platform
> > > configurations
> > > for Hotplug, AER, DPC, PM, etc and enables or disables these
> > > features
> > > on VMD bridge which is not correct in case of Hotplug.
> >
> > This needs some background about why it's correct to copy the ACPI
> > settings in the case of AER, DPC, PM, etc, but incorrect for
> > hotplug.
> >
> > > Also during the Guest boot up, ACPI settings along with VMD UEFI
> > > driver are not present in Guest BIOS which results in assigning
> > > default values to Hotplug, AER, DPC, etc. As a result Hotplug is
> > > disabled on VMD in the Guest OS.
> > >
> > > This patch will make sure that Hotplug is enabled properly in
> > > Host
> > > as well as in VM.
> >
> > Did we come to some consensus about how or whether _OSC for the
> > host
> > bridge above the VMD device should apply to devices in the separate
> > domain below the VMD?
> We are not able to come to any consensus. Someone suggested to copy
> either all _OSC flags or none. But logic behind that assumption is
> that the VMD is a bridge device which is not completely true. VMD is
> an
> endpoint device and it owns its domain.
>
> Also please keep this in your consideration, since Guest BIOS
> doesn't have _OSC implementation, all of the flags Hotplug, AER, DPC
> are set to power state default value and VMD's very important hotplug
> functionality is broken.
In case of Host OS, when VMD copies all the _OSC flags, Hotplug, AER,
DPC, etc, it reflects Host BIOS settings.
But in case of Guest OS or VM, the _OSC flags do not reflect Host BIOS
settings. Instead what we have is power ON default values in VM, thus
does not reflect any Host BIOS settings. For example, disabling Hotplug
in VM eventhough it is enabled in Host BIOS.
The patch 04b12ef163d1 broke the settings in Guest kernel by applying
non-host default _OSC values. This long discussion is about restoring
some of these settings to correct Host BIOS settings. i.e. Hotplug.
>
> The patch 04b12ef163d1 assumes VMD is a bridge device and borrows/
> *imposes system settings* for AER, DPC, Hotplug, PM, etc on VMD.
> VMD is*type 0 PCI endpoint* device and all the PCI devices under VMD
> are *privately *owned by VMD not by the OS.
>
> Also VMD has its own Hotplug setting for its rootports in BIOS under
> VMD settings that are different from global BIOS system settings. It
> is
> these settings that give VMD its own unique functionality.
>
> That is why I suggested three solutions but never got any
> confirmation.
>
> #1: Revert the patch 04b12ef163d1 which was added under wrong
> assumption. This patch didn't need to be added to VMD code if AER
> was disabled from BIOS platform settings.
>
> #2: VMD driver disables AER by copying AER BIOS system settings
> which the patch 04b12ef163d1 does but do not change Hotplug.
> I proposed this patch and didn't get approval.
>
> #3: If hotplug is enabled on VMD root ports, make sure hotplug is
> enabled on the bridge rootports are connected to. The proposed patch
> does that.
>
> Can we please come to some decision? VM Hotplug is an important part
> of
> VMD and customers are reporting VM Hotplug issues. I would like to
> get
> some progress on this one.
>
> Thanks
> nirmal
> > I think this warrants some clarification and possibly discussion in
> > the PCI firmware SIG.
> >
> > At the very least, the commit log should mention _OSC and say
> > something about the fact that this is assuming PCIe hotplug
> > ownership
> > for devices below VMD, regardless of what the upstream _OSC said.
> >
> > > Signed-off-by: Nirmal Patel <nirmal.patel@xxxxxxxxxxxxxxx>
> > > ---
> > > ---
> > > drivers/pci/controller/vmd.c | 11 ++++++++++-
> > > 1 file changed, 10 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/pci/controller/vmd.c
> > > b/drivers/pci/controller/vmd.c
> > > index 769eedeb8802..e39eaef5549a 100644
> > > --- a/drivers/pci/controller/vmd.c
> > > +++ b/drivers/pci/controller/vmd.c
> > > @@ -720,6 +720,7 @@ static int vmd_enable_domain(struct vmd_dev
> > > *vmd, unsigned long features)
> > > resource_size_t membar2_offset = 0x2000;
> > > struct pci_bus *child;
> > > struct pci_dev *dev;
> > > + struct pci_host_bridge *vmd_bridge;
> > > int ret;
> > >
> > > /*
> > > @@ -886,8 +887,16 @@ static int vmd_enable_domain(struct vmd_dev
> > > *vmd, unsigned long features)
> > > * and will fail pcie_bus_configure_settings() early. It can
> > > instead be
> > > * run on each of the real root ports.
> > > */
> > > - list_for_each_entry(child, &vmd->bus->children, node)
> > > + vmd_bridge = to_pci_host_bridge(vmd->bus->bridge);
> > > + list_for_each_entry(child, &vmd->bus->children, node) {
> > > pcie_bus_configure_settings(child);
> > > + /*
> > > + * When Hotplug is enabled on vmd root-port, enable it
> > > on vmd
> > > + * bridge.
> > > + */
> > > + if (child->self->is_hotplug_bridge)
> > > + vmd_bridge->native_pcie_hotplug = 1;
> > > + }
> > >
> > > pci_bus_add_devices(vmd->bus);
> > >
> > > --
> > > 2.31.1
> > >
