On Thu, Jul 18, 2024 at 09:01:17PM +0800, Huacai Chen wrote: > On Thu, Jul 11, 2024 at 3:48 AM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: > > On Wed, Jul 10, 2024 at 11:04:24AM +0800, Huacai Chen wrote: > > > On Wed, Jul 10, 2024 at 5:24 AM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: > > > > > > > > On Wed, Jun 12, 2024 at 02:53:15PM +0800, Huacai Chen wrote: > > > > > LS7A chipset can be used as a downstream bridge which connected to a > > > > > high-level host bridge. In this case DEV_LS7A_PCIE_PORT5 is used as the > > > > > upward port. We should always enable MSI caps of this port, otherwise > > > > > downstream devices cannot use MSI. > > > > > > > > Can you clarify this topology a bit? Since DEV_LS7A_PCIE_PORT5 > > > > apparently has a class of PCI_CLASS_BRIDGE_HOST, I guess that in PCIe > > > > terms, it is basically a PCI host bridge (Root Complex, if you prefer) > > > > that is materialized as a PCI Endpoint? > > > > > > Now most of the existing LoongArch CPUs don't have an integrated PCIe > > > RC, instead they have HyperTransport controllers. But the latest CPU > > > (Loongson-3C6000) has an integrated PCIe RC and removed > > > HyperTransport. > > > > > > LS7A bridge can work together with both old (HT) CPUs and new (PCIe) > > > CPUs. If it is connected to an old CPU, its upstream port is a HT > > > port, and DEV_LS7A_PCIE_PORT5 works as a normal downstream PCIe port. > > > If it is connected to a new CPU, DEV_LS7A_PCIE_PORT5 works as an > > > upstream port (the class code becomes PCI_CLASS_BRIDGE_HOST) and the > > > HT port is idle. > > > > What does lspci look like for both the old HT and the new PCIe CPUs? > When LS7A connect to HT, > > 00:00.0 Host bridge: Loongson Technology LLC Hyper Transport Bridge Controller > 00:00.1 Host bridge: Loongson Technology LLC Hyper Transport Bridge > Controller (rev 01) > 00:00.2 Host bridge: Loongson Technology LLC Device 7a20 (rev 01) > 00:00.3 Host bridge: Loongson Technology LLC Device 7a30 > 00:03.0 Ethernet controller: Loongson Technology LLC Device 7a13 > 00:04.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02) > 00:04.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02) > 00:05.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02) > 00:05.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02) > 00:07.0 Audio device: Loongson Technology LLC HDA (High Definition > Audio) Controller > 00:08.0 SATA controller: Loongson Technology LLC Device 7a18 > 00:09.0 PCI bridge: Loongson Technology LLC Device 7a49 > 00:0d.0 PCI bridge: Loongson Technology LLC Device 7a49 > 00:0f.0 PCI bridge: Loongson Technology LLC Device 7a69 > 00:10.0 PCI bridge: Loongson Technology LLC Device 7a59 > 00:13.0 PCI bridge: Loongson Technology LLC Device 7a59 > 00:16.0 System peripheral: Loongson Technology LLC Device 7a1b > 00:19.0 USB controller: Loongson Technology LLC Device 7a34 > 02:00.0 Non-Volatile memory controller: Shenzhen Longsys Electronics > Co., Ltd. SM2263EN/SM2263XT-based OEM NVME SSD (DRAM-less) (rev 03) > 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. > [AMD/ATI] Oland [Radeon HD 8570 / R5 430 OEM / R7 240/340 / Radeon 520 > OEM] (rev 87) > 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] > Oland/Hainan/Cape Verde/Pitcairn HDMI Audio [Radeon HD 7000 Series] > > -[0000:00]-+-00.0 0014:7a00 > +-00.1 0014:7a10 > +-00.2 0014:7a20 > +-00.3 0014:7a30 > +-03.0 0014:7a13 > +-04.0 0014:7a24 > +-04.1 0014:7a14 > +-05.0 0014:7a24 > +-05.1 0014:7a14 > +-07.0 0014:7a07 > +-08.0 0014:7a18 > +-09.0-[01]-- > +-0d.0-[02]----00.0 1d97:2263 > +-0f.0-[03]-- > +-10.0-[04]--+-00.0 1002:6611 > | \-00.1 1002:aab0 > +-13.0-[05]-- > +-16.0 0014:7a1b > \-19.0 0014:7a34 > > DEV_LS7A_PCIE_PORT5 is 00:13.0 In this case, the DEV_LS7A_PCIE_PORT5 at 00:13.0 is a Header Type 1 (PCI-to-PCI bridge). I assume it has a PCIe Capability that identifies it as Root Port, and the secondary bus 05 is probably a PCIe Link leading to a slot. I found "lspci -v" output similar to this at https://linux-hardware.org/?probe=ad154077da&log=lspci_all > When LS7A connect to PCIe, > > 00:00.0 Host bridge: Loongson Technology LLC Device 7a59 > 00:03.0 Ethernet controller: Loongson Technology LLC Device 7a13 > 00:04.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02) > 00:04.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02) > 00:05.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02) > 00:05.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02) > 00:06.0 Multimedia video controller: Loongson Technology LLC Device > 7a25 (rev 01) > 00:06.1 VGA compatible controller: Loongson Technology LLC Device 7a36 (rev 02) > 00:06.2 Audio device: Loongson Technology LLC Device 7a37 > 00:07.0 Audio device: Loongson Technology LLC HDA (High Definition > Audio) Controller > 00:08.0 SATA controller: Loongson Technology LLC Device 7a18 > 00:09.0 PCI bridge: Loongson Technology LLC Device 7a49 > 00:0a.0 PCI bridge: Loongson Technology LLC Device 7a39 > 00:0b.0 PCI bridge: Loongson Technology LLC Device 7a39 > 00:0c.0 PCI bridge: Loongson Technology LLC Device 7a39 > 00:0d.0 PCI bridge: Loongson Technology LLC Device 7a49 > 00:0f.0 PCI bridge: Loongson Technology LLC Device 7a69 > 00:10.0 PCI bridge: Loongson Technology LLC Device 7a59 > 00:16.0 System peripheral: Loongson Technology LLC Device 7a1b > 00:17.0 ISA bridge: Loongson Technology LLC LPC Controller (rev 01) > 00:19.0 USB controller: Loongson Technology LLC Device 7a34 > 00:1c.0 PCI bridge: Loongson Technology LLC Device 3c09 > 00:1d.0 IOMMU: Loongson Technology LLC Device 3c0f > 00:1e.0 PCI bridge: Loongson Technology LLC Device 3c09 > 02:00.0 Ethernet controller: Device 1f0a:6801 (rev 01) > 08:00.0 PCI bridge: Loongson Technology LLC Device 3c19 > 08:01.0 PCI bridge: Loongson Technology LLC Device 3c29 > 08:02.0 PCI bridge: Loongson Technology LLC Device 3c29 > 0c:00.0 PCI bridge: Loongson Technology LLC Device 3c19 > 0c:01.0 PCI bridge: Loongson Technology LLC Device 3c19 > 0c:04.0 IOMMU: Loongson Technology LLC Device 3c0f > > -[0000:00]-+-00.0 0014:7a59 > +-03.0 0014:7a13 > +-04.0 0014:7a24 > +-04.1 0014:7a14 > +-05.0 0014:7a24 > +-05.1 0014:7a14 > +-06.0 0014:7a25 > +-06.1 0014:7a36 > +-06.2 0014:7a37 > +-07.0 0014:7a07 > +-08.0 0014:7a18 > +-09.0-[01]-- > +-0a.0-[02]----00.0 1f0a:6801 > +-0b.0-[03]-- > +-0c.0-[04]-- > +-0d.0-[05]-- > +-0f.0-[06]-- > +-10.0-[07]-- > +-16.0 0014:7a1b > +-17.0 0014:7a0c > +-19.0 0014:7a34 > +-1c.0-[08-0b]--+-00.0-[09]-- > | +-01.0-[0a]-- > | \-02.0-[0b]-- > +-1d.0 0014:3c0f > \-1e.0-[0c-0e]--+-00.0-[0d]-- > +-01.0-[0e]-- > \-04.0 0014:3c0f > > DEV_LS7A_PCIE_PORT5 becomes 00:00.0 In this case, the DEV_LS7A_PCIE_PORT5 at 00:00.0 looks like a Header Type 0 device (not a bridge), and the bus 00 it is on is not a normal PCIe Link. Since it doesn't connect to a PCIe Link, 00:00.0 might not have a PCIe Capability at all, or it could be an RCiEP. Or it's possible it has a PCIe Capability left over from the old model that says it's a Root Port, although this would potentially confuse an OS. The other bridges (00:09.0, 00:0a.0, etc) are probably PCIe Root Ports, which are logically part of the Root Complex, so the Root Complex would be implemented partly in the new CPU and partly in the LS7A. The connection between the new CPU and the LS7A might be electrically similar to PCIe, but it's not part of a PCIe hierarchy that Linux sees. That connection would have to be managed in some non-PCI way, the same as all PCI host bridges, i.e., by firmware, ACPI, or a Linux native host bridge driver like pci-loongson.c. Here's the current commit log: The LS7A chipset can be used as a downstream bridge that connects to a high-level host bridge, and in such case the DEV_LS7A_PCIE_PORT5 is used as the upstream port. Thus, always enable MSI caps of this port, otherwise downstream devices cannot use MSI. In this configuration, DEV_LS7A_PCIE_PORT5 is not a PCI-to-PCI bridge itself. It might be some sort of downstream end of a connection from the CPU, but that's not really relevant here. And DEV_LS7A_PCIE_PORT5 doesn't have any devices directly downstream from it. I think something like this would be clearer: The LS7A chipset can be used as part of a PCIe Root Complex with Loongson-3C6000 and similar CPUs. In this case, DEV_LS7A_PCIE_PORT5 has a PCI_CLASS_BRIDGE_HOST class code, and it is a Type 0 Function whose config space provides access to Root Complex registers. The DEV_LS7A_PCIE_PORT5 has an MSI Capability, and its MSI Enable bit must be set before other devices below the Root Complex can use MSI. This is not the standard PCI behavior of MSI Enable, so the normal PCI MSI code does not set it. Set the DEV_LS7A_PCIE_PORT5 MSI Enable bit via a quirk so other devices below the Root Complex can use MSI. What do you think?
