On Fri, Jul 19, 2024 at 7:03 AM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: > > On Thu, Jul 18, 2024 at 09:01:17PM +0800, Huacai Chen wrote: > > On Thu, Jul 11, 2024 at 3:48 AM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: > > > On Wed, Jul 10, 2024 at 11:04:24AM +0800, Huacai Chen wrote: > > > > On Wed, Jul 10, 2024 at 5:24 AM Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote: > > > > > > > > > > On Wed, Jun 12, 2024 at 02:53:15PM +0800, Huacai Chen wrote: > > > > > > LS7A chipset can be used as a downstream bridge which connected to a > > > > > > high-level host bridge. In this case DEV_LS7A_PCIE_PORT5 is used as the > > > > > > upward port. We should always enable MSI caps of this port, otherwise > > > > > > downstream devices cannot use MSI. > > > > > > > > > > Can you clarify this topology a bit? Since DEV_LS7A_PCIE_PORT5 > > > > > apparently has a class of PCI_CLASS_BRIDGE_HOST, I guess that in PCIe > > > > > terms, it is basically a PCI host bridge (Root Complex, if you prefer) > > > > > that is materialized as a PCI Endpoint? > > > > > > > > Now most of the existing LoongArch CPUs don't have an integrated PCIe > > > > RC, instead they have HyperTransport controllers. But the latest CPU > > > > (Loongson-3C6000) has an integrated PCIe RC and removed > > > > HyperTransport. > > > > > > > > LS7A bridge can work together with both old (HT) CPUs and new (PCIe) > > > > CPUs. If it is connected to an old CPU, its upstream port is a HT > > > > port, and DEV_LS7A_PCIE_PORT5 works as a normal downstream PCIe port. > > > > If it is connected to a new CPU, DEV_LS7A_PCIE_PORT5 works as an > > > > upstream port (the class code becomes PCI_CLASS_BRIDGE_HOST) and the > > > > HT port is idle. > > > > > > What does lspci look like for both the old HT and the new PCIe CPUs? > > When LS7A connect to HT, > > > > 00:00.0 Host bridge: Loongson Technology LLC Hyper Transport Bridge Controller > > 00:00.1 Host bridge: Loongson Technology LLC Hyper Transport Bridge > > Controller (rev 01) > > 00:00.2 Host bridge: Loongson Technology LLC Device 7a20 (rev 01) > > 00:00.3 Host bridge: Loongson Technology LLC Device 7a30 > > 00:03.0 Ethernet controller: Loongson Technology LLC Device 7a13 > > 00:04.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02) > > 00:04.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02) > > 00:05.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02) > > 00:05.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02) > > 00:07.0 Audio device: Loongson Technology LLC HDA (High Definition > > Audio) Controller > > 00:08.0 SATA controller: Loongson Technology LLC Device 7a18 > > 00:09.0 PCI bridge: Loongson Technology LLC Device 7a49 > > 00:0d.0 PCI bridge: Loongson Technology LLC Device 7a49 > > 00:0f.0 PCI bridge: Loongson Technology LLC Device 7a69 > > 00:10.0 PCI bridge: Loongson Technology LLC Device 7a59 > > 00:13.0 PCI bridge: Loongson Technology LLC Device 7a59 > > 00:16.0 System peripheral: Loongson Technology LLC Device 7a1b > > 00:19.0 USB controller: Loongson Technology LLC Device 7a34 > > 02:00.0 Non-Volatile memory controller: Shenzhen Longsys Electronics > > Co., Ltd. SM2263EN/SM2263XT-based OEM NVME SSD (DRAM-less) (rev 03) > > 04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. > > [AMD/ATI] Oland [Radeon HD 8570 / R5 430 OEM / R7 240/340 / Radeon 520 > > OEM] (rev 87) > > 04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] > > Oland/Hainan/Cape Verde/Pitcairn HDMI Audio [Radeon HD 7000 Series] > > > > -[0000:00]-+-00.0 0014:7a00 > > +-00.1 0014:7a10 > > +-00.2 0014:7a20 > > +-00.3 0014:7a30 > > +-03.0 0014:7a13 > > +-04.0 0014:7a24 > > +-04.1 0014:7a14 > > +-05.0 0014:7a24 > > +-05.1 0014:7a14 > > +-07.0 0014:7a07 > > +-08.0 0014:7a18 > > +-09.0-[01]-- > > +-0d.0-[02]----00.0 1d97:2263 > > +-0f.0-[03]-- > > +-10.0-[04]--+-00.0 1002:6611 > > | \-00.1 1002:aab0 > > +-13.0-[05]-- > > +-16.0 0014:7a1b > > \-19.0 0014:7a34 > > > > DEV_LS7A_PCIE_PORT5 is 00:13.0 > > In this case, the DEV_LS7A_PCIE_PORT5 at 00:13.0 is a Header Type 1 > (PCI-to-PCI bridge). I assume it has a PCIe Capability that > identifies it as Root Port, and the secondary bus 05 is probably a > PCIe Link leading to a slot. > > I found "lspci -v" output similar to this at > https://linux-hardware.org/?probe=ad154077da&log=lspci_all > > > When LS7A connect to PCIe, > > > > 00:00.0 Host bridge: Loongson Technology LLC Device 7a59 > > 00:03.0 Ethernet controller: Loongson Technology LLC Device 7a13 > > 00:04.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02) > > 00:04.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02) > > 00:05.0 USB controller: Loongson Technology LLC OHCI USB Controller (rev 02) > > 00:05.1 USB controller: Loongson Technology LLC EHCI USB Controller (rev 02) > > 00:06.0 Multimedia video controller: Loongson Technology LLC Device > > 7a25 (rev 01) > > 00:06.1 VGA compatible controller: Loongson Technology LLC Device 7a36 (rev 02) > > 00:06.2 Audio device: Loongson Technology LLC Device 7a37 > > 00:07.0 Audio device: Loongson Technology LLC HDA (High Definition > > Audio) Controller > > 00:08.0 SATA controller: Loongson Technology LLC Device 7a18 > > 00:09.0 PCI bridge: Loongson Technology LLC Device 7a49 > > 00:0a.0 PCI bridge: Loongson Technology LLC Device 7a39 > > 00:0b.0 PCI bridge: Loongson Technology LLC Device 7a39 > > 00:0c.0 PCI bridge: Loongson Technology LLC Device 7a39 > > 00:0d.0 PCI bridge: Loongson Technology LLC Device 7a49 > > 00:0f.0 PCI bridge: Loongson Technology LLC Device 7a69 > > 00:10.0 PCI bridge: Loongson Technology LLC Device 7a59 > > 00:16.0 System peripheral: Loongson Technology LLC Device 7a1b > > 00:17.0 ISA bridge: Loongson Technology LLC LPC Controller (rev 01) > > 00:19.0 USB controller: Loongson Technology LLC Device 7a34 > > 00:1c.0 PCI bridge: Loongson Technology LLC Device 3c09 > > 00:1d.0 IOMMU: Loongson Technology LLC Device 3c0f > > 00:1e.0 PCI bridge: Loongson Technology LLC Device 3c09 > > 02:00.0 Ethernet controller: Device 1f0a:6801 (rev 01) > > 08:00.0 PCI bridge: Loongson Technology LLC Device 3c19 > > 08:01.0 PCI bridge: Loongson Technology LLC Device 3c29 > > 08:02.0 PCI bridge: Loongson Technology LLC Device 3c29 > > 0c:00.0 PCI bridge: Loongson Technology LLC Device 3c19 > > 0c:01.0 PCI bridge: Loongson Technology LLC Device 3c19 > > 0c:04.0 IOMMU: Loongson Technology LLC Device 3c0f > > > > -[0000:00]-+-00.0 0014:7a59 > > +-03.0 0014:7a13 > > +-04.0 0014:7a24 > > +-04.1 0014:7a14 > > +-05.0 0014:7a24 > > +-05.1 0014:7a14 > > +-06.0 0014:7a25 > > +-06.1 0014:7a36 > > +-06.2 0014:7a37 > > +-07.0 0014:7a07 > > +-08.0 0014:7a18 > > +-09.0-[01]-- > > +-0a.0-[02]----00.0 1f0a:6801 > > +-0b.0-[03]-- > > +-0c.0-[04]-- > > +-0d.0-[05]-- > > +-0f.0-[06]-- > > +-10.0-[07]-- > > +-16.0 0014:7a1b > > +-17.0 0014:7a0c > > +-19.0 0014:7a34 > > +-1c.0-[08-0b]--+-00.0-[09]-- > > | +-01.0-[0a]-- > > | \-02.0-[0b]-- > > +-1d.0 0014:3c0f > > \-1e.0-[0c-0e]--+-00.0-[0d]-- > > +-01.0-[0e]-- > > \-04.0 0014:3c0f > > > > DEV_LS7A_PCIE_PORT5 becomes 00:00.0 > > In this case, the DEV_LS7A_PCIE_PORT5 at 00:00.0 looks like a Header > Type 0 device (not a bridge), and the bus 00 it is on is not a normal > PCIe Link. Since it doesn't connect to a PCIe Link, 00:00.0 might not > have a PCIe Capability at all, or it could be an RCiEP. Or it's > possible it has a PCIe Capability left over from the old model that > says it's a Root Port, although this would potentially confuse an OS. > > The other bridges (00:09.0, 00:0a.0, etc) are probably PCIe Root > Ports, which are logically part of the Root Complex, so the Root > Complex would be implemented partly in the new CPU and partly in the > LS7A. > > The connection between the new CPU and the LS7A might be electrically > similar to PCIe, but it's not part of a PCIe hierarchy that Linux > sees. That connection would have to be managed in some non-PCI way, > the same as all PCI host bridges, i.e., by firmware, ACPI, or a Linux > native host bridge driver like pci-loongson.c. > > Here's the current commit log: > > The LS7A chipset can be used as a downstream bridge that connects to > a high-level host bridge, and in such case the DEV_LS7A_PCIE_PORT5 > is used as the upstream port. > > Thus, always enable MSI caps of this port, otherwise downstream > devices cannot use MSI. > > In this configuration, DEV_LS7A_PCIE_PORT5 is not a PCI-to-PCI bridge > itself. It might be some sort of downstream end of a connection from > the CPU, but that's not really relevant here. And DEV_LS7A_PCIE_PORT5 > doesn't have any devices directly downstream from it. > > I think something like this would be clearer: > > The LS7A chipset can be used as part of a PCIe Root Complex with > Loongson-3C6000 and similar CPUs. In this case, DEV_LS7A_PCIE_PORT5 > has a PCI_CLASS_BRIDGE_HOST class code, and it is a Type 0 Function > whose config space provides access to Root Complex registers. > > The DEV_LS7A_PCIE_PORT5 has an MSI Capability, and its MSI Enable > bit must be set before other devices below the Root Complex can use > MSI. This is not the standard PCI behavior of MSI Enable, so the > normal PCI MSI code does not set it. > > Set the DEV_LS7A_PCIE_PORT5 MSI Enable bit via a quirk so other > devices below the Root Complex can use MSI. > > What do you think? Yes, that's more better. Huacai
