On Fri, Oct 21, 2022 at 05:47:59AM +0200, Richard Rogalski wrote: > Hello, very very sorry about the late reply. Life has been hectic. Also, not sure if this is how I reply to one of these, sorry if I screwed it up :) > > > This is great, thanks a lot for your report! Is this a regression? > > Believe it or not, I am a brand new SPARC user :). So I can't say > right now. Should I try a few old kernel releases to check? I wouldn't bother trying older kernels. In fact, I just noticed that you're running a 5.15 kernel, which is about a year old. It would be much more interesting to try to reproduce the problem on a current kernel, e.g., v6.0. At https://packages.gentoo.org/packages/sys-kernel/gentoo-kernel, it doesn't look like sparc gets much attention ;) > > Any chance you could collect a dmesg log with "ofpci_debug=1"? > > https://gitlab.freedesktop.org/drm/amd/uploads/0ed3c92921d7f88b06654b5f46e9756d/dmesg > > > Do the devices we complain about (NICs and storage HBAs 09:00.0, > > 09:00.1, 0d:00.0, 0d:00.1, 0e:00.0, 0f:00.0, 0001:03:00.0, > > 0001:03:00.1, 0001:0:00.0, 0001:0a:00.1) work? > > Well, I don't have any fiber optic equipment: these just came with > the server. Also it has wayy too many NICs. I can't quite say. > However... for the HBAs, that's where my root is :O. This is mildly > concerning :D. I spent way too long looking at these PCI resource weirdnesses. Bottom line: ignore them. >From your ofpci_debug dmesg log (annotated with logging the PCI core would do if it were doing this instead of the sparc OF code): pci@400: PCI MEM [mem 0x84000100000-0x8407f7fffff] offset 84000000000 pci@400: PCI MEM64 [mem 0x84100000000-0x84dffffffff] offset 80000000000 pci_bus 0000:00: root bus resource [mem 0x84000100000-0x8407f7fffff] (bus address [0x00100000-0x7f7fffff]) pci_bus 0000:00: root bus resource [mem 0x84100000000-0x84dffffffff] (bus address [0x4100000000-0x4dffffffff]) pci 0000:04:00.0: can't claim VGA legacy [mem 0x000a0000-0x000bffff]: no compatible bridge window This one happens because according to OF, there is no bridge aperture to the PCI bus 0xa0000-0xbffff region. The only accessible PCI bus regions are [0x00100000-0x7f7fffff] and [0x4100000000-0x4dffffffff]. Probably an OF defect. pci 0000:02:0c.0: PCI bridge to [bus 09] pci 0000:02:0c.0: Using flags[0010220c] start[0000004120000000] size[0000000010000000] pci 0000:02:0c.0: bridge window [mem 0x84120000000-0x8412fffffff 64bit pref] pci 0000:09:00.0: can't claim BAR 0 [mem 0x84120000000-0x8412007ffff 64bit]: no compatible bridge window These and similar warnings happen because OF says the upstream bridge window is prefetchable, but this is a non-prefetchable BAR. These likely work fine because in most cases prefetching will not occur on PCIe, even though the bridge window allows it. So the warnings above are mostly harmless. If you were to hot-add something, there could be issues because we aren't keeping track of the space these devices use. lspci on sparc is unusual: it shows PCI bus addresses, not CPU physical addresses like other arches [1], which means we see things like this in dmesg, which shows the CPU physical address: pci_bus 0000:00: root bus resource [mem 0x84000100000-0x8407f7fffff] (bus address [0x00100000-0x7f7fffff]) pci 0000:04:00.0: reg 0x10: [mem 0x84000800000-0x84000ffffff] and this in lspci, which is the PCI bus address: 0000:04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 10) (prog-if 00 [VGA controller]) Region 0: Memory at 00800000 (32-bit, non-prefetchable) [size=8M] Annoying but harmless. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/?id=v5.18#n1
