[+to Christian, resizable BAR expert] On Wed, Apr 17, 2024 at 03:16:06PM +0200, Dag B wrote: > Hi. > > In short, I have a GPU for which lspci reveals: > > Capabilities: [bb0 v1] Physical Resizable BAR > > BAR 0: current size: 16MB, supported: 16MB > BAR 1: current size: 128MB, supported: 64MB 128MB 256MB 512MB 1GB > 2GB 4GB 8GB 16GB 32GB > BAR 3: current size: 32MB, supported: 32MB > > In dmesg I see: > > [ 0.517191] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit > pref]: assigned > [ 0.517238] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit > pref]: assigned > [ 0.517261] pci 0000:08:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]: assigned > > I take it the location of BAR 3 right after BAR 1 explains why I get: > > p53 # echo 9 > resource1_resize > -bash: echo: write error: No space left on device > > Shrinking it and increasing it to the orginal size works. > > > Is there anything I can do with current kernel functionality to reserve > memory address space for the full-fat BAR 1? Or relocate it? > > If not, is this something which *can* be worked around in the kernel? And if > so, does it belong with the PCI subsystem? Or the devicedriver for the > device in question? > > Is there a good ELI13 resource explaining how resizable BAR works in Linux? > > My current kernel command-line contains: pci=assign-busses,realloc > > My GPU is attached via TB3 to a system for which resizable BAR is and will > remain a foreign concept in the BIOS. > > dmesg excerpt below reflects resizing the BAR to 128MB and then back to > 256MB. > > [ 1730.091789] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit > pref]: releasing > [ 1730.091875] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit > pref]: releasing > [ 1730.092072] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x6017ffffff 64bit pref]: releasing > [ 1730.092151] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x6017ffffff 64bit pref]: assigned > [ 1730.092223] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit > pref]: assigned > [ 1730.092335] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit > pref]: assigned > [ 1730.092444] pcieport 0000:06:00.0: PCI bridge to [bus 07-09] > [ 1730.092510] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff] > [ 1730.092595] pcieport 0000:06:00.0: bridge window [mem > 0xa4000000-0xafffffff] > [ 1730.092680] pcieport 0000:06:00.0: bridge window [mem > 0x6000000000-0x601fffffff 64bit pref] > [ 1730.092778] pcieport 0000:07:01.0: PCI bridge to [bus 08] > [ 1730.092850] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff] > [ 1730.092933] pcieport 0000:07:01.0: bridge window [mem > 0xa4000000-0xa57fffff] > [ 1730.093018] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x6017ffffff 64bit pref] > [ 1759.817306] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit > pref]: releasing > [ 1759.817394] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit > pref]: releasing > [ 1759.817591] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x6017ffffff 64bit pref]: releasing > [ 1759.817668] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x600bffffff 64bit pref]: assigned > [ 1759.817740] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit > pref]: assigned > [ 1759.817853] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit > pref]: assigned > [ 1759.817964] pcieport 0000:06:00.0: PCI bridge to [bus 07-09] > [ 1759.818035] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff] > [ 1759.818120] pcieport 0000:06:00.0: bridge window [mem > 0xa4000000-0xafffffff] > [ 1759.818204] pcieport 0000:06:00.0: bridge window [mem > 0x6000000000-0x601fffffff 64bit pref] > [ 1759.818303] pcieport 0000:07:01.0: PCI bridge to [bus 08] > [ 1759.818374] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff] > [ 1759.818459] pcieport 0000:07:01.0: bridge window [mem > 0xa4000000-0xa57fffff] > [ 1759.818544] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x600bffffff 64bit pref] > [ 1769.797178] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit > pref]: releasing > [ 1769.797241] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit > pref]: releasing > [ 1769.797417] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x600bffffff 64bit pref]: releasing > [ 1769.797473] pcieport 0000:07:01.0: bridge window [mem size 0x30000000 > 64bit pref]: can't assign; no space > [ 1769.797515] pcieport 0000:07:01.0: bridge window [mem size 0x30000000 > 64bit pref]: failed to assign > [ 1769.797557] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]: > can't assign; no space > [ 1769.797594] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]: > failed to assign > [ 1769.797630] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]: > can't assign; no space > [ 1769.797666] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]: > failed to assign > [ 1769.797703] pcieport 0000:06:00.0: PCI bridge to [bus 07-09] > [ 1769.797761] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff] > [ 1769.797829] pcieport 0000:06:00.0: bridge window [mem > 0xa4000000-0xafffffff] > [ 1769.797895] pcieport 0000:06:00.0: bridge window [mem > 0x6000000000-0x601fffffff 64bit pref] > [ 1769.797972] pcieport 0000:07:01.0: PCI bridge to [bus 08] > [ 1769.798027] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff] > [ 1769.798089] pcieport 0000:07:01.0: bridge window [mem > 0xa4000000-0xa57fffff] > [ 1769.798155] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x600bffffff 64bit pref] > [ 1769.798270] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit > pref]: assigned > [ 1769.798358] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit > pref]: assigned > [ 2669.324929] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit > pref]: releasing > [ 2669.324992] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit > pref]: releasing > [ 2669.325164] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x600bffffff 64bit pref]: releasing > [ 2669.325219] pcieport 0000:07:01.0: bridge window [mem size 0x30000000 > 64bit pref]: can't assign; no space > [ 2669.327023] pcieport 0000:07:01.0: bridge window [mem size 0x30000000 > 64bit pref]: failed to assign > [ 2669.328798] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]: > can't assign; no space > [ 2669.330482] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]: > failed to assign > [ 2669.331104] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]: > can't assign; no space > [ 2669.331682] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]: > failed to assign > [ 2669.332258] pcieport 0000:06:00.0: PCI bridge to [bus 07-09] > [ 2669.332855] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff] > [ 2669.333444] pcieport 0000:06:00.0: bridge window [mem > 0xa4000000-0xafffffff] > [ 2669.334130] pcieport 0000:06:00.0: bridge window [mem > 0x6000000000-0x601fffffff 64bit pref] > [ 2669.334821] pcieport 0000:07:01.0: PCI bridge to [bus 08] > [ 2669.335460] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff] > [ 2669.336063] pcieport 0000:07:01.0: bridge window [mem > 0xa4000000-0xa57fffff] > [ 2669.336657] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x600bffffff 64bit pref] > [ 2669.337442] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit > pref]: assigned > [ 2669.338073] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit > pref]: assigned > [ 2673.200263] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit > pref]: releasing > [ 2673.201935] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit > pref]: releasing > [ 2673.203801] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x600bffffff 64bit pref]: releasing > [ 2673.205461] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x6017ffffff 64bit pref]: assigned > [ 2673.206197] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit > pref]: assigned > [ 2673.206800] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit > pref]: assigned > [ 2673.207534] pcieport 0000:06:00.0: PCI bridge to [bus 07-09] > [ 2673.208143] pcieport 0000:06:00.0: bridge window [io 0x4000-0x6fff] > [ 2673.208734] pcieport 0000:06:00.0: bridge window [mem > 0xa4000000-0xafffffff] > [ 2673.209323] pcieport 0000:06:00.0: bridge window [mem > 0x6000000000-0x601fffffff 64bit pref] > [ 2673.209916] pcieport 0000:07:01.0: PCI bridge to [bus 08] > [ 2673.210526] pcieport 0000:07:01.0: bridge window [io 0x4000-0x4fff] > [ 2673.211170] pcieport 0000:07:01.0: bridge window [mem > 0xa4000000-0xa57fffff] > [ 2673.211755] pcieport 0000:07:01.0: bridge window [mem > 0x6000000000-0x6017ffffff 64bit pref] > > > Unsure what else from dmesg or kernel config is relevant here. Can post the > full 100k here or somewhere else. Or excerpts. > > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable > [ 0.000000] BIOS-e820: [mem 0x000000000009f000-0x00000000000fffff] > reserved > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000006f23bfff] usable > [ 0.000000] BIOS-e820: [mem 0x000000006f23c000-0x000000006f23cfff] > reserved > [ 0.000000] BIOS-e820: [mem 0x000000006f23d000-0x000000007a792fff] usable > [ 0.000000] BIOS-e820: [mem 0x000000007a793000-0x000000007fa6cfff] > reserved > [ 0.000000] BIOS-e820: [mem 0x000000007fa6d000-0x000000007fca9fff] ACPI > NVS > [ 0.000000] BIOS-e820: [mem 0x000000007fcaa000-0x000000007fd0efff] ACPI > data > [ 0.000000] BIOS-e820: [mem 0x000000007fd0f000-0x000000007fd0ffff] usable > [ 0.000000] BIOS-e820: [mem 0x000000007fd10000-0x0000000087ffffff] > reserved > [ 0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000887fffff] usable > [ 0.000000] BIOS-e820: [mem 0x0000000088800000-0x000000008f7fffff] > reserved > [ 0.000000] BIOS-e820: [mem 0x00000000fe010000-0x00000000fe010fff] > reserved > [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000106c7fffff] usable > [ 0.000000] efi: Not removing mem55: MMIO range=[0xfe010000-0xfe010fff] > (4KB) from e820 map > [ 0.000003] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved > [ 0.000006] e820: remove [mem 0x000a0000-0x000fffff] usable > [ 0.503260] e820: reserve RAM buffer [mem 0x0009f000-0x0009ffff] > [ 0.503262] e820: reserve RAM buffer [mem 0x6f23c000-0x6fffffff] > [ 0.503263] e820: reserve RAM buffer [mem 0x7a793000-0x7bffffff] > [ 0.503264] e820: reserve RAM buffer [mem 0x7fd10000-0x7fffffff] > [ 0.503265] e820: reserve RAM buffer [mem 0x88800000-0x8bffffff] > [ 0.503266] e820: reserve RAM buffer [mem 0x106c800000-0x106fffffff] > > > [ 0.255546] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 > [ 0.445755] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM > ClockPM Segments MSI HPX-Type3] > [ 0.447023] acpi PNP0A08:00: _OSC: platform does not support [AER] > [ 0.449507] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME > PCIeCapability LTR] > [ 0.449511] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using > BIOS configuration > > [ 0.522380] caller snb_uncore_imc_init_box+0x73/0xd0 mapping multiple > BARs > > > Thanks, > > > Dag B >
