Re: PCIE BAR resizing blocked by another BAR on same device?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



[+to Christian, resizable BAR expert]

On Wed, Apr 17, 2024 at 03:16:06PM +0200, Dag B wrote:
> Hi.
> 
> In short, I have a GPU for which lspci reveals:
> 
> Capabilities: [bb0 v1] Physical Resizable BAR
> 
>         BAR 0: current size: 16MB, supported: 16MB
>         BAR 1: current size: 128MB, supported: 64MB 128MB 256MB 512MB 1GB
> 2GB 4GB 8GB 16GB 32GB
>         BAR 3: current size: 32MB, supported: 32MB
> 
> In dmesg I see:
> 
> [    0.517191] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: assigned
> [    0.517238] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: assigned
> [    0.517261] pci 0000:08:00.0: BAR 0 [mem 0xa4000000-0xa4ffffff]: assigned
> 
> I take it the location of BAR 3 right after BAR 1 explains why I get:
> 
> p53 # echo 9 > resource1_resize
> -bash: echo: write error: No space left on device
> 
> Shrinking it and increasing it to the orginal size works.
> 
> 
> Is there anything I can do with current kernel functionality to reserve
> memory address space for the full-fat BAR 1? Or relocate it?
> 
> If not, is this something which *can* be worked around in the kernel? And if
> so, does it belong with the PCI subsystem? Or the devicedriver for the
> device in question?
> 
> Is there a good ELI13 resource explaining how resizable BAR works in Linux?
> 
> My current kernel command-line contains: pci=assign-busses,realloc
> 
> My GPU is attached via TB3 to a system for which resizable BAR is and will
> remain a foreign concept in the BIOS.
> 
> dmesg excerpt below reflects resizing the BAR to 128MB and then back to
> 256MB.
> 
> [ 1730.091789] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: releasing
> [ 1730.091875] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: releasing
> [ 1730.092072] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: releasing
> [ 1730.092151] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: assigned
> [ 1730.092223] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: assigned
> [ 1730.092335] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: assigned
> [ 1730.092444] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 1730.092510] pcieport 0000:06:00.0:   bridge window [io 0x4000-0x6fff]
> [ 1730.092595] pcieport 0000:06:00.0:   bridge window [mem
> 0xa4000000-0xafffffff]
> [ 1730.092680] pcieport 0000:06:00.0:   bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 1730.092778] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 1730.092850] pcieport 0000:07:01.0:   bridge window [io 0x4000-0x4fff]
> [ 1730.092933] pcieport 0000:07:01.0:   bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 1730.093018] pcieport 0000:07:01.0:   bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]
> [ 1759.817306] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: releasing
> [ 1759.817394] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: releasing
> [ 1759.817591] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: releasing
> [ 1759.817668] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: assigned
> [ 1759.817740] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: assigned
> [ 1759.817853] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: assigned
> [ 1759.817964] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 1759.818035] pcieport 0000:06:00.0:   bridge window [io 0x4000-0x6fff]
> [ 1759.818120] pcieport 0000:06:00.0:   bridge window [mem
> 0xa4000000-0xafffffff]
> [ 1759.818204] pcieport 0000:06:00.0:   bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 1759.818303] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 1759.818374] pcieport 0000:07:01.0:   bridge window [io 0x4000-0x4fff]
> [ 1759.818459] pcieport 0000:07:01.0:   bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 1759.818544] pcieport 0000:07:01.0:   bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]
> [ 1769.797178] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: releasing
> [ 1769.797241] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: releasing
> [ 1769.797417] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: releasing
> [ 1769.797473] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: can't assign; no space
> [ 1769.797515] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: failed to assign
> [ 1769.797557] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> can't assign; no space
> [ 1769.797594] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> failed to assign
> [ 1769.797630] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> can't assign; no space
> [ 1769.797666] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> failed to assign
> [ 1769.797703] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 1769.797761] pcieport 0000:06:00.0:   bridge window [io 0x4000-0x6fff]
> [ 1769.797829] pcieport 0000:06:00.0:   bridge window [mem
> 0xa4000000-0xafffffff]
> [ 1769.797895] pcieport 0000:06:00.0:   bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 1769.797972] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 1769.798027] pcieport 0000:07:01.0:   bridge window [io 0x4000-0x4fff]
> [ 1769.798089] pcieport 0000:07:01.0:   bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 1769.798155] pcieport 0000:07:01.0:   bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]
> [ 1769.798270] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: assigned
> [ 1769.798358] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: assigned
> [ 2669.324929] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: releasing
> [ 2669.324992] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: releasing
> [ 2669.325164] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: releasing
> [ 2669.325219] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: can't assign; no space
> [ 2669.327023] pcieport 0000:07:01.0: bridge window [mem size 0x30000000
> 64bit pref]: failed to assign
> [ 2669.328798] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> can't assign; no space
> [ 2669.330482] pci 0000:08:00.0: BAR 1 [mem size 0x20000000 64bit pref]:
> failed to assign
> [ 2669.331104] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> can't assign; no space
> [ 2669.331682] pci 0000:08:00.0: BAR 3 [mem size 0x02000000 64bit pref]:
> failed to assign
> [ 2669.332258] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 2669.332855] pcieport 0000:06:00.0:   bridge window [io 0x4000-0x6fff]
> [ 2669.333444] pcieport 0000:06:00.0:   bridge window [mem
> 0xa4000000-0xafffffff]
> [ 2669.334130] pcieport 0000:06:00.0:   bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 2669.334821] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 2669.335460] pcieport 0000:07:01.0:   bridge window [io 0x4000-0x4fff]
> [ 2669.336063] pcieport 0000:07:01.0:   bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 2669.336657] pcieport 0000:07:01.0:   bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]
> [ 2669.337442] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: assigned
> [ 2669.338073] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: assigned
> [ 2673.200263] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x6007ffffff 64bit
> pref]: releasing
> [ 2673.201935] pci 0000:08:00.0: BAR 3 [mem 0x6008000000-0x6009ffffff 64bit
> pref]: releasing
> [ 2673.203801] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x600bffffff 64bit pref]: releasing
> [ 2673.205461] pcieport 0000:07:01.0: bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]: assigned
> [ 2673.206197] pci 0000:08:00.0: BAR 1 [mem 0x6000000000-0x600fffffff 64bit
> pref]: assigned
> [ 2673.206800] pci 0000:08:00.0: BAR 3 [mem 0x6010000000-0x6011ffffff 64bit
> pref]: assigned
> [ 2673.207534] pcieport 0000:06:00.0: PCI bridge to [bus 07-09]
> [ 2673.208143] pcieport 0000:06:00.0:   bridge window [io 0x4000-0x6fff]
> [ 2673.208734] pcieport 0000:06:00.0:   bridge window [mem
> 0xa4000000-0xafffffff]
> [ 2673.209323] pcieport 0000:06:00.0:   bridge window [mem
> 0x6000000000-0x601fffffff 64bit pref]
> [ 2673.209916] pcieport 0000:07:01.0: PCI bridge to [bus 08]
> [ 2673.210526] pcieport 0000:07:01.0:   bridge window [io 0x4000-0x4fff]
> [ 2673.211170] pcieport 0000:07:01.0:   bridge window [mem
> 0xa4000000-0xa57fffff]
> [ 2673.211755] pcieport 0000:07:01.0:   bridge window [mem
> 0x6000000000-0x6017ffffff 64bit pref]
> 
> 
> Unsure what else from dmesg or kernel config is relevant here. Can post the
> full 100k here or somewhere else. Or excerpts.
> 
> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
> [    0.000000] BIOS-e820: [mem 0x000000000009f000-0x00000000000fffff]
> reserved
> [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000006f23bfff] usable
> [    0.000000] BIOS-e820: [mem 0x000000006f23c000-0x000000006f23cfff]
> reserved
> [    0.000000] BIOS-e820: [mem 0x000000006f23d000-0x000000007a792fff] usable
> [    0.000000] BIOS-e820: [mem 0x000000007a793000-0x000000007fa6cfff]
> reserved
> [    0.000000] BIOS-e820: [mem 0x000000007fa6d000-0x000000007fca9fff] ACPI
> NVS
> [    0.000000] BIOS-e820: [mem 0x000000007fcaa000-0x000000007fd0efff] ACPI
> data
> [    0.000000] BIOS-e820: [mem 0x000000007fd0f000-0x000000007fd0ffff] usable
> [    0.000000] BIOS-e820: [mem 0x000000007fd10000-0x0000000087ffffff]
> reserved
> [    0.000000] BIOS-e820: [mem 0x0000000088000000-0x00000000887fffff] usable
> [    0.000000] BIOS-e820: [mem 0x0000000088800000-0x000000008f7fffff]
> reserved
> [    0.000000] BIOS-e820: [mem 0x00000000fe010000-0x00000000fe010fff]
> reserved
> [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000106c7fffff] usable
> [    0.000000] efi: Not removing mem55: MMIO range=[0xfe010000-0xfe010fff]
> (4KB) from e820 map
> [    0.000003] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
> [    0.000006] e820: remove [mem 0x000a0000-0x000fffff] usable
> [    0.503260] e820: reserve RAM buffer [mem 0x0009f000-0x0009ffff]
> [    0.503262] e820: reserve RAM buffer [mem 0x6f23c000-0x6fffffff]
> [    0.503263] e820: reserve RAM buffer [mem 0x7a793000-0x7bffffff]
> [    0.503264] e820: reserve RAM buffer [mem 0x7fd10000-0x7fffffff]
> [    0.503265] e820: reserve RAM buffer [mem 0x88800000-0x8bffffff]
> [    0.503266] e820: reserve RAM buffer [mem 0x106c800000-0x106fffffff]
> 
> 
> [    0.255546] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
> [    0.445755] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM
> ClockPM Segments MSI HPX-Type3]
> [    0.447023] acpi PNP0A08:00: _OSC: platform does not support [AER]
> [    0.449507] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME
> PCIeCapability LTR]
> [    0.449511] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using
> BIOS configuration
> 
> [    0.522380] caller snb_uncore_imc_init_box+0x73/0xd0 mapping multiple
> BARs
> 
> 
> Thanks,
> 
> 
> Dag B
> 




[Index of Archives]     [DMA Engine]     [Linux Coverity]     [Linux USB]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [Greybus]

  Powered by Linux