Hello Bjorn,
I have reproduced the problem in the 6.3.6 kernel and debugged the source of the conflict.
Here is the OVMF log:
PciBus: Resource Map for Root Bridge PciRoot(0x0)^M
Type = Io16; Base = 0x6000; Length = 0x3000; Alignment = 0xFFF^M
Base = 0x6000; Length = 0x200; Alignment = 0xFFF; Owner = PPB [00|02|02:**]^M
Base = 0x7000; Length = 0x200; Alignment = 0xFFF; Owner = PPB [00|02|01:**]^M
Base = 0x8000; Length = 0x200; Alignment = 0xFFF; Owner = PPB [00|02|00:**]^M
Base = 0x8200; Length = 0x40; Alignment = 0x3F; Owner = PCI [00|1F|03:20]^M
Base = 0x8240; Length = 0x20; Alignment = 0x1F; Owner = PCI [00|1F|02:20]^M
Base = 0x8260; Length = 0x20; Alignment = 0x1F; Owner = PCI [00|1D|02:20]^M
Base = 0x8280; Length = 0x20; Alignment = 0x1F; Owner = PCI [00|1D|01:20]^M
Base = 0x82A0; Length = 0x20; Alignment = 0x1F; Owner = PCI [00|1D|00:20]^M
Base = 0x82C0; Length = 0x20; Alignment = 0x1F; Owner = PCI [00|03|00:10]^M
[nutanix@localhost ~]$ lspci -s 0:2.0 -vvv
00:02.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 22
Region 0: Memory at c1645000 (32-bit, non-prefetchable) [size=4K]
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00008000-00008fff [size=4K]
Memory behind bridge: c1400000-c15fffff [size=2M]
Prefetchable memory behind bridge: 0000084000000000-00000840000fffff [size=1M]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport
Dmesg log:
[ 2.232081] pci 0000:00:02.0: PCI bridge to [bus 01]
[ 2.232098] pci_read_bridge_io:base 0x8000 limit 0x8000 io_granulatiry 0x1000
[ 2.232099] pci 0000:00:02.0: bridge window [io 0x8000-0x8fff]
[ 2.233005] pci 0000:00:02.0: bridge window [mem 0xc1400000-0xc15fffff]
[ 2.233034] pci 0000:00:02.0: bridge window [mem 0x84000000000-0x840000fffff 64bit
Kernel code:
static void pci_read_bridge_io(struct pci_bus *child)
{
struct pci_dev *dev = child->self;
u8 io_base_lo, io_limit_lo;
unsigned long io_mask, io_granularity, base, limit;
struct pci_bus_region region;
struct resource *res;
io_mask = PCI_IO_RANGE_MASK;
io_granularity = 0x1000;
if (dev->io_window_1k) {
/* Support 1K I/O space granularity */
io_mask = PCI_IO_1K_RANGE_MASK;
io_granularity = 0x400;
}
printk("pci_read_bridge_io:base 0x%x limit 0x%x io_granulatiry 0x%x\n", base, limit, io_granularity); <= my print
if (base <= limit) {
res->flags = (io_base_lo & PCI_IO_RANGE_TYPE_MASK) | IORESOURCE_IO;
region.start = base;
region.end = limit + io_granularity - 1;
pcibios_bus_to_resource(dev->bus, res, ®ion);
pci_info(dev, " bridge window %pR\n", res);
}
OVMF sets the base for 0:2.0 as 0x8000 and length as 0x200 but kernel io_granularity is 0x1000
So, the bridge window becomes 0x8000 to 0x8fff, which overlaps the OVMF programmed IO base
registers for other endpoints.
[ 2.996029] pci 0000:00:03.0: can't claim BAR 0 [io 0x82c0-0x82df]: address conflict with PCI Bus 0000:01 [io 0x8000-0x8fff]
[ 2.996049] pci 0000:00:1d.0: can't claim BAR 4 [io 0x82a0-0x82bf]: address conflict with PCI Bus 0000:01 [io 0x8000-0x8fff]
[ 2.996058] pci 0000:00:1d.1: can't claim BAR 4 [io 0x8280-0x829f]: address conflict with PCI Bus 0000:01 [io 0x8000-0x8fff]
[ 2.996068] pci 0000:00:1d.2: can't claim BAR 4 [io 0x8260-0x827f]: address conflict with PCI Bus 0000:01 [io 0x8000-0x8fff]
[ 2.996093] pci 0000:00:1f.2: can't claim BAR 4 [io 0x8240-0x825f]: address conflict with PCI Bus 0000:01 [io 0x8000-0x8fff]
[ 2.996997] pci 0000:00:1f.3: can't claim BAR 4 [io 0x8200-0x823f]: address conflict with PCI Bus 0000:01 [io 0x8000-0x8fff]
Sorry, did not get time to debug this before.
Question, why does the kernel set IO granularity to 4k?
Kallol
-----Original Message-----
From: Bjorn Helgaas <helgaas@xxxxxxxxxx>
Sent: Tuesday, December 13, 2022 1:31 PM
To: Kallol Biswas [C] <kallol.biswas@xxxxxxxxxxx>
Cc: linux-pci@xxxxxxxxxxxxxxx
Subject: Re: uefi secureboot vm and IO window overlap
On Sat, Dec 10, 2022 at 05:45:50PM +0000, Kallol Biswas [C] wrote:
> The part1 of the dmesg:
>
> [ 0.000000] Initializing cgroup subsys cpuset
> [ 0.000000] Initializing cgroup subsys cpu
> [ 0.000000] Initializing cgroup subsys cpuacct
> [ 0.000000] Linux version 3.10.0-957.el7.x86_64 (mockbuild@xxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Nov 8 23:39:32 UTC 2018
Is there any chance you can reproduce the problem on a current kernel?
If it's been fixed by now, maybe we could identify the fix and you could backport it?
Bjorn
