Reproduced the issue with Q35 machine type and turning off secureboot. Debugging will be easier now. -----Original Message----- From: Kallol Biswas [C] <kallol.biswas@xxxxxxxxxxx> Sent: Friday, December 9, 2022 10:45 AM To: linux-pci@xxxxxxxxxxxxxxx Subject: uefi secureboot vm and IO window overlap We are observing an io window overlap issue in a secureboot enabled uefi vm. Linux displays: pci 0000:00:1d.0: can't claim BAR 4 [io 0x92a0-0x92bf]: address conflict with PCI Bus 0000:01 [io 0x9000-0x9fff] Eventually conflict gets resolved but we need to understand why get the conflict in the first place. Details: The VM is a uefi based VM and the issue shows up if secure boot is enabled. We have enabled ovmf log and uefi/ovmf programs a bridge IO window with the range 0x9000-0x91ff, but in Linux we see the same bridge is programmed with 0x9000-0x9fff. This results in an address conflict with subsequent devices. The PCI tree(lspci -t): -[0000:00]-+-00.0 +-01.0 +-02.0-[01]----00.0 +-02.1-[02]----00.0 +-02.2-[03]----00.0 +-02.3-[04]-- +-1d.0 +-1d.1 +-1d.2 +-1d.7 +-1f.0 +-1f.2 \-1f.3 [root@localhost ~]# lspci -vvv -s 0:02.0 | grep "I/O" Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ I/O behind bridge: 00009000-00009fff [size=4K] lspci -vvv -s 0:1d.0 | grep "I/O" Region 4: I/O ports at 1040 [size=32] root@localhost ~]# lspci -vvv -s 01:00.0 01:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01) Subsystem: Red Hat, Inc. Device 1100 Physical Slot: 0 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 22 Region 1: Memory at c1600000 (32-bit, non-prefetchable) [size=4K] Region 4: Memory at 84000000000 (64-bit, prefetchable) [size=16K] Expansion ROM at c1640000 [disabled] [size=256K] Capabilities: [dc] MSI-X: Enable+ Count=3 Masked- The uefi/ovmf log: PciBus: HostBridge->NotifyPhase(AllocateResources) - Success Process Option ROM: BAR Base/Length = C1800000/40000 PciBus: Resource Map for Root Bridge PciRoot(0x0) Type = Io16; Base = 0x6000; Length = 0x4000; Alignment = 0xFFF Base = 0x6000; Length = 0x200; Alignment = 0xFFF; Owner = PPB [00|02|03:**] Base = 0x7000; Length = 0x200; Alignment = 0xFFF; Owner = PPB [00|02|02:**] Base = 0x8000; Length = 0x200; Alignment = 0xFFF; Owner = PPB [00|02|01:**] Base = 0x9000; Length = 0x200; Alignment = 0xFFF; Owner = PPB [00|02|00:**] Base = 0x9200; Length = 0x40; Alignment = 0x3F; Owner = PCI [00|1F|03:20] Base = 0x9240; Length = 0x20; Alignment = 0x1F; Owner = PCI [00|1F|02:20] Base = 0x9260; Length = 0x20; Alignment = 0x1F; Owner = PCI [00|1D|02:20] Base = 0x9280; Length = 0x20; Alignment = 0x1F; Owner = PCI [00|1D|01:20] Base = 0x92A0; Length = 0x20; Alignment = 0x1F; Owner = PCI [00|1D|00:20] Type = Mem32; Base = 0xC0000000; Length = 0x1900000; Alignment = 0xFFFFFF Base = 0xC0000000; Length = 0x1000000; Alignment = 0xFFFFFF; Owner = PCI [00|01|00:10]; Type = PMem32 Base = 0xC1000000; Length = 0x200000; Alignment = 0x1FFFFF; Owner = PPB [00|02|03:**] Base = 0xC1200000; Length = 0x200000; Alignment = 0x1FFFFF; Owner = PPB [00|02|02:**] Base = 0xC1400000; Length = 0x200000; Alignment = 0x1FFFFF; Owner = PPB [00|02|01:**] Base = 0xC1600000; Length = 0x200000; Alignment = 0x1FFFFF; Owner = PPB [00|02|00:**] Base = 0xC1800000; Length = 0x40000; Alignment = 0x3FFFF; Owner = PCI [00|00|00:00]; Type = OpRom Base = 0xC1840000; Length = 0x1000; Alignment = 0xFFF; Owner = PCI [00|1F|02:24] Base = 0xC1841000; Length = 0x1000; Alignment = 0xFFF; Owner = PCI [00|1D|07:10] Base = 0xC1842000; Length = 0x1000; Alignment = 0xFFF; Owner = PPB [00|02|03:10] Base = 0xC1843000; Length = 0x1000; Alignment = 0xFFF; Owner = PPB [00|02|02:10] Base = 0xC1844000; Length = 0x1000; Alignment = 0xFFF; Owner = PPB [00|02|01:10] Base = 0xC1845000; Length = 0x1000; Alignment = 0xFFF; Owner = PPB [00|02|00:10] Base = 0xC1846000; Length = 0x1000; Alignment = 0xFFF; Owner = PCI [00|01|00:18] Type = Mem64; Base = 0x84000000000; Length = 0x300000; Alignment = 0xFFFFF Base = 0x84000000000; Length = 0x100000; Alignment = 0xFFFFF; Owner = PPB [00|02|00:**]; Type = PMem64 Base = 0x84000100000; Length = 0x100000; Alignment = 0xFFFFF; Owner = PPB [00|02|01:**]; Type = PMem64 Base = 0x84000200000; Length = 0x100000; Alignment = 0xFFFFF; Owner = PPB [00|02|02:**]; Type = PMem64 PciBus: Resource Map for Bridge [00|02|00] Type = Io16; Base = 0x9000; Length = 0x200; Alignment = 0xFFF Base = Padding; Length = 0x200; Alignment = 0x1FF Type = Mem32; Base = 0xC1600000; Length = 0x200000; Alignment = 0x1FFFFF Base = Padding; Length = 0x200000; Alignment = 0x1FFFFF Base = 0xC1600000; Length = 0x1000; Alignment = 0xFFF; Owner = PCI [01|00|00:14] Type = Mem32; Base = 0xC1845000; Length = 0x1000; Alignment = 0xFFF Type = PMem64; Base = 0x84000000000; Length = 0x100000; Alignment = 0xFFFFF Base = 0x84000000000; Length = 0x4000; Alignment = 0x3FFF; Owner = PCI [01|00|00:20] PciBus: Resource Map for Bridge [00|02|01] Type = Io16; Base = 0x8000; Length = 0x200; Alignment = 0xFFF Base = Padding; Length = 0x200; Alignment = 0x1FF Type = Mem32; Base = 0xC1400000; Length = 0x200000; Alignment = 0x1FFFFF Base = Padding; Length = 0x200000; Alignment = 0x1FFFFF Base = 0xC1400000; Length = 0x1000; Alignment = 0xFFF; Owner = PCI [02|00|00:14] Type = Mem32; Base = 0xC1844000; Length = 0x1000; Alignment = 0xFFF Type = PMem64; Base = 0x84000100000; Length = 0x100000; Alignment = 0xFFFFF Base = 0x84000100000; Length = 0x4000; Alignment = 0x3FFF; Owner = PCI [02|00|00:20] PciBus: Resource Map for Bridge [00|02|02] Type = Io16; Base = 0x7000; Length = 0x200; Alignment = 0xFFF Base = Padding; Length = 0x200; Alignment = 0x1FF Type = Mem32; Base = 0xC1200000; Length = 0x200000; Alignment = 0x1FFFFF Base = Padding; Length = 0x200000; Alignment = 0x1FFFFF Type = Mem32; Base = 0xC1843000; Length = 0x1000; Alignment = 0xFFF Type = PMem64; Base = 0x84000200000; Length = 0x100000; Alignment = 0xFFFFF Base = 0x84000200000; Length = 0x4000; Alignment = 0x3FFF; Owner = PCI [03|00|00:20] PciBus: Resource Map for Bridge [00|02|03] Type = Io16; Base = 0x6000; Length = 0x200; Alignment = 0xFFF Base = Padding; Length = 0x200; Alignment = 0x1FF Type = Mem32; Base = 0xC1000000; Length = 0x200000; Alignment = 0x1FFFFF Base = Padding; Length = 0x200000; Alignment = 0x1FFFFF Type = Mem32; Base = 0xC1842000; Length = 0x1000; Alignment = 0x The bus 1 is off 02:00.0 bridge device. IO resource for the bridge: Base = 0x9000; Length = 0x200; Alignment = 0xFFF; Owner = PPB [00|02|00:**] The alignment is 0xfff, start 0x9000, and length 0x200. So 0x9000-0x91ff would be sufficient for this, there would be no conflict. Linux appears to assign 0x9000-0x9fff to the bridge, resulting in conflicts with subsequent devices. [root@localhost ~]# dmesg | grep conflict [ 0.426170] pci 0000:00:1d.0: can't claim BAR 4 [io 0x92a0-0x92bf]: address conflict with PCI Bus 0000:01 [io 0x9000-0x9fff] [ 0.426217] pci 0000:00:1d.1: can't claim BAR 4 [io 0x9280-0x929f]: address conflict with PCI Bus 0000:01 [io 0x9000-0x9fff] [ 0.426228] pci 0000:00:1d.2: can't claim BAR 4 [io 0x9260-0x927f]: address conflict with PCI Bus 0000:01 [io 0x9000-0x9fff] [ 0.426258] pci 0000:00:1f.2: can't claim BAR 4 [io 0x9240-0x925f]: address conflict with PCI Bus 0000:01 [io 0x9000-0x9fff] [ 0.426270] pci 0000:00:1f.3: can't claim BAR 4 [io 0x9200-0x923f]: address conflict with PCI Bus 0000:01 [io 0x9000-0x9fff] Dmesg output with debug=1 "dyndbg=file *pci* +pfm" [ 0.280118] probe:pci_scan_child_bus_extend: pci_bus 0000:01: fixups for bus [ 0.280122] pci 0000:00:02.0: PCI bridge to [bus 01] [ 0.280141] pci 0000:00:02.0: bridge window [io 0x9000-0x9fff] [ 0.280158] pci 0000:00:02.0: bridge window [mem 0xc1600000-0xc17fffff] [ 0.280191] pci 0000:00:02.0: bridge window [mem 0x8000000000-0x80000fffff 64bit pref] [ 0.280194] probe:pci_scan_child_bus_extend: pci_bus 0000:01: bus scan returning with max=01 Nucleodyne @ Nutanix 408-718-8164
