On Wed, 25 Oct 2023 08:28:44 +0000 "Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote: > > From: Alex Williamson <alex.williamson@xxxxxxxxxx> > > Sent: Tuesday, October 24, 2023 10:29 PM > > > > On Tue, 24 Oct 2023 14:03:25 +0000 > > Ankit Agrawal <ankita@xxxxxxxxxx> wrote: > > > > > >> > After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated > > > >> > as an IO Port BAR, it occurs to me that there's no config space > > > >> > emulation of BAR2 (or BAR3) here. Doesn't this mean that QEMU > > registers > > > >> > the BAR as 32-bit, non-prefetchable? ie. VFIOBAR.type & .mem64 are > > > >> > wrong? > > > >> > > > >> Maybe I didn't understand the question, but the PCI config space > > read/write > > > >> would still be handled by vfio_pci_core_read/write() which returns the > > > >> appropriate flags. I have checked that the device BARs are 64b and > > > >> prefetchable in the VM. > > > > > > > > vfio_pci_core_read/write() accesses the physical device, which doesn't > > > > implement BAR2. Why would an unimplemented BAR2 on the physical > > device > > > > report 64-bit, prefetchable? > > > > > > > > QEMU records VFIOBAR.type and .mem64 from reading the BAR register > > in > > > > vfio_bar_prepare() and passes this type to pci_register_bar() in > > > > vfio_bar_register(). Without an implementation of a config space read > > > > op in the variant driver and with no physical implementation of BAR2 on > > > > the device, I don't see how we get correct values in these fields. > > > > > > I think I see the cause of confusion. There are real PCIe compliant BARs > > > present on the device, just that it isn't being used once the C2C > > > interconnect is active. The BARs are 64b prefetchable. Here it the lspci > > > snippet of the device on the host. > > > # lspci -v -s 9:1:0.0 > > > 0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1) > > > Subsystem: NVIDIA Corporation Device 16eb > > > Physical Slot: 0-5 > > > Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node 0, > > IOMMU group 19 > > > Memory at 661002000000 (64-bit, prefetchable) [size=16M] > > > Memory at 662000000000 (64-bit, prefetchable) [size=128G] > > > Memory at 661000000000 (64-bit, prefetchable) [size=32M] > > > > > > I suppose this answers the BAR sizing question as well? > > > > Does this BAR2 size match the size we're reporting for the region? Now > > I'm confused why we need to intercept the BAR2 region info if there's > > physically a real BAR behind it. Thanks, > > > > same confusion. > > probably vfio-pci-core can include a helper for cfg space emulation > on emulated BARs to be used by all variant drivers in that category? > > btw intel vgpu also includes an emulation of BAR sizing. same for > future SIOV devices. so there sounds like a general requirement but > of course sharing it between vfio-pci and mdev/siov would be more > difficult. Yes, I see a need for this in the virtio-vfio-pci driver as well. It would simplify config emulation a lot if the variant driver could manipulate the perm_bits.virt and .write bit arrays and simply update vconfig for things like device-id and revision. Thanks, Alex
