On Tue, 9 May 2023 23:41:44 +0000
"Tian, Kevin" <kevin.tian@xxxxxxxxx> wrote:
> > From: Jason Gunthorpe <jgg@xxxxxxxxxx>
> > Sent: Wednesday, May 10, 2023 7:13 AM
> >
> > On Tue, May 09, 2023 at 10:57:04PM +0000, Tian, Kevin wrote:
> > > > From: Jason Gunthorpe <jgg@xxxxxxxxxx>
> > > > Sent: Wednesday, May 10, 2023 6:44 AM
> > > >
> > > > On Tue, May 09, 2023 at 08:34:53AM +0000, Tian, Kevin wrote:
> > > > > According to PCIe spec (7.8.9 PASID Extended Capability Structure):
> > > > >
> > > > > The PASID configuration of the single non-VF Function representing
> > > > > the device is also used by all VFs in the device. A PF is permitted
> > > > > to implement the PASID capability, but VFs must not implement it.
> > > > >
> > > > > To enable PASID on VF then one open is where to locate the PASID
> > > > > capability in VF's vconfig space. vfio-pci doesn't know which offset
> > > > > may contain VF specific config registers. Finding such offset must
> > > > > come from a device specific knowledge.
> > > >
> > > > Why? Can't vfio probe the cap tree and just find a gap to insert a new
> > > > cap? We already mangle the cap list, I'm not sure I see what
> > > > the problem is?
> > > >
> > >
> > > PCI config space includes not only caps, but also device specific
> > > defined fields. e.g. Intel IGD defines offset 0xfc as a pointer to
> > > OpRegion. I'm sure Alex can give many other examples.
> >
> > Do we even expose those over VIFO? I thought in general we blocked of
>
> Yes. I did a quick check:
>
> /*
> * Default unassigned regions to raw read-write access. Some devices
> * require this to function as they hide registers between the gaps in
> * config space (be2net). Like MMIO and I/O port registers, we have
> * to trust the hardware isolation.
> */
> static struct perm_bits unassigned_perms = {
> .readfn = vfio_raw_config_read,
> .writefn = vfio_raw_config_write
> };
>
> vfio_config_do_rw()
> {
> ...
> if (cap_id == PCI_CAP_ID_INVALID) {
> perm = &unassigned_perms;
> cap_start = *ppos;
> } ...
> }
>
> vfio_config_init()
> {
> ...
> memset(map, PCI_CAP_ID_BASIC, PCI_STD_HEADER_SIZEOF);
> memset(map + PCI_STD_HEADER_SIZEOF, PCI_CAP_ID_INVALID,
> pdev->cfg_size - PCI_STD_HEADER_SIZEOF);
> ...
> }
>
> > various parts of the config space. I keep seeing patches to unblock
> > parts of config space?
> >
> > I'd do the reverse and say devices that want to pass parts of their
> > config space should have a special hook to do it and otherwise we
> > should sanitize and block?
>
> This then may break backward compatibility. We don't know how
> many devices have such hidden registers so if anyone misses a hook
> immediately it cannot be assigned after we start blocking as default.
>
> >
> > eg we already have a hook to pass the opregion
> >
> > > So it's easy to find the gap between caps, but not easy to know
> > > whether that gap is actually free to use.
> >
> > Because, let's face it, this is a horrible thing to do, and the
> > opregion stuff is just ugly as s sin.
> >
>
> It's ugly, but that is the reality. :/
Have a peak at the config space of an NVIDIA GPU and tell me which of
those non-zero fields between capabilities are used as well. Glass
houses... ;-)
IIRC we originally needed to enable this for a Broadcom NIC that
stuffed device specific registers in un-architected config space. The
capabilities we're {un}hiding are architected things that we know are
unsupported or unsafe, the gaps, just like device specific
capabilities, we're obliged to expose for functionality. Thanks,
Alex
