Hi Bjorn, Could you let me know if Jason's reply answers your question? Please let me know if you are looking for any more information. Thanks, Vidya Sagar > -----Original Message----- > From: Jason Gunthorpe <jgg@xxxxxxxxxx> > Sent: Thursday, May 23, 2024 8:46 PM > To: Bjorn Helgaas <helgaas@xxxxxxxxxx> > Cc: Vidya Sagar <vidyas@xxxxxxxxxx>; corbet@xxxxxxx; bhelgaas@xxxxxxxxxx; Gal > Shalom <galshalom@xxxxxxxxxx>; Leon Romanovsky <leonro@xxxxxxxxxx>; Thierry > Reding <treding@xxxxxxxxxx>; Jon Hunter <jonathanh@xxxxxxxxxx>; Masoud > Moshref Javadi <mmoshrefjava@xxxxxxxxxx>; Shahaf Shuler <shahafs@xxxxxxxxxx>; > Vikram Sethi <vsethi@xxxxxxxxxx>; Shanker Donthineni <sdonthineni@xxxxxxxxxx>; > Jiandi An <jan@xxxxxxxxxx>; Tushar Dave <tdave@xxxxxxxxxx>; linux- > doc@xxxxxxxxxxxxxxx; linux-pci@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; > Krishna Thota <kthota@xxxxxxxxxx>; Manikanta Maddireddy > <mmaddireddy@xxxxxxxxxx>; sagar.tv@xxxxxxxxx; Joerg Roedel <joro@xxxxxxxxxx>; > Will Deacon <will@xxxxxxxxxx>; Robin Murphy <robin.murphy@xxxxxxx>; > iommu@xxxxxxxxxxxxxxx > Subject: Re: [PATCH V3] PCI: Extend ACS configurability > > On Thu, May 23, 2024 at 09:59:36AM -0500, Bjorn Helgaas wrote: > > [+cc iommu folks] > > > > On Thu, May 23, 2024 at 12:05:28PM +0530, Vidya Sagar wrote: > > > For iommu_groups to form correctly, the ACS settings in the PCIe > > > fabric need to be setup early in the boot process, either via the > > > BIOS or via the kernel disable_acs_redir parameter. > > > > Can you point to the iommu code that is involved here? It sounds like > > the iommu_groups are built at boot time and are immutable after that? > > They are created when the struct device is plugged in. pci_device_group() does the > logic. > > Notably groups can't/don't change if details like ACS change after the groups are > setup. > > There are alot of instructions out there telling people to boot their servers and then > manually change the ACS flags with set_pci or something, and these are not good > instructions since it defeats the VFIO group based security mechanisms. > > > If we need per-device ACS config that depends on the workload, it > > seems kind of problematic to only be able to specify this at boot > > time. I guess we would need to reboot if we want to run a workload > > that needs a different config? > > Basically. The main difference I'd see is if the server is a VM host or running bare > metal apps. You can get more efficicenty if you change things for the bare metal case, > and often bare metal will want to turn the iommu off while a VM host often wants > more of it turned on. > > > Is this the iommu usage model we want in the long term? > > There is some path to more dynamic behavior here, but it would require separating > groups into two components - devices that are together because they are physically > sharing translation (aliases and things) from devices that are together because they > share a security boundary (ACS). > > It is more believable we could dynamically change security group assigments for VFIO > than translation group assignment. I don't know anyone interested in this right now - > Alex and I have only talked about it as a possibility a while back. > > FWIW I don't view patch as excluding more dynamisism in the future, but it is the best > way to work with the current state of affairs, and definitely better than set_pci > instructions. > > Thanks, > Jason
