On Tue, Jun 25, 2024 at 09:01:50PM +0530, Vidya Sagar wrote: > PCIe ACS settings control the level of isolation and the possible P2P > paths between devices. With greater isolation the kernel will create > smaller iommu_groups and with less isolation there is more HW that > can achieve P2P transfers. From a virtualization perspective all > devices in the same iommu_group must be assigned to the same VM as > they lack security isolation. > > There is no way for the kernel to automatically know the correct > ACS settings for any given system and workload. Existing command line > options (ex:- disable_acs_redir) allow only for large scale change, > disabling all isolation, but this is not sufficient for more complex cases. > > Add a kernel command-line option 'config_acs' to directly control all the > ACS bits for specific devices, which allows the operator to setup the > right level of isolation to achieve the desired P2P configuration. > The definition is future proof, when new ACS bits are added to the spec > the open syntax can be extended. > > ACS needs to be setup early in the kernel boot as the ACS settings > effect how iommu_groups are formed. iommu_group formation is a one > time event during initial device discovery, changing ACS bits after > kernel boot can result in an inaccurate view of the iommu_groups > compared to the current isolation configuration. > > ACS applies to PCIe Downstream Ports and multi-function devices. > The default ACS settings are strict and deny any direct traffic > between two functions. This results in the smallest iommu_group the > HW can support. Frequently these values result in slow or > non-working P2PDMA. > > ACS offers a range of security choices controlling how traffic is > allowed to go directly between two devices. Some popular choices: > - Full prevention > - Translated requests can be direct, with various options > - Asymmetric direct traffic, A can reach B but not the reverse > - All traffic can be direct > Along with some other less common ones for special topologies. > > The intention is that this option would be used with expert knowledge > of the HW capability and workload to achieve the desired > configuration. > > Signed-off-by: Vidya Sagar <vidyas@xxxxxxxxxx> > --- > v4: > * Changed commit message (Courtesy: Jason) to provide more details > > v3: > * Fixed a documentation issue reported by kernel test bot > > v2: > * Refactored the code as per Jason's suggestion > > .../admin-guide/kernel-parameters.txt | 22 +++ > drivers/pci/pci.c | 148 +++++++++++------- > 2 files changed, 112 insertions(+), 58 deletions(-) Bjorn? Jason
