On 2020-01-28 12:13 a.m., Skidanov, Alexey wrote: >> On 2020-01-27 12:12 p.m., Christian König wrote: >>> Am 27.01.20 um 17:58 schrieb Logan Gunthorpe: >>>> >>>> On 2020-01-27 1:18 a.m., Christian König wrote: >>>>> Am 27.01.20 um 08:18 schrieb Skidanov, Alexey: >>>>>> Hello, >>>>>> >>>>>> I have recently found the below commit to disabling ACS bits. Using kernel parameter >> is pretty simple but requires to know in advance which devices might be participated in >> peer-to-peer sessions. >>>>>> >>>>>> Why we can't disable the ACS bits *after* the driver is initialized (and there is a >> request to connect between two peers) and not *during* device discovering?. >>>>> That's exactly what was initially proposed but we have seen hardware >>>>> which reacts allergic to disabling those bits on the fly. >>>> I wasn't aware of that and haven't seen anything like that. >>>> >>>>> Please read up the discussion on the mailing list leading to this patch. >>>> The issue was the IOMMU code does not allow for any kind of dynamic >>>> changes in the groups devices are assigned in. In theory, this could be >>>> possible but you'd still at least have to unbind the devices from their >>>> driver because you definitely can't change the IOMMU group while there >>>> are DMA requests in flight. Ultimately it's easier for most use cases to >>>> just disable it on boot. >>> >>> As far as I know you can't change the ACS bit either when there are >>> transactions in flight on the affected devices/bridges. >> >> No, I think the ACS bits are fine to change at any time. I've never had >> any issue changing them. The problem is the act of changing them changes >> the isolation between the devices which means the IOMMU groups have to >> change. >> >> It's certainly possible today to just use setpci to adjust those bits at >> any time. It just means the isolation the IOMMU is expecting will be >> wrong and that may mean you broke the security between VMs on your machine. >> > > According to the PCIe spec, there are two mechanisms to deal with isolation: > - Redirected Request Validation logic within the RC and > - ACS P2P Egress Control > So anyone who cares about the isolation must use at least one of these mechanisms. > I would expect that on VM creation, the above mechanisms will be configured appropriately. That was my expectation too, a long time ago, but that is not how it works. IOMMU groups are created, on boot, based on the ACS bit settings (and other mechanisms). Any devices that are not isolated are put in the same groups. The groups are then used to program the IOMMU such that each group has it's own virtual address region. When a VM is created with a PCI passthru device, it will fail if you don't pass through every device in a each group involved. If the ACS bits are updated after boot, there is no mechanism to change the groups so you could create a VM with a device that is not isolated. Logan
