On 23/10/17 11:37, Kenneth Lee wrote: > On Mon, Oct 23, 2017 at 09:31:25AM +0100, Jean-Philippe Brucker wrote: >> Date: Mon, 23 Oct 2017 09:31:25 +0100 >> From: Jean-Philippe Brucker <jean-philippe.brucker@xxxxxxx> >> To: Kenneth Lee <liguozhu@xxxxxxxxxxxxx>, Alex Williamson >> <alex.williamson@xxxxxxxxxx> >> CC: "kenneth-lee-2012@xxxxxxxxxxx" <kenneth-lee-2012@xxxxxxxxxxx>, Jon >> Masters <jcm@xxxxxxxxxxxxxx>, "jcm@xxxxxxxxxx" <jcm@xxxxxxxxxx>, >> "xuzaibo@xxxxxxxxxx" <xuzaibo@xxxxxxxxxx>, >> "linux-accelerators@xxxxxxxxxxxxxxxx" >> <linux-accelerators@xxxxxxxxxxxxxxxx> >> Subject: Re: To extend the feature of vfio-mdev >> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 >> Thunderbird/52.4.0 >> Message-ID: <6fed6c27-4b38-7d65-ee5d-63d6bb283e95@xxxxxxx> >> >> On 23/10/17 06:18, Kenneth Lee wrote: >>>>>>> 3. Let iommu driver accept more than one iommu_domain for the same >>>>>>> device. The substream id or pasid should be support for that >>>>>> >>>>>> You're really extending the definition of an iommu_domain to include >>>>>> PASID to do this, I don't think it makes sense in the general case. So >>>>>> perhaps you're talking about a PASID management layer sitting on top of >>>>>> an iommu_domain. AIUI for PCIe, a device has a requester ID which is >>>>>> used to find the context entry for that device. The IOMMU may support >>>>>> PASID, which would cause a first level lookup via those set of page >>>>>> tables, or it might only support second level translation. The >>>>>> iommu_domain is a reflection of that initial, single requester ID. >>>>> >>>>> Maybe I misunderstand this. But the IOMMU hardware, such as SMMU for ARM, >>>>> support multiple page table and is referred by something like ASID. If we should >>>>> support it in Linux, iommu_domain should be the best choice (no matter you call >>>>> it cookie or id or something else). Or where you can get a object referring to it? >>>> >>>> For PASID, a PASID is unique only within the requester ID. I don't >>>> know of anything equivalent to your ASID within PCIe. >>>> >>> >>> As my understanding to the ARM Spec: >>> >>> PCIE[PASID]=ARM_SMMU[ASID]=ARM_SMMU[SubstreamID], >>> PCIE[RequestID]=ARM_SMMU[StreamID] >> >> In the SMMUv3 spec, the PASID is dissociated from the ASID. We set the >> ASID in the context descriptor indexed by PASID, which provides more >> flexibility when the CPU's ASID capacity and endpoint's PASID capacity differ. > > Yes. Sorry, I tried to make the concept simply. But it is wrong slightly. That's all right :) I'm still not sure if we should implement ASID=PASID for arm. I think it would be overly complicated to implement, because of the capacity difference and because it prevents users from allocating and controlling PASIDs contexts on their own with map/unmap, without attaching to a Linux process (some users want this). >>> For ARM SMMU, the Stream ID is used to index a Context Descriptors >>> Table while and Sub-stream ID is used to index a Descriptor which refer >>> to a general page table in the same format as MMU. >>> >>> So both RequestID and PASID uniquely identify a address space. Then >>> every iommu-enabled device can service more than one user process at the >>> same time. >>> >>> So the IOMMU should support more than one page table, which in turn >>> should be added to somewhere in Linux. If the iommu_domain refer to one >>> address space, the iommu driver should accept more than one >>> iommu_domain. >> >> The current design choice is to have multiple address space per >> iommu_domain (and one PASID table per domain). It fits better with >> existing IOMMU and VFIO APIs. >> > > I am still reading the patchset. But if the PASID is used to index the address > space table in iommu_domain, it will become a general concept in the whole > system. But if we would bind the address space with iommu_domain, the > iommu_domain become a handle for any process to control the device dma. It will > be easier for one process with two or more domains to control the device for > different purpose. And it is also easier for a process hand over the domain to > another without change anything. And the process need not to know anything about > pasid. Unfortunately this cannot work with vfio-pci, where one userspace process owns the whole function. Userspace has to obtain the PASID from the kernel allocator and somehow program it into the device, with an implementation-defined method. > A domain is a handle to control the device separately. I think this make > thing simpler. Similarly, how different parts of a function can be controlled and assigned separate PASIDs is implementation-defined in PCI. The userspace driver is given a single device fd in a VFIO container, and has to do the partitioning itself. Thanks, Jean
