On Tue, Sep 01, 2015 at 06:28:28PM +0200, Jan Kiszka wrote: > On 2015-09-01 18:02, Michael S. Tsirkin wrote: > > On Tue, Sep 01, 2015 at 05:34:37PM +0200, Jan Kiszka wrote: > >> On 2015-09-01 16:34, Michael S. Tsirkin wrote: > >>> On Tue, Sep 01, 2015 at 04:09:44PM +0200, Jan Kiszka wrote: > >>>> On 2015-09-01 11:24, Michael S. Tsirkin wrote: > >>>>> On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan Kiszka wrote: > >>>>>> On 2015-09-01 10:01, Michael S. Tsirkin wrote: > >>>>>>> On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote: > >>>>>>>> Leaving all the implementation and interface details aside, this > >>>>>>>> discussion is first of all about two fundamentally different approaches: > >>>>>>>> static shared memory windows vs. dynamically remapped shared windows (a > >>>>>>>> third one would be copying in the hypervisor, but I suppose we all agree > >>>>>>>> that the whole exercise is about avoiding that). Which way do we want or > >>>>>>>> have to go? > >>>>>>>> > >>>>>>>> Jan > >>>>>>> > >>>>>>> Dynamic is a superset of static: you can always make it static if you > >>>>>>> wish. Static has the advantage of simplicity, but that's lost once you > >>>>>>> realize you need to invent interfaces to make it work. Since we can use > >>>>>>> existing IOMMU interfaces for the dynamic one, what's the disadvantage? > >>>>>> > >>>>>> Complexity. Having to emulate even more of an IOMMU in the hypervisor > >>>>>> (we already have to do a bit for VT-d IR in Jailhouse) and doing this > >>>>>> per platform (AMD IOMMU, ARM SMMU, ...) is out of scope for us. In that > >>>>>> sense, generic grant tables would be more appealing. > >>>>> > >>>>> That's not how we do things for KVM, PV features need to be > >>>>> modular and interchangeable with emulation. > >>>> > >>>> I know, and we may have to make some compromise for Jailhouse if that > >>>> brings us valuable standardization and broad guest support. But we will > >>>> surely not support an arbitrary amount of IOMMU models for that reason. > >>>> > >>>>> > >>>>> If you just want something that's cross-platform and easy to > >>>>> implement, just build a PV IOMMU. Maybe use virtio for this. > >>>> > >>>> That is likely required to keep the complexity manageable and to allow > >>>> static preconfiguration. > >>> > >>> Real IOMMU allow static configuration just fine. This is exactly > >>> what VFIO uses. > >> > >> Please specify more precisely which feature in which IOMMU you are > >> referring to. Also, given that you refer to VFIO, I suspect we have > >> different thing in mind. I'm talking about an IOMMU device model, like > >> the one we have in QEMU now for VT-d. That one is not at all > >> preconfigured by the host for VFIO. > > > > I really just mean that VFIO creates a mostly static IOMMU configuration. > > > > It's configured by the guest, not the host. > > OK, that resolves my confusion. > > > > > I don't see host control over configuration as being particularly important. > > We do, see below. > > > > > > >>> > >>>> Well, we could declare our virtio-shmem device to be an IOMMU device > >>>> that controls access of a remote VM to RAM of the one that owns the > >>>> device. In the static case, this access may at most be enabled/disabled > >>>> but not moved around. The static regions would have to be discoverable > >>>> for the VM (register read-back), and the guest's firmware will likely > >>>> have to declare those ranges reserved to the guest OS. > >>>> In the dynamic case, the guest would be able to create an alternative > >>>> mapping. > >>> > >>> > >>> I don't think we want a special device just to support the > >>> static case. It might be a bit less code to write, but > >>> eventually it should be up to the guest. > >>> Fundamentally, it's policy that host has no business > >>> dictating. > >> > >> "A bit less" is to be validated, and I doubt its just "a bit". But if > >> KVM and its guests will also support some PV-IOMMU that we can reuse for > >> our scenarios, than that is fine. KVM would not have to mandate support > >> for it while we would, that's all. > > > > Someone will have to do this work. > > > >>> > >>>> We would probably have to define a generic page table structure > >>>> for that. Or do you rather have some MPU-like control structure in mind, > >>>> more similar to the memory region descriptions vhost is already using? > >>> > >>> I don't care much. Page tables use less memory if a lot of memory needs > >>> to be covered. OTOH if you want to use virtio (e.g. to allow command > >>> batching) that likely means commands to manipulate the IOMMU, and > >>> maintaining it all on the host. You decide. > >> > >> I don't care very much about the dynamic case as we won't support it > >> anyway. However, if the configuration concept used for it is applicable > >> to static mode as well, then we could reuse it. But preconfiguration > >> will required register-based region description, I suspect. > > > > I don't know what you mean by preconfiguration exactly. > > > > Do you want the host to configure the IOMMU? Why not let the > > guest do this? > > We simply freeze GPA-to-HPA mappings during runtime. Avoids having to > validate and synchronize guest-triggered changes. Fine, but this assumes guest does very specific things, right? E.g. should guest reconfigure device's BAR, you would have to change GPA to HPA mappings? > >>> > >>>> Also not yet clear to me are how the vhost-pci device and the > >>>> translations it will have to do should look like for VM2. > >>> > >>> I think we can use vhost-pci BAR + VM1 bus address as the > >>> VM2 physical address. In other words, all memory exposed to > >>> virtio-pci by VM1 through it's IOMMU is mapped into BAR of > >>> vhost-pci. > >>> > >>> Bus addresses can be validated to make sure they fit > >>> in the BAR. > >> > >> Sounds simple but may become challenging for VMs that have many of such > >> devices (in order to connect to many possibly large VMs). > > > > You don't need to be able to map all guest memory if you know > > guest won't try to allow device access to all of it. > > It's a question of how good is the bus address allocator. > > But those BARs need to allocate a guest-physical address range as large > as the other guest's RAM is, possibly even larger if that RAM is not > contiguous, and you can't put other resources into potential holes > because VM2 does not know where those holes will be. No - only the RAM that you want addressable by VM2. IOW if you wish, you actually can create a shared memory device, make it accessible to the IOMMU and place some or all data there. > > > >>> > >>> > >>> One issue to consider is that VM1 can trick VM2 into writing > >>> into bus address that isn't mapped in the IOMMU, or > >>> is mapped read-only. > >>> We probably would have to teach KVM to handle this somehow, > >>> e.g. exit to QEMU, or even just ignore. Maybe notify guest > >>> e.g. by setting a bit in the config space of the device, > >>> to avoid easy DOS. > >> > >> Well, that would be trivial for VM1 to check if there are only one or > >> two memory windows. Relying on the hypervisor to handle it may be > >> unacceptable for real-time VMs. > >> > >> Jan > > > > Why? real-time != fast. I doubt you can avoid vm exits completely. > > We can, one property of Jailhouse (on x86, ARM is waiting for GICv4). > > Real-time == deterministic. And if you have such vm exits potentially in > your code path, you have them always - for worst-case analysis. One may > argue about probability in certain scenarios, but if the triggering side > is malicious, probability may become 1. > > Jan You are doing a special hypervisor anyway, I think you could detect that setup is done, and freeze the configuration. If afterwards a VM attempts to modify mappings, you can say it's malicious and ignore it, or kill it, or whatever. > -- > Siemens AG, Corporate Technology, CT RTC ITP SES-DE > Corporate Competence Center Embedded Linux _______________________________________________ Virtualization mailing list Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx https://lists.linuxfoundation.org/mailman/listinfo/virtualization