On Thu, Sep 03, 2015 at 10:21:28AM +0200, Jan Kiszka wrote: > On 2015-09-03 10:08, Michael S. Tsirkin wrote: > > On Tue, Sep 01, 2015 at 06:28:28PM +0200, Jan Kiszka wrote: > >> On 2015-09-01 18:02, Michael S. Tsirkin wrote: > >>> On Tue, Sep 01, 2015 at 05:34:37PM +0200, Jan Kiszka wrote: > >>>> On 2015-09-01 16:34, Michael S. Tsirkin wrote: > >>>>> On Tue, Sep 01, 2015 at 04:09:44PM +0200, Jan Kiszka wrote: > >>>>>> On 2015-09-01 11:24, Michael S. Tsirkin wrote: > >>>>>>> On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan Kiszka wrote: > >>>>>>>> On 2015-09-01 10:01, Michael S. Tsirkin wrote: > >>>>>>>>> On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote: > >>>>>>>>>> Leaving all the implementation and interface details aside, this > >>>>>>>>>> discussion is first of all about two fundamentally different approaches: > >>>>>>>>>> static shared memory windows vs. dynamically remapped shared windows (a > >>>>>>>>>> third one would be copying in the hypervisor, but I suppose we all agree > >>>>>>>>>> that the whole exercise is about avoiding that). Which way do we want or > >>>>>>>>>> have to go? > >>>>>>>>>> > >>>>>>>>>> Jan > >>>>>>>>> > >>>>>>>>> Dynamic is a superset of static: you can always make it static if you > >>>>>>>>> wish. Static has the advantage of simplicity, but that's lost once you > >>>>>>>>> realize you need to invent interfaces to make it work. Since we can use > >>>>>>>>> existing IOMMU interfaces for the dynamic one, what's the disadvantage? > >>>>>>>> > >>>>>>>> Complexity. Having to emulate even more of an IOMMU in the hypervisor > >>>>>>>> (we already have to do a bit for VT-d IR in Jailhouse) and doing this > >>>>>>>> per platform (AMD IOMMU, ARM SMMU, ...) is out of scope for us. In that > >>>>>>>> sense, generic grant tables would be more appealing. > >>>>>>> > >>>>>>> That's not how we do things for KVM, PV features need to be > >>>>>>> modular and interchangeable with emulation. > >>>>>> > >>>>>> I know, and we may have to make some compromise for Jailhouse if that > >>>>>> brings us valuable standardization and broad guest support. But we will > >>>>>> surely not support an arbitrary amount of IOMMU models for that reason. > >>>>>> > >>>>>>> > >>>>>>> If you just want something that's cross-platform and easy to > >>>>>>> implement, just build a PV IOMMU. Maybe use virtio for this. > >>>>>> > >>>>>> That is likely required to keep the complexity manageable and to allow > >>>>>> static preconfiguration. > >>>>> > >>>>> Real IOMMU allow static configuration just fine. This is exactly > >>>>> what VFIO uses. > >>>> > >>>> Please specify more precisely which feature in which IOMMU you are > >>>> referring to. Also, given that you refer to VFIO, I suspect we have > >>>> different thing in mind. I'm talking about an IOMMU device model, like > >>>> the one we have in QEMU now for VT-d. That one is not at all > >>>> preconfigured by the host for VFIO. > >>> > >>> I really just mean that VFIO creates a mostly static IOMMU configuration. > >>> > >>> It's configured by the guest, not the host. > >> > >> OK, that resolves my confusion. > >> > >>> > >>> I don't see host control over configuration as being particularly important. > >> > >> We do, see below. > >> > >>> > >>> > >>>>> > >>>>>> Well, we could declare our virtio-shmem device to be an IOMMU device > >>>>>> that controls access of a remote VM to RAM of the one that owns the > >>>>>> device. In the static case, this access may at most be enabled/disabled > >>>>>> but not moved around. The static regions would have to be discoverable > >>>>>> for the VM (register read-back), and the guest's firmware will likely > >>>>>> have to declare those ranges reserved to the guest OS. > >>>>>> In the dynamic case, the guest would be able to create an alternative > >>>>>> mapping. > >>>>> > >>>>> > >>>>> I don't think we want a special device just to support the > >>>>> static case. It might be a bit less code to write, but > >>>>> eventually it should be up to the guest. > >>>>> Fundamentally, it's policy that host has no business > >>>>> dictating. > >>>> > >>>> "A bit less" is to be validated, and I doubt its just "a bit". But if > >>>> KVM and its guests will also support some PV-IOMMU that we can reuse for > >>>> our scenarios, than that is fine. KVM would not have to mandate support > >>>> for it while we would, that's all. > >>> > >>> Someone will have to do this work. > >>> > >>>>> > >>>>>> We would probably have to define a generic page table structure > >>>>>> for that. Or do you rather have some MPU-like control structure in mind, > >>>>>> more similar to the memory region descriptions vhost is already using? > >>>>> > >>>>> I don't care much. Page tables use less memory if a lot of memory needs > >>>>> to be covered. OTOH if you want to use virtio (e.g. to allow command > >>>>> batching) that likely means commands to manipulate the IOMMU, and > >>>>> maintaining it all on the host. You decide. > >>>> > >>>> I don't care very much about the dynamic case as we won't support it > >>>> anyway. However, if the configuration concept used for it is applicable > >>>> to static mode as well, then we could reuse it. But preconfiguration > >>>> will required register-based region description, I suspect. > >>> > >>> I don't know what you mean by preconfiguration exactly. > >>> > >>> Do you want the host to configure the IOMMU? Why not let the > >>> guest do this? > >> > >> We simply freeze GPA-to-HPA mappings during runtime. Avoids having to > >> validate and synchronize guest-triggered changes. > > > > Fine, but this assumes guest does very specific things, right? > > E.g. should guest reconfigure device's BAR, you would have > > to change GPA to HPA mappings? > > > > Yes, that's why we only support size exploration, not reallocation. > > > > >>>>> > >>>>>> Also not yet clear to me are how the vhost-pci device and the > >>>>>> translations it will have to do should look like for VM2. > >>>>> > >>>>> I think we can use vhost-pci BAR + VM1 bus address as the > >>>>> VM2 physical address. In other words, all memory exposed to > >>>>> virtio-pci by VM1 through it's IOMMU is mapped into BAR of > >>>>> vhost-pci. > >>>>> > >>>>> Bus addresses can be validated to make sure they fit > >>>>> in the BAR. > >>>> > >>>> Sounds simple but may become challenging for VMs that have many of such > >>>> devices (in order to connect to many possibly large VMs). > >>> > >>> You don't need to be able to map all guest memory if you know > >>> guest won't try to allow device access to all of it. > >>> It's a question of how good is the bus address allocator. > >> > >> But those BARs need to allocate a guest-physical address range as large > >> as the other guest's RAM is, possibly even larger if that RAM is not > >> contiguous, and you can't put other resources into potential holes > >> because VM2 does not know where those holes will be. > > > > No - only the RAM that you want addressable by VM2. > > That's in the hand of VM1, not VM2 or the hypervisor, in case of > reconfigurable mapping. It's indeed a non-issue in our static case. > > > > > IOW if you wish, you actually can create a shared memory device, > > make it accessible to the IOMMU and place some or all > > data there. > > > > Actually, that could also be something more sophisticated, including > virtio-net, IF that device will be able to express its DMA window > restrictions (a bit like 32-bit PCI devices being restricted to <4G > addresses or ISA devices <1M). > > Jan Actually, it's the bus restriction, not the device restriction. So if you want to use bounce buffers in the name of security or real-time requirements, you should be able to do this if virtio uses the DMA API. > -- > Siemens AG, Corporate Technology, CT RTC ITP SES-DE > Corporate Competence Center Embedded Linux _______________________________________________ Virtualization mailing list Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx https://lists.linuxfoundation.org/mailman/listinfo/virtualization
