On Mon, 18 Feb 2019 15:24:25 +0100 Christian Borntraeger <borntraeger@xxxxxxxxxx> wrote: > On 18.02.2019 11:00, Cornelia Huck wrote: > > [Moving this away from virtio-comments, as requested; instead cc:ing > > linux-s390, where there's a chance of s390 folks commenting. > > > > For those new to the thread, here's some background: > > > > We want to be able to map shared memory regions from the host into the > > guest for virtio devices, see > > https://lists.oasis-open.org/archives/virtio-comment/201901/msg00000.html . > > This needs to be modeled for virtio-ccw on s390 as well, see > > https://lists.oasis-open.org/archives/virtio-comment/201902/msg00002.html > > and following. > > > > The question is how to map host regions into the guest so that they > > don't get into the way of normal operation. David made a proposal with > > Still trying to understand what you are saying. > Not that deep into PCI, so lets try to get an explanation that makes it > easy for me to understand things. Would this be something like doing an > mmap in the qemu address space that is then mapped into the guest real > memory (not virtual)? (e.g. via an extra memslot). The virtio device > would then have a way to find out the address of that memory (via virtio > config and ideally not via pci bar config). It's not pci specific; the regions exist regardless whether we use the virtio-pci or the virtio-ccw transport, just the means of discovery is different. The memory is controlled by the device (the host); it is supposed to be somewhere the guest can see it and create page tables for it, but out of the way for the usual allocations (so that guests not supporting this won't trip over it). > > > > > a 1 GB memory hole which is similar to that seen on older PCs.] > > That would be a hole inside the memory or beyond the memory? For obvious > reasons a hole is not good. Considering usecases like a DAX-enabled > filesystem 1GB seem pretty small for such cases. So would that rather be > a mapping after the maximum memory size until the architecture limit. The orginal idea was guest memory shared regions memory for virtio-mem and friends Alternative layout see below; I understand that the 1 GB limit would be problematic, but I'm not sure what virtio-mem will need. David probably understands more :) > > Do we know the size of the device mappings upfront (So that we can setup > the max memory size)? > Something like > 4GB guest + 2 deviced with 1TB each. So we would set the max memory size > to 2.004TB For a non-hotplugged device, we should know. For hotplugged, this is more difficult; maybe we can fail it if a pre-alloced pool is exhausted? > > Christian > > > > On Fri, 15 Feb 2019 23:08:12 +0100 > > David Hildenbrand <david@xxxxxxxxxx> wrote: > > > >>>> On s390x, we have to define the size of the host->guest page table when > >>>> starting the guest. So we need some upper limit. > >>> > >>> That's OK; x86 also has that because they have a limited physical > >>> and virtual address size [which may or may not be correctly passed to > >>> the guest!]. > >>> > >>>> Mapping anywhere, I > >>>> really don't like. Letting the guest define the mapping, I really don't > >>>> like. > >>> > >>> Well it's OK to have a hole for it, but letting the guest choose where > >>> those mappings go in the hole is the norm for PCI (there are > >>> exceptions). > >> > >> Yes, but at least if we need a new interface for this for s390x (what I > >> assume), not allowing the guest to map is obviously easier. However if > >> there is good reason why it is needed, nothing speaks against it. We > >> just have to offer the guest enough information to do it. > > > > Speaking of host/guest: It's probably fine to do something that > > requires a hypervisor; it just needs to work within the constraints of > > the architecture. Not sure if SCLP or a diag hypercall should come into > > play. > > > >> > >>> > >>>> We can of course switch the order of mappings > >>>> > >>>> [0x000000000000000 ] > >>>> ... Memory region containing RAM > >>>> [ram_size ] > >>>> ... Memory region for memory devices (virtio-pmem, virtio-mem ...) > >>>> [maxram_size - ram_size ] > >>>> ... Memory region for e.g. special PCI/CCW devices > >>>> [ TBD] > >>>> > >>>> We can size TBD in a way that we e.g. max out the current page table > >>>> size before having to switch to more levels. This is what I meant with the alternative approach. > >>> > >>> Yes, that's fine to set some upper limit; you've just got to make sure > >>> that the hypervisor knows where it can put stuff and if the guest > >>> does PCI that it knows where it's allowed to put stuff and as long > >>> as the two don't overlap everyone is happy. > > > > Hm... is that an issue for pci? Do we need to care, as s390 uses > > special instructions anyway? Or do we want to avoid going through them, > > so that the guest can use normal read/write? > > > >>> > >>> [We should probably take this level of detail off this list - it's > >>> parsecs away from the detail of virtio] > >> > >> Yes indeed, I guess this gives Conny an idea on how it could work, so > >> now she can design an interface - and clarify all the details ;) > > > > I really need someone familiar with memory handling for that, though :) > > > >> I guess we'll need something for both, virtio-ccw and virtio-pci on s390x. > > > > My guess is that we can use the same mechanism for pci. The hard part > > is probably coming up with a memory layout... I'm also still unsure about possible zpci implications; maybe someone who is familiar with that one has an idea.
