Avi Kivity wrote:
Cam Macdonell wrote:
I think there is value for static memory sharing. It can be used for
fast, simple synchronization and communication between guests (and the
host) that use need to share data that needs to be updated frequently
(such as a simple cache or notification system). It may not be a
common task, but I think static sharing has its place and that's what
this device is for at this point.
It would be good to detail a use case for reference.
I'll try my best...
We are using the (static) shared memory region for fast, interprocess
communications (IPC). Of course, shared-memory IPC is an old idea, and
the IPC here is actually between VMs (i.e., ivshmem), not processes
inside a single VM. But, fast IPC is useful for shared caches, OS
bypass (guest-to-guest, and host-to-guest), and low-latency IPC use-cases.
For example, one use of ivshmem is as as a file cache between VMs. Note
that, unlike stream-oriented IPC, this file cache can be shared between,
say, four VMs simultaneously. In using VMs as sandboxes for distributed
computing (condor, cloud, etc.), if two (or more) VMs are co-located on
the same server, they can effectively share a single, unified cache.
Any VM can bring in the data, and other VMs can use it. Otherwise, two
VMs might transfer (over the WAN, in the worst case, as in a cloud) and
buffer cache the same file in multiple VMs. In some ways, the
configuration would look like an in-memory cluster file system, but
instead of shared disks, we have shared memory.
Alternative forms of file sharing between VMs (e.g., via SAMBA or NFS)
are possible, but also results in multiple cached copies of the same
file data on the same physical server. Furthermore, ivshmem has the
(usual, planned) latency (e.g., for file metadata stats) and bandwidth
advantages between most forms of stream-oriented IPC for file sharing
protocols.
Other (related) use cases include bulk-data movement between the host
and guest VMs, due to the OS bypass properties of the ivshmem. Since
static shared memory shares a file (or memory object) on the host,
host-guest sharing is simpler than with dynamic shared memory.
We acknowledge that work has to be done with thread/process scheduling
to truly gain low IPC latency; that is to come, possibly with
PCI interrupts. And, as the VMware experience shows (see below), VM
migration *is* affected by ivshmem, but we think a good (but
non-trivial) attach-to-ivshmem and detach-from-ivshmem protocol (in the
future) can mostly address that issue.
As an aside, VMware ditched shared memory as part of their VMCI
interface. We emailed with some of their people who suggested to use
sockets since shared memory "de-virtualizes" the VM (i.e. it breaks
migration). But on their forums there were users that used shared
memory for their work and were disappointed to see it go. One person I
emailed with used shared memory for simulations running across VMs.
Using shared memory freed him from having to come up with a protocol to
exchange updates and having a central VM responsible for receiving and
broadcasting updates. When he did try to use a socket-based approach,
the performance dropped substantially due to the communication overhead.
Then you need a side channel to communicate the information to the
guest.
Couldn't one of the registers in BAR0 be used to store the actual
(non-power-of-two) size?
The PCI config space (where the BARs reside) is a good place for it.
Registers 0x40+ are device specific IIRC.
Ok.
Cam
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
