Avi Kivity wrote: > On 06/16/2009 04:42 PM, Gregory Haskins wrote: >> iosignalfd is a mechanism to register PIO/MMIO regions to trigger an >> eventfd >> signal when written to by a guest. Host userspace can register any >> arbitrary >> IO address with a corresponding eventfd and then pass the eventfd to a >> specific end-point of interest for handling. >> >> Normal IO requires a blocking round-trip since the operation may cause >> side-effects in the emulated model or may return data to the caller. >> Therefore, an IO in KVM traps from the guest to the host, causes a >> VMX/SVM >> "heavy-weight" exit back to userspace, and is ultimately serviced by >> qemu's >> device model synchronously before returning control back to the vcpu. >> >> However, there is a subclass of IO which acts purely as a trigger for >> other IO (such as to kick off an out-of-band DMA request, etc). For >> these >> patterns, the synchronous call is particularly expensive since we really >> only want to simply get our notification transmitted asychronously and >> return as quickly as possible. All the sychronous infrastructure to >> ensure >> proper data-dependencies are met in the normal IO case are just >> unecessary >> overhead for signalling. This adds additional computational load on the >> system, as well as latency to the signalling path. >> >> Therefore, we provide a mechanism for registration of an in-kernel >> trigger >> point that allows the VCPU to only require a very brief, lightweight >> exit just long enough to signal an eventfd. This also means that any >> clients compatible with the eventfd interface (which includes userspace >> and kernelspace equally well) can now register to be notified. The end >> result should be a more flexible and higher performance notification API >> for the backend KVM hypervisor and perhipheral components. >> >> To test this theory, we built a test-harness called "doorbell". This >> module has a function called "doorbell_ring()" which simply increments a >> counter for each time the doorbell is signaled. It supports signalling >> from either an eventfd, or an ioctl(). >> >> We then wired up two paths to the doorbell: One via QEMU via a >> registered >> io region and through the doorbell ioctl(). The other is direct via >> iosignalfd. >> >> You can download this test harness here: >> >> ftp://ftp.novell.com/dev/ghaskins/doorbell.tar.bz2 >> >> The measured results are as follows: >> >> qemu-mmio: 110000 iops, 9.09us rtt >> iosignalfd-mmio: 200100 iops, 5.00us rtt >> iosignalfd-pio: 367300 iops, 2.72us rtt >> >> I didn't measure qemu-pio, because I have to figure out how to >> register a >> PIO region with qemu's device model, and I got lazy. However, for >> now we >> can extrapolate based on the data from the NULLIO runs of +2.56us for >> MMIO, >> and -350ns for HC, we get: >> >> qemu-pio: 153139 iops, 6.53us rtt >> iosignalfd-hc: 412585 iops, 2.37us rtt >> >> these are just for fun, for now, until I can gather more data. >> >> Here is a graph for your convenience: >> >> http://developer.novell.com/wiki/images/7/76/Iofd-chart.png >> >> The conclusion to draw is that we save about 4us by skipping the >> userspace >> hop. >> >> >> +config KVM_MAX_IOSIGNALFD_ITEMS >> + int "Maximum IOSIGNALFD items per address" >> + depends on KVM >> + default "32" >> + ---help--- >> + This option influences the maximum number of fd's per PIO/MMIO >> + address that are allowed to register >> + >> > > Is there a per-vm limit on iosignalfds? if not, userspace can exhaust > kernel memory in that way. Yeah, its already naturally limited by the maximum number of MMIO/PIO devices we can register (today this is 6 per VM). I should have documented that fact somewhere, tho. > > We could limit the just total number of iosignafds, it's somewhat more > natural. >> diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig >> index daece36..a4b427f 100644 >> --- a/virt/kvm/Kconfig >> +++ b/virt/kvm/Kconfig >> @@ -12,3 +12,5 @@ config HAVE_KVM_EVENTFD >> >> config KVM_APIC_ARCHITECTURE >> bool >> + >> + >> > > Spurious, please drop. Ack >> +/* >> + * Design note: We create one PIO/MMIO device (iosignalfd_group) which >> + * aggregates one or more iosignalfd_items. Each item points to >> exactly one >> + * eventfd, and can be registered to trigger on any write to the group >> + * (wildcard), or to a write of a specific value. If more than one >> item is to >> + * be supported, the addr/len ranges must all be identical in the >> group. If a >> + * trigger value is to be supported on a particular item, the group >> range must >> + * be exactly the width of the trigger. >> + */ >> + >> +struct _iosignalfd_item { >> + struct list_head list; >> + struct file *file; >> + unsigned char *match; >> + struct rcu_head rcu; >> +}; >> > > Why not u64 match? Well, tbh it was primarily because it was starting to make my head hurt w.r.t. endianness ;). For instance, if someone wanted a u16 match, I would presumably have to understand the relevant endianess of the u64 so I compare the appropriate bytes against the data-register coming in from the [MM|P]IO. Using a pointer, I simply copy/memcmp the specified number of bytes and never have to worry about endianness. As a minor bonus, item->match == NULL tells me its a wildcard. If I had item->match as a u64, I'd need a different state flag for "wildcard". NBD, but thought I would point it out. > >> +static int >> +iosignalfd_is_match(struct _iosignalfd_group *group, >> + struct _iosignalfd_item *item, >> + const void *val, >> + int len) >> +{ >> + if (!item->match) >> + /* wildcard is a hit */ >> + return true; >> + >> + if (len != group->length) >> + /* mis-matched length is a miss */ >> + return false; >> > > Should check length before match (i.e. require correctly sized access). Perhaps, but my thinking is that group->length only matters for data-matching. You could conceivably have a larger window registered if you are using all wildcards. Not sure if this is really useful, but its the reason the code is that way today. Thanks Avi, -Greg
Attachment:
signature.asc
Description: OpenPGP digital signature
