Hi Don. On Mon, Jan 23, 2017 at 4:35 PM, Don Bowman <db@xxxxxxxxxxxx> wrote: > The below C program, run on my host, uses ~0% of the host CPU when run > with 100000 (e.g. wake up every 100us). > > When run under kvm, this uses ~100% of the processor. The reason is there > is a vmexit for every timer wakeup. This is probably halt-polling. When a vCPU halts, KVM will sometimes spin until a virtual interrupt arrives to wake up the vCPU. This gives better wake-up latency, but at the cost of CPU usage. The upside is halt-polling only kicks in if the host CPU is otherwise idle. So it doesn't steal runtime from other threads. This presentation has more gritty details on the problem if you are interested: http://events.linuxfoundation.org/sites/events/files/slides/Message%20Passing%20Workloads%20in%20KVM%20(SLIDES).pdf > > Now, my host processor is E5-2699 v3, and this includes apicv > functionality. My host kernel is 4.9 (as is the guest). qemu is 2.5. > (ubuntu 16.04). > > My understanding from the apicv feature was this timer could be entirely > done inside the guest on this generation (haswell) of processor. Both arming the timer and delivering the interrupt will cause a VM-exit in KVM. APICv does support posting interrupts without taking a VM-exit, but it requires doing it from another CPU. In your case, the vCPU has probably already VM-exited because it is idle (halted) while waiting for the timer. > > In my application I'm finding that nearly all of my CPU(s) are going to > servicing vmexits from various threads that are waking up. > > a) is there an efficient way to sleep/wakeup in a kvm guest? > b) is this what i should expect? > c) is there any work around or features i should be looking for? You can disable halt-polling by setting the maximum poll time to 0 via the kvm module parameter halt_poll_ns. i.e. modprobe kvm halt_poll_ns=0 > > Does anyone have any suggestions here? > > #include <assert.h> > #include <unistd.h> > #include <stdio.h> > #include <stdlib.h> > #include <sys/timerfd.h> > > int > main(int argc, char **argv) > { > int nr; > long long ns; > char buf[8]; > struct itimerspec o, n; > if (argc < 2) > { > fprintf(stderr, "Usage: %s ns-to-wake\n", argv[0]); > exit(1); > } > ns = atoll(argv[1]); > printf("Wake up every %llu ns (%6.2fs)\n", ns, (1.0 * ns) / (1000 > * 1000 * 1000)); > int fd = timerfd_create(CLOCK_MONOTONIC, 0); > assert(fd > 0); > n.it_interval.tv_sec = 0; > n.it_interval.tv_nsec = 1; > n.it_value.tv_sec = (ns / (1000 * 1000 * 1000)); > n.it_value.tv_nsec = ns % (1000 * 1000 * 1000); > > do > { > assert(timerfd_settime(fd, 0, &n, &o) >= 0); > nr = read(fd, buf, sizeof(buf)); > } > while (nr >= 0); > }
