It all started by Svante Signell on IRC, see http://bugs.debian.org/634149 . The short story: running hurd guest image (on ide drive); when changing only host kernel from 2.6.32 to (in my case) 3.0.0-rc7, the i/o (especially write) speed of the guest reduces dramatically, as is host CPU usage. I used a cd-1.iso install image from the URL provided in the bugreport, http://people.debian.org/~sthibault/hurd-i386/installer/cdimage/ -- even installing packages is very significantly slower in current kernels than it was in 2.6.32. Typical kvm_stats output on 3.0.0 host kernel: kvm statistics exits 350842433 277118 io_exits 335217689 274154 insn_emulation 276092554 259665 host_state_reload 265635092 259610 mmu_pte_write 83612845 3413 irq_injections 643860 449 irq_exits 134069 100 fpu_reload 6284 0 pf_fixed 144563 0 halt_exits 4544 0 halt_wakeup 4544 0 mmio_exits 2864 0 mmu_shadow_zapped 1675 0 mmu_cache_miss 1402 0 remote_tlb_flush 7 0 signal_exits 1 0 The same, on 2.6.32: kvm statistics exits 47692767 460212 io_exits 46815092 459058 host_state_reload 5093435 23353 fpu_reload 5093387 23344 irq_injections 475229 973 insn_emulation 852122 864 irq_exits 51672 235 mmio_exits 173630 0 pf_fixed 148051 0 halt_exits 3150 0 mmu_shadow_zapped 2650 0 mmu_cache_miss 1531 0 halt_wakeup 60 0 remote_tlb_flush 12 0 signal_exits 1 0 (this is while the installation process is running). typical output from trace-cmd, for 3.0.0: kvm-18166 [001] 47471.398399: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398399: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398400: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398400: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398401: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398402: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398403: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398403: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398403: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398405: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398405: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398406: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398406: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398407: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398408: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398409: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398409: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398410: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398410: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398411: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398412: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398413: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398413: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398413: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398415: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398415: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398416: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398416: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398417: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398418: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398419: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398419: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398420: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398420: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398421: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398422: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398423: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398423: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398423: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398425: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398425: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398426: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398427: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398427: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398428: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398429: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398429: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398430: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398430: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398431: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398432: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398433: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398433: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398433: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398435: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398435: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398436: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398437: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398437: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398438: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398439: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398440: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398440: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398440: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398442: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398442: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398443: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398444: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398444: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398445: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398446: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398447: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398447: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398447: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398449: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398449: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398450: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398450: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398451: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398452: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398453: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398453: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398454: kvm_pio: pio_write at 0x1f0 size 2 count 1 kvm-18166 [001] 47471.398454: kvm_userspace_exit: reason KVM_EXIT_IO (2) kvm-18166 [001] 47471.398456: kvm_entry: vcpu 0 kvm-18166 [001] 47471.398456: kvm_exit: reason EXIT_IOIO rip 0x160750 info 1f0012c 160753 kvm-18166 [001] 47471.398457: kvm_emulate_insn: 80000000:160750: f3 66 6f kvm-18166 [001] 47471.398458: kvm_pio: pio_write at 0x1f0 size 2 count 1 The same, on 2.6.32: kvm-2937 [001] 310.808825: kvm_entry: vcpu 0 kvm-2937 [001] 310.808828: kvm_exit: reason UNKNOWN rip 0x107c08 kvm-2937 [001] 310.808829: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808829: kvm_entry: vcpu 0 kvm-2937 [001] 310.808830: kvm_exit: reason UNKNOWN rip 0x107c11 kvm-2937 [001] 310.808830: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808831: kvm_entry: vcpu 0 kvm-2937 [001] 310.808832: kvm_exit: reason UNKNOWN rip 0x107a9d kvm-2937 [001] 310.808832: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808832: kvm_entry: vcpu 0 kvm-2937 [001] 310.808833: kvm_exit: reason UNKNOWN rip 0x107aa6 kvm-2937 [001] 310.808833: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808833: kvm_entry: vcpu 0 kvm-2937 [001] 310.808846: kvm_exit: reason UNKNOWN rip 0x107c08 kvm-2937 [001] 310.808846: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808847: kvm_entry: vcpu 0 kvm-2937 [001] 310.808847: kvm_exit: reason UNKNOWN rip 0x107c11 kvm-2937 [001] 310.808847: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808848: kvm_entry: vcpu 0 kvm-2937 [001] 310.808849: kvm_exit: reason UNKNOWN rip 0x107a9d kvm-2937 [001] 310.808849: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808849: kvm_entry: vcpu 0 kvm-2937 [001] 310.808850: kvm_exit: reason UNKNOWN rip 0x107aa6 kvm-2937 [001] 310.808850: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808850: kvm_entry: vcpu 0 kvm-2937 [001] 310.808851: kvm_exit: reason UNKNOWN rip 0x107c08 kvm-2937 [001] 310.808851: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808852: kvm_entry: vcpu 0 kvm-2937 [001] 310.808852: kvm_exit: reason UNKNOWN rip 0x107c11 kvm-2937 [001] 310.808853: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808853: kvm_entry: vcpu 0 kvm-2937 [001] 310.808854: kvm_exit: reason UNKNOWN rip 0x107a9d kvm-2937 [001] 310.808854: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808854: kvm_entry: vcpu 0 kvm-2937 [001] 310.808855: kvm_exit: reason UNKNOWN rip 0x107aa6 kvm-2937 [001] 310.808855: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808856: kvm_entry: vcpu 0 kvm-2937 [001] 310.808856: kvm_exit: reason UNKNOWN rip 0x107c08 kvm-2937 [001] 310.808856: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808857: kvm_entry: vcpu 0 kvm-2937 [001] 310.808858: kvm_exit: reason UNKNOWN rip 0x107c11 kvm-2937 [001] 310.808858: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808858: kvm_entry: vcpu 0 kvm-2937 [001] 310.808859: kvm_exit: reason UNKNOWN rip 0x107a9d kvm-2937 [001] 310.808859: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808860: kvm_entry: vcpu 0 kvm-2937 [001] 310.808860: kvm_exit: reason UNKNOWN rip 0x107aa6 kvm-2937 [001] 310.808861: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808861: kvm_entry: vcpu 0 kvm-2937 [001] 310.808866: kvm_exit: reason UNKNOWN rip 0x107c08 kvm-2937 [001] 310.808866: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808866: kvm_entry: vcpu 0 kvm-2937 [001] 310.808867: kvm_exit: reason UNKNOWN rip 0x107c11 kvm-2937 [001] 310.808868: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808869: kvm_entry: vcpu 0 kvm-2937 [001] 310.808870: kvm_exit: reason UNKNOWN rip 0x107a9d kvm-2937 [001] 310.808870: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808870: kvm_entry: vcpu 0 kvm-2937 [001] 310.808871: kvm_exit: reason UNKNOWN rip 0x107aa6 kvm-2937 [001] 310.808871: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808871: kvm_entry: vcpu 0 kvm-2937 [001] 310.808901: kvm_exit: reason UNKNOWN rip 0x107c08 kvm-2937 [001] 310.808901: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808902: kvm_entry: vcpu 0 kvm-2937 [001] 310.808903: kvm_exit: reason UNKNOWN rip 0x107c11 kvm-2937 [001] 310.808903: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808903: kvm_entry: vcpu 0 kvm-2937 [001] 310.808904: kvm_exit: reason UNKNOWN rip 0x107a9d kvm-2937 [001] 310.808904: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808905: kvm_entry: vcpu 0 kvm-2937 [001] 310.808905: kvm_exit: reason UNKNOWN rip 0x107aa6 kvm-2937 [001] 310.808905: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808906: kvm_entry: vcpu 0 kvm-2937 [001] 310.808914: kvm_exit: reason UNKNOWN rip 0x107c08 kvm-2937 [001] 310.808914: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808915: kvm_entry: vcpu 0 kvm-2937 [001] 310.808916: kvm_exit: reason UNKNOWN rip 0x107c11 kvm-2937 [001] 310.808916: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808916: kvm_entry: vcpu 0 kvm-2937 [001] 310.808917: kvm_exit: reason UNKNOWN rip 0x107a9d kvm-2937 [001] 310.808917: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808917: kvm_entry: vcpu 0 kvm-2937 [001] 310.808918: kvm_exit: reason UNKNOWN rip 0x107aa6 kvm-2937 [001] 310.808918: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808919: kvm_entry: vcpu 0 kvm-2937 [001] 310.808919: kvm_exit: reason UNKNOWN rip 0x107c08 kvm-2937 [001] 310.808920: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808920: kvm_entry: vcpu 0 kvm-2937 [001] 310.808921: kvm_exit: reason UNKNOWN rip 0x107c11 kvm-2937 [001] 310.808921: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808921: kvm_entry: vcpu 0 kvm-2937 [001] 310.808922: kvm_exit: reason UNKNOWN rip 0x107a9d kvm-2937 [001] 310.808922: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808922: kvm_entry: vcpu 0 kvm-2937 [001] 310.808923: kvm_exit: reason UNKNOWN rip 0x107aa6 kvm-2937 [001] 310.808923: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808924: kvm_entry: vcpu 0 kvm-2937 [001] 310.808924: kvm_exit: reason UNKNOWN rip 0x107c08 kvm-2937 [001] 310.808925: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808925: kvm_entry: vcpu 0 kvm-2937 [001] 310.808926: kvm_exit: reason UNKNOWN rip 0x107c11 kvm-2937 [001] 310.808926: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808926: kvm_entry: vcpu 0 kvm-2937 [001] 310.808927: kvm_exit: reason UNKNOWN rip 0x107a9d kvm-2937 [001] 310.808927: kvm_pio: pio_write at 0x21 size 1 count 1 kvm-2937 [001] 310.808927: kvm_entry: vcpu 0 kvm-2937 [001] 310.808928: kvm_exit: reason UNKNOWN rip 0x107aa6 kvm-2937 [001] 310.808928: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808929: kvm_entry: vcpu 0 kvm-2937 [001] 310.808938: kvm_exit: reason UNKNOWN rip 0x150b17 kvm-2937 [001] 310.808939: kvm_pio: pio_write at 0xa1 size 1 count 1 kvm-2937 [001] 310.808939: kvm_entry: vcpu 0 kvm-2937 [001] 310.808940: kvm_exit: reason UNKNOWN rip 0x1633cc kvm-2937 [001] 310.808940: kvm_pio: pio_write at 0x1f6 size 1 count 1 kvm-2937 [001] 310.808943: kvm_entry: vcpu 0 Gleb, Avi -- what we're missing here? It does not look like the problem we have with FreeBSD, this is apparently something different... Thank you! /mjt -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
