I'm running kernel-3.14.4-200.fc20.x86_64 qemu-1.6.2-5.fc20.x86_64 xen-4.4.0-4.fc21 In process of trying to get a Xen hypervisor running inside a KVM guest I found that there's a problem with x2apic. NB I do *not* use nested-VMX here, just trying to get plain Xen paravirt working before trying todo nested HVM. Any time I enable the 'x2apic' CPU flag for the KVM guest, the Xen hypervisor running inside the guest will fail to boot: The QEMU/KVM -cpu arg is -cpu core2duo,+erms,+smep,+fsgsbase,+lahf_lm,+rdtscp,+rdrand,+f16c,+avx,+osxsave,+xsave,+aes,+tsc-deadline,+popcnt,+x2apic,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds The Xen logs indicate it isn't liking the x2apic feature and is disabling it, but then it obviously fails to setup the non-x2apic codepath it is following - even though the non-x2apic codepath works fine if you don't have +x2apic set for the KVM guest. (XEN) Not enabling x2APIC: depends on iommu_supports_eim. (XEN) XSM Framework v1.0.0 initialized (XEN) Flask: Initializing. (XEN) AVC INITIALIZED (XEN) Flask: Starting in permissive mode. (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Detected 2693.939 MHz processor. (XEN) Initing memory sharing. (XEN) traps.c:3071: GPF (0000): ffff82d0801b83c7 -> ffff82d08023386b (XEN) mce_intel.c:717: MCA Capability: BCAST 1 SER 1 CMCI 0 firstbank 1 extended MCE MSR 0 (XEN) Intel machine check reporting enabled (XEN) I/O virtualisation disabled (XEN) Getting VERSION: 1050014 (XEN) Getting VERSION: 1050014 (XEN) Enabled directed EOI with ioapic_ack_old on! (XEN) Getting ID: 0 (XEN) Getting LVT0: 8700 (XEN) Getting LVT1: 8400 (XEN) Suppress EOI broadcast on CPU#0 (XEN) enabled ExtINT on CPU#0 (XEN) ENABLING IO-APIC IRQs (XEN) -> Using old ACK method (XEN) init IO_APIC IRQs (XEN) IO-APIC (apicid-pin) 0-0, 0-16, 0-17, 0-18, 0-19, 0-20, 0-21, 0-22, 0-23 not connected. (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 (XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC (XEN) ...trying to set up timer (IRQ0) through the 8259A ... failed. (XEN) ...trying to set up timer as Virtual Wire IRQ... failed. (XEN) ...trying to set up timer as ExtINT IRQ... failed :(. (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) IO-APIC + timer doesn't work! Boot with apic_verbosity=debug and send a report. Then try booting with the 'noapic' option (XEN) **************************************** Will attach the full non-trimmed Xen log to this mail, along with a log showing successful boot when 'x2apic' isn't given to KVM. I'm unclear if this is a Xen bug or KVM bug or QEMU bug, or a combination of them.... Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Xen 4.4.0-4.fc21 (XEN) Xen version 4.4.0 (mockbuild@[unknown]) (gcc (GCC) 4.9.0 20140506 (Red Hat 4.9.0-3)) debug=n Mon May 12 18:38:23 UTC 2014 (XEN) Latest ChangeSet: (XEN) Bootloader: GRUB 2.00 (XEN) Command line: placeholder loglvl=all guest_loglvl=all com1=115200,8n1 console=com1,vga apic_verbosity=debug (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) Disc information: (XEN) Found 1 MBR signatures (XEN) Found 1 EDD information structures (XEN) Xen-e820 RAM map: (XEN) 0000000000000000 - 000000000009fc00 (usable) (XEN) 000000000009fc00 - 00000000000a0000 (reserved) (XEN) 00000000000f0000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 000000005dbfe000 (usable) (XEN) 000000005dbfe000 - 000000005dc00000 (reserved) (XEN) 00000000feffc000 - 00000000ff000000 (reserved) (XEN) 00000000fffc0000 - 0000000100000000 (reserved) (XEN) System RAM: 1499MB (1535604kB) (XEN) ACPI: RSDP 000F1690, 0014 (r0 BOCHS ) (XEN) ACPI: RSDT 5DBFE4A0, 0030 (r1 BOCHS BXPCRSDT 1 BXPC 1) (XEN) ACPI: FACP 5DBFFF80, 0074 (r1 BOCHS BXPCFACP 1 BXPC 1) (XEN) ACPI: DSDT 5DBFE4D0, 1137 (r1 BXPC BXDSDT 1 INTL 20140114) (XEN) ACPI: FACS 5DBFFF40, 0040 (XEN) ACPI: SSDT 5DBFF700, 0838 (r1 BOCHS BXPCSSDT 1 BXPC 1) (XEN) ACPI: APIC 5DBFF610, 0078 (r1 BOCHS BXPCAPIC 1 BXPC 1) (XEN) No NUMA configuration found (XEN) Faking a node at 0000000000000000-000000005dbfe000 (XEN) Domain heap initialised (XEN) found SMP MP-table at 000f17f0 (XEN) DMI 2.4 present. (XEN) APIC boot state is 'xapic' (XEN) Using APIC driver default (XEN) ACPI: PM-Timer IO Port: 0xb008 (XEN) ACPI: SLEEP INFO: pm1x_cnt[b004,0], pm1x_evt[b000,0] (XEN) ACPI: wakeup_vec[5dbfff4c], vec_size[20] (XEN) ACPI: Local APIC address 0xfee00000 (XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) (XEN) Processor #0 6:15 APIC version 20 (XEN) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) (XEN) IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) (XEN) ACPI: IRQ0 used by override. (XEN) ACPI: IRQ2 used by override. (XEN) ACPI: IRQ5 used by override. (XEN) ACPI: IRQ9 used by override. (XEN) ACPI: IRQ10 used by override. (XEN) ACPI: IRQ11 used by override. (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) ERST table was not found (XEN) Using ACPI (MADT) for SMP configuration information (XEN) SMP: Allowing 1 CPUs (0 hotplug CPUs) (XEN) mapped APIC to ffff82cfffdfb000 (fee00000) (XEN) mapped IOAPIC to ffff82cfffdfa000 (fec00000) (XEN) IRQ limits: 24 GSI, 184 MSI/MSI-X (XEN) Not enabling x2APIC: depends on iommu_supports_eim. (XEN) XSM Framework v1.0.0 initialized (XEN) Flask: Initializing. (XEN) AVC INITIALIZED (XEN) Flask: Starting in permissive mode. (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Detected 2693.939 MHz processor. (XEN) Initing memory sharing. (XEN) traps.c:3071: GPF (0000): ffff82d0801b83c7 -> ffff82d08023386b (XEN) mce_intel.c:717: MCA Capability: BCAST 1 SER 1 CMCI 0 firstbank 1 extended MCE MSR 0 (XEN) Intel machine check reporting enabled (XEN) I/O virtualisation disabled (XEN) Getting VERSION: 1050014 (XEN) Getting VERSION: 1050014 (XEN) Enabled directed EOI with ioapic_ack_old on! (XEN) Getting ID: 0 (XEN) Getting LVT0: 8700 (XEN) Getting LVT1: 8400 (XEN) Suppress EOI broadcast on CPU#0 (XEN) enabled ExtINT on CPU#0 (XEN) ENABLING IO-APIC IRQs (XEN) -> Using old ACK method (XEN) init IO_APIC IRQs (XEN) IO-APIC (apicid-pin) 0-0, 0-16, 0-17, 0-18, 0-19, 0-20, 0-21, 0-22, 0-23 not connected. (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 (XEN) ..MP-BIOS bug: 8254 timer not connected to IO-APIC (XEN) ...trying to set up timer (IRQ0) through the 8259A ... failed. (XEN) ...trying to set up timer as Virtual Wire IRQ... failed. (XEN) ...trying to set up timer as ExtINT IRQ... failed :(. (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) IO-APIC + timer doesn't work! Boot with apic_verbosity=debug and send a report. Then try booting with the 'noapic' option (XEN) **************************************** (XEN) (XEN) Reboot in five seconds...
Xen 4.4.0-4.fc21 (XEN) Xen version 4.4.0 (mockbuild@[unknown]) (gcc (GCC) 4.9.0 20140506 (Red Hat 4.9.0-3)) debug=n Mon May 12 18:38:23 UTC 2014 (XEN) Latest ChangeSet: (XEN) Bootloader: GRUB 2.00 (XEN) Command line: placeholder loglvl=all guest_loglvl=all com1=115200,8n1 console=com1,vga apic_verbosity=debug (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) Disc information: (XEN) Found 1 MBR signatures (XEN) Found 1 EDD information structures (XEN) Xen-e820 RAM map: (XEN) 0000000000000000 - 000000000009fc00 (usable) (XEN) 000000000009fc00 - 00000000000a0000 (reserved) (XEN) 00000000000f0000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 000000005dbfe000 (usable) (XEN) 000000005dbfe000 - 000000005dc00000 (reserved) (XEN) 00000000feffc000 - 00000000ff000000 (reserved) (XEN) 00000000fffc0000 - 0000000100000000 (reserved) (XEN) System RAM: 1499MB (1535604kB) (XEN) ACPI: RSDP 000F1690, 0014 (r0 BOCHS ) (XEN) ACPI: RSDT 5DBFE4A0, 0030 (r1 BOCHS BXPCRSDT 1 BXPC 1) (XEN) ACPI: FACP 5DBFFF80, 0074 (r1 BOCHS BXPCFACP 1 BXPC 1) (XEN) ACPI: DSDT 5DBFE4D0, 1137 (r1 BXPC BXDSDT 1 INTL 20140114) (XEN) ACPI: FACS 5DBFFF40, 0040 (XEN) ACPI: SSDT 5DBFF700, 0838 (r1 BOCHS BXPCSSDT 1 BXPC 1) (XEN) ACPI: APIC 5DBFF610, 0078 (r1 BOCHS BXPCAPIC 1 BXPC 1) (XEN) No NUMA configuration found (XEN) Faking a node at 0000000000000000-000000005dbfe000 (XEN) Domain heap initialised (XEN) found SMP MP-table at 000f17f0 (XEN) DMI 2.4 present. (XEN) APIC boot state is 'xapic' (XEN) Using APIC driver default (XEN) ACPI: PM-Timer IO Port: 0xb008 (XEN) ACPI: SLEEP INFO: pm1x_cnt[b004,0], pm1x_evt[b000,0] (XEN) ACPI: wakeup_vec[5dbfff4c], vec_size[20] (XEN) ACPI: Local APIC address 0xfee00000 (XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) (XEN) Processor #0 6:15 APIC version 20 (XEN) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) (XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) (XEN) IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) (XEN) ACPI: IRQ0 used by override. (XEN) ACPI: IRQ2 used by override. (XEN) ACPI: IRQ5 used by override. (XEN) ACPI: IRQ9 used by override. (XEN) ACPI: IRQ10 used by override. (XEN) ACPI: IRQ11 used by override. (XEN) Enabling APIC mode: Flat. Using 1 I/O APICs (XEN) ERST table was not found (XEN) Using ACPI (MADT) for SMP configuration information (XEN) SMP: Allowing 1 CPUs (0 hotplug CPUs) (XEN) mapped APIC to ffff82cfffdfb000 (fee00000) (XEN) mapped IOAPIC to ffff82cfffdfa000 (fec00000) (XEN) IRQ limits: 24 GSI, 184 MSI/MSI-X (XEN) XSM Framework v1.0.0 initialized (XEN) Flask: Initializing. (XEN) AVC INITIALIZED (XEN) Flask: Starting in permissive mode. (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Detected 2694.602 MHz processor. (XEN) Initing memory sharing. (XEN) traps.c:3071: GPF (0000): ffff82d0801b83c7 -> ffff82d08023386b (XEN) mce_intel.c:717: MCA Capability: BCAST 1 SER 1 CMCI 0 firstbank 1 extended MCE MSR 0 (XEN) Intel machine check reporting enabled (XEN) I/O virtualisation disabled (XEN) Getting VERSION: 50014 (XEN) Getting VERSION: 50014 (XEN) Getting ID: 0 (XEN) Getting LVT0: 8700 (XEN) Getting LVT1: 8400 (XEN) enabled ExtINT on CPU#0 (XEN) ENABLING IO-APIC IRQs (XEN) -> Using new ACK method (XEN) init IO_APIC IRQs (XEN) IO-APIC (apicid-pin) 0-0, 0-16, 0-17, 0-18, 0-19, 0-20, 0-21, 0-22, 0-23 not connected. (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1 (XEN) number of MP IRQ sources: 15. (XEN) number of IO-APIC #0 registers: 24. (XEN) testing the IO APIC....................... (XEN) IO APIC #0...... (XEN) .... register #00: 00000000 (XEN) ....... : physical APIC id: 00 (XEN) ....... : Delivery Type: 0 (XEN) ....... : LTS : 0 (XEN) .... register #01: 00170011 (XEN) ....... : max redirection entries: 0017 (XEN) ....... : PRQ implemented: 0 (XEN) ....... : IO APIC version: 0011 (XEN) .... register #02: 00000000 (XEN) ....... : arbitration: 00 (XEN) .... IRQ redirection table: (XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect: (XEN) 00 000 00 1 0 0 0 0 0 0 00 (XEN) 01 001 01 0 0 0 0 0 1 1 28 (XEN) 02 001 01 0 0 0 0 0 1 1 F0 (XEN) 03 001 01 0 0 0 0 0 1 1 30 (XEN) 04 001 01 0 0 0 0 0 1 1 F1 (XEN) 05 001 01 1 1 0 0 0 1 1 38 (XEN) 06 001 01 0 0 0 0 0 1 1 40 (XEN) 07 001 01 0 0 0 0 0 1 1 48 (XEN) 08 001 01 0 0 0 0 0 1 1 50 (XEN) 09 001 01 1 1 0 0 0 1 1 58 (XEN) 0a 001 01 1 1 0 0 0 1 1 60 (XEN) 0b 001 01 1 1 0 0 0 1 1 68 (XEN) 0c 001 01 0 0 0 0 0 1 1 70 (XEN) 0d 001 01 0 0 0 0 0 1 1 78 (XEN) 0e 001 01 0 0 0 0 0 1 1 88 (XEN) 0f 001 01 0 0 0 0 0 1 1 90 (XEN) 10 000 00 1 0 0 0 0 0 0 00 (XEN) 11 000 00 1 0 0 0 0 0 0 00 (XEN) 12 000 00 1 0 0 0 0 0 0 00 (XEN) 13 000 00 1 0 0 0 0 0 0 00 (XEN) 14 000 00 1 0 0 0 0 0 0 00 (XEN) 15 000 00 1 0 0 0 0 0 0 00 (XEN) 16 000 00 1 0 0 0 0 0 0 00 (XEN) 17 000 00 1 0 0 0 0 0 0 00 (XEN) Using vector-based indexing (XEN) IRQ to pin mappings: (XEN) IRQ240 -> 0:2 (XEN) IRQ40 -> 0:1 (XEN) IRQ48 -> 0:3 (XEN) IRQ241 -> 0:4 (XEN) IRQ56 -> 0:5 (XEN) IRQ64 -> 0:6 (XEN) IRQ72 -> 0:7 (XEN) IRQ80 -> 0:8 (XEN) IRQ88 -> 0:9 (XEN) IRQ96 -> 0:10 (XEN) IRQ104 -> 0:11 (XEN) IRQ112 -> 0:12 (XEN) IRQ120 -> 0:13 (XEN) IRQ136 -> 0:14 (XEN) IRQ144 -> 0:15 (XEN) .................................... done. (XEN) Using local APIC timer interrupts. (XEN) calibrating APIC timer ... (XEN) ..... CPU clock speed is 2693.9298 MHz. (XEN) ..... host bus clock speed is 999.9993 MHz. (XEN) ..... bus_scale = 0x40000 (XEN) TSC deadline timer enabled (XEN) Platform timer is 3.579MHz ACPI PM Timer (XEN) Allocated console ring of 16 KiB. (XEN) VMX: CPU0 has insufficient VMExit Control (000f6fff; requires 00008200) (XEN) VMX: failed to initialise. (XEN) Brought up 1 CPUs (XEN) CPUIDLE: disabled due to no HPET. Force enable with 'cpuidle'. (XEN) ACPI sleep modes: S3 (XEN) mcheck_poll: Machine check polling timer started. (XEN) *** LOADING DOMAIN 0 *** (XEN) Xen kernel: 64-bit, lsb, compat32 (XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x340a000 (XEN) PHYSICAL MEMORY ARRANGEMENT: (XEN) Dom0 alloc.: 0000000054000000->0000000058000000 (335359 pages to be allocated) (XEN) Init. ramdisk: 000000005c9a8000->000000005d5ff01f (XEN) VIRTUAL MEMORY ARRANGEMENT: (XEN) Loaded kernel: ffffffff81000000->ffffffff8340a000 (XEN) Init. ramdisk: ffffffff8340a000->ffffffff8406101f (XEN) Phys-Mach map: ffffffff84062000->ffffffff843172b8 (XEN) Start info: ffffffff84318000->ffffffff843184b4 (XEN) Page tables: ffffffff84319000->ffffffff8433e000 (XEN) Boot stack: ffffffff8433e000->ffffffff8433f000 (XEN) TOTAL: ffffffff80000000->ffffffff84400000 (XEN) ENTRY ADDRESS: ffffffff8218a1f0 (XEN) Dom0 has maximum 1 VCPUs (XEN) Scrubbing Free RAM: done. (XEN) Initial low memory virq threshold set at 0x4000 pages. (XEN) Std. Loglevel: All (XEN) Guest Loglevel: All (XEN) Xen is relinquishing VGA console. (XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen) (XEN) Freed 292kB init memory. mapping kernel into physical memory about to get started... (XEN) IOAPIC[0]: Set PCI routing entry (0-9 -> 0x58 -> IRQ 9 Mode:1 Active:0) (XEN) PCI add device 0000:00:00.0 (XEN) PCI add device 0000:00:01.0 (XEN) PCI add device 0000:00:01.1 (XEN) PCI add device 0000:00:01.2 (XEN) PCI add device 0000:00:01.3 (XEN) PCI add device 0000:00:02.0 (XEN) PCI add device 0000:00:03.0 (XEN) PCI add device 0000:00:04.0 (XEN) IOAPIC[0]: Set PCI routing entry (0-8 -> 0x50 -> IRQ 8 Mode:0 Active:0) (XEN) IOAPIC[0]: Set PCI routing entry (0-1 -> 0x28 -> IRQ 1 Mode:0 Active:0) (XEN) IOAPIC[0]: Set PCI routing entry (0-12 -> 0x70 -> IRQ 12 Mode:0 Active:0) (XEN) IOAPIC[0]: Set PCI routing entry (0-6 -> 0x40 -> IRQ 6 Mode:0 Active:0) (XEN) IOAPIC[0]: Set PCI routing entry (0-7 -> 0x48 -> IRQ 7 Mode:0 Active:0) (XEN) IOAPIC[0]: Set PCI routing entry (0-3 -> 0x30 -> IRQ 3 Mode:0 Active:0) (XEN) IOAPIC[0]: Set PCI routing entry (0-11 -> 0x68 -> IRQ 11 Mode:1 Active:0) (XEN) IOAPIC[0]: Set PCI routing entry (0-10 -> 0x60 -> IRQ 10 Mode:1 Active:0) (XEN) traps.c:3071: GPF (0000): ffff82d08019fd19 -> ffff82d0802336f8 (XEN) traps.c:2514:d0 Domain attempted WRMSR 0000000000000079 from 0x0000000000000000 to 0xffffc9000055a030.
