On Wed, 2018-04-18, Allen Yu wrote: > Hi, > > I have been tracing the source of a nasty bug that happens on my > Openstack Liberty cluster recently, but in vain. > > Basically when I start a new instance, Openstack would report it as > "Paused" immediately. When I login to the compute node and check the > qemu logs, I saw KVM internal error. Suberror: 3. Any chance the failing VM is a nested VM? The failure signature you are seeing is identical to a recently fixed bug that manifested when injecting an APIC timer event to L2. https://patchwork.kernel.org/patch/9593073/ > /var/log/libvirt/qemu/instance.log > 2018-04-18 22:51:49.503+0000: starting up > LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin > QEMU_AUDIO_DRV=none /usr/bin/kvm -name instance-000000fc -S -machine > pc-i440fx-trusty,accel=kvm,usb=off -cpu > SandyBridge,+invpcid,+erms,+bmi2,+smep,+avx2,+bmi1,+fsgsbase,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+movbe,+dca,+pcid,+pdcm,+xtpr,+fma,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme > -m 245000 -realtime mlock=off -smp 46,sockets=46,cores=1,threads=1 > -uuid d6bec617-7977-4fd6-a6ad-cd9757735fdc -smbios > type=1,manufacturer=OpenStack Foundation,product=OpenStack > Nova,version=12.0.5,serial=d8eca30f-2e73-4dfb-9270-244084458637,uuid=d6bec617-7977-4fd6-a6ad-cd9757735fdc,family=Virtual > Machine -no-user-config -nodefaults -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000000fc.monitor,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc > base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard > -no-hpet -no-shutdown -boot strict=on -device > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive > file=/var/lib/nova/instances/d6bec617-7977-4fd6-a6ad-cd9757735fdc/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none > -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 > -drive file=/var/lib/nova/instances/d6bec617-7977-4fd6-a6ad-cd9757735fdc/disk.swap,if=none,id=drive-virtio-disk1,format=qcow2,cache=none > -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 > -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:42:9d:6f,bus=pci.0,addr=0x3 > -chardev file,id=charserial0,path=/var/lib/nova/instances/d6bec617-7977-4fd6-a6ad-cd9757735fdc/console.log > -device isa-serial,chardev=charserial0,id=serial0 -chardev > pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 > -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on > Domain id=6 is tainted: high-privileges > char device redirected to /dev/pts/7 (label charserial1) > KVM internal error. Suberror: 3 > extra data[0]: 800000ef > extra data[1]: 31 > RAX=0000000000000000 RBX=ffff883a8fee5fd8 RCX=00000000ffffffff > RDX=0000000000000000 > RSI=0000000000000001 RDI=ffffffff81dd9e48 RBP=ffff883a8fee5ec8 > RSP=ffff883a8fee5ec8 > R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 > R11=0000000000000000 > R12=ffffffff81cdbe60 R13=000000000000000a R14=0000000000000000 > R15=0000000000000000 > RIP=ffffffff8103cf6b RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 > ES =0000 0000000000000000 ffffffff 00c00000 > CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA] > SS =0000 0000000000000000 ffffffff 00c00000 > DS =0000 0000000000000000 ffffffff 00c00000 > FS =0000 0000000000000000 ffffffff 00c00000 > GS =0000 ffff883b20340000 ffffffff 00c00000 > LDT=0000 0000000000000000 ffffffff 00c00000 > TR =0040 ffff883b203512c0 00002087 00008b00 DPL=0 TSS64-busy > GDT= ffff883b20344000 0000007f > IDT= ffffffff81dd6000 00000fff > CR0=8005003b CR2=00000000ffffffff CR3=0000000001c05000 CR4=001406e0 > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 > DR3=0000000000000000 > DR6=00000000ffff0ff0 DR7=0000000000000400 > EFER=0000000000000d01 > Code=66 90 fb 5d c3 0f 1f 40 00 55 48 89 e5 66 66 66 66 90 fb f4 <5d> > c3 0f 1f 00 55 48 89 e5 66 66 66 66 90 f4 5d c3 0f 1f 40 00 55 48 89 > e5 66 66 66 66 90 > > Next I enabled kernel tracing according to the instructions on > https://www.linux-kvm.org/page/Tracing. I noted a lot of page faults > and IO errors. Here shows an excerpt: > qemu-system-x86-4848 [021] 561909.361044: kvm_update_master_clock: > masterclock 0 hostclock tsc offsetmatched 0 > qemu-system-x86-4848 [021] 561909.361148: kvm_fpu: load > qemu-system-x86-4848 [021] 561909.361151: kvm_entry: vcpu 0 > qemu-system-x86-4848 [021] 561909.361155: kvm_exit: > reason EPT_VIOLATION rip 0xfff0 info 184 0 > qemu-system-x86-4848 [021] 561909.361157: kvm_page_fault: > address fffffff0 error_code 184 > qemu-system-x86-4848 [021] 561909.361167: kvm_entry: vcpu 0 > qemu-system-x86-4848 [021] 561909.361168: kvm_exit: > reason EPT_VIOLATION rip 0xe05b info 184 0 > qemu-system-x86-4848 [021] 561909.361168: kvm_page_fault: > address fe05b error_code 184 > qemu-system-x86-4848 [021] 561909.361171: kvm_entry: vcpu 0 > qemu-system-x86-4848 [021] 561909.361172: kvm_exit: > reason EPT_VIOLATION rip 0xe05b info 181 0 > qemu-system-x86-4848 [021] 561909.361172: kvm_page_fault: > address f6574 error_code 181 > > The full tracing report is also available at > https://www.dropbox.com/s/hmee8sr0zcruqyh/trace-cmd.report.gz?dl=0 > > Other system info: > OS: Ubuntu 14.04.5 > Kernel: 3.13.0-123-generic > QEMU: version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.40) > CPU: 2 x Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz VT-d enabled > Motherboard: X10DRT-PT Intel C610 chipset > BIOS: American Megatrends Inc. version: 1.0c date: 04/10/2015 > RAM: 16 x 16GB Samsung M393A2G40DB0-CPB > > Any comments or hints would be greatly appreciated. Thank you very much! > > Best regards, > Allen >
