Seeing an.. interesting thing here. Kvm crashes during migrate,
instantly. Freshly-built 0.12.4, with a simple idle guest
(linux 2.6.32).
Starting TCP receiver on the same host, so using exactly the same
kvm binary. And it drops into shell with large diagnostics output
from glibc:
(qemu) migrate tcp:localhost:3210
*** glibc detected *** /usr/bin/kvm: free(): invalid next size (fast): 0x084c7fd0 ***
======= Backtrace: =========
/lib/libc.so.6[0xf7ae1905]
/lib/libc.so.6[0xf7ae31a3]
/lib/libc.so.6(cfree+0x6d)[0xf7ae622d]
/usr/bin/kvm[0x806ed6f]
/usr/bin/kvm[0x806ee53]
/usr/bin/kvm[0x8050fb6]
/usr/bin/kvm[0x805114b]
/usr/bin/kvm[0x810e090]
/usr/bin/kvm[0x8105ee9]
/usr/bin/kvm[0x8106cce]
/usr/bin/kvm[0x8051d28]
/usr/bin/kvm[0x806d3c4]
/usr/bin/kvm[0x8054bbd]
/lib/libc.so.6(__libc_start_main+0xe5)[0xf7a8cb55]
/usr/bin/kvm[0x804e711]
======= Memory map: ========
08048000-08244000 r-xp 00000000 08:01 77392 /usr/bin/kvm
08244000-08256000 rw-p 001fc000 08:01 77392 /usr/bin/kvm
08256000-08506000 rw-p 00000000 00:00 0
08506000-08516000 rw-p 00000000 00:00 0
08516000-08517000 rw-p 00000000 00:00 0
08517000-0851f000 rw-p 00000000 00:00 0
0851f000-085f9000 rw-p 00000000 00:00 0
e8f4a000-e8f4b000 ---p 00000000 00:00 0
e8f4b000-e974a000 rw-p 00000000 00:00 0
e9f4a000-e9f4b000 ---p 00000000 00:00 0
e9f4b000-ea74a000 rw-p 00000000 00:00 0
ea74a000-ea74b000 ---p 00000000 00:00 0
ea74b000-eaf4a000 rw-p 00000000 00:00 0
eaf4a000-eaf4b000 ---p 00000000 00:00 0
eaf4b000-eb74a000 rw-p 00000000 00:00 0
eb74a000-eb74b000 ---p 00000000 00:00 0
eb74b000-ebf4a000 rw-p 00000000 00:00 0
ec600000-ec621000 rw-p 00000000 00:00 0
ec621000-ec700000 ---p 00000000 00:00 0
ec74a000-ec864000 rw-s 00000000 00:04 2293771 /SYSV00000000 (deleted)
ec864000-ec89e000 rw-p 00000000 00:00 0
ec91b000-ec923000 r-xp 00000000 08:01 32779 /usr/lib/libXcursor.so.1.0.2
ec923000-ec924000 rw-p 00007000 08:01 32779 /usr/lib/libXcursor.so.1.0.2
ec939000-eca27000 r--p 00180000 08:01 106154 /usr/lib/locale/locale-archive
eca27000-ecc27000 r--p 00000000 08:01 106154 /usr/lib/locale/locale-archive
ecc27000-ecc2d000 r-xp 00000000 08:01 75964 /usr/lib/libXrandr.so.2.2.0
ecc2d000-ecc2e000 rw-p 00005000 08:01 75964 /usr/lib/libXrandr.so.2.2.0
ecc43000-ecd28000 rw-p 00000000 00:00 0
ecded000-ece4e000 rw-p 00000000 00:00 0
ece4e000-ece55000 r--s 00000000 08:01 84015 /usr/lib/gconv/gconv-modules.cache
ece55000-ece56000 rw-p 00000000 00:00 0
ece56000-ede56000 rw-p 00000000 00:00 0
ede56000-ede58000 rw-p 00000000 00:00 0
ede58000-ede78000 rw-p 00000000 00:00 0
ede78000-ede79000 rw-p 00000000 00:00 0
ede79000-ede83000 r-xp 00000000 08:01 188305 /lib/libnss_files-2.10.2.so
ede83000-ede84000 r--p 00009000 08:01 188305 /lib/libnss_files-2.10.2.so
ede84000-ede85000 rw-p 0000a000 08:01 188305 /lib/libnss_files-2.10.2.so
ede9a000-ede9b000 rw-p 00000000 00:00 0
ede9b000-edebb000 rw-p 00000000 00:00 0
edebb000-edebd000 rw-p 00000000 00:00 0
edebd000-f5ebd000 rw-p 00000000 00:00 0
f5ebd000-f5ebe000 rw-p 00000000 00:00 0
f5ebe000-f5ebf000 ---p 00000000 00:00 0
f5ebf000-f66c2000 rw-p 00000000 00:00 0
f66c2000-f66c6000 r-xp 00000000 08:01 33553 /usr/lib/libXdmcp.so.6.0.0
f66c6000-f66c7000 rw-p 00003000 08:01 33553 /usr/lib/libXdmcp.so.6.0.0
f66c7000-f66c9000 r-xp 00000000 08:01 33554 /usr/lib/libXau.so.6.0.0
f66c9000-f66ca000 rw-p 00001000 08:01 33554 /usr/lib/libXau.so.6.0.0
f66ca000-f66cc000 r-xp 00000000 08:01 187523 /lib/libx86.so.1
f66cc000-f66cd000 rw-p 00001000 08:01 187523 /lib/libx86.so.1
f66cd000-f66ce000 rw-p 00000000 00:00 0
f66ce000-f66d2000 r-xp 00000000 08:01 187387 /lib/libattr.so.1.1.0
f66d2000-f66d3000 rw-p 00003000 08:01 187387 /lib/libattr.so.1.1.0
f66d3000-f66d7000 r-xp 00000000 08:01 33792 /usr/lib/libogg.so.0.5.3
f66d7000-f66d8000 rw-p 00003000 08:01 33792 /usr/lib/libogg.so.0.5.3
f66d8000-f66ff000 r-xp 00000000 08:01 76459 /usr/lib/libvorbis.so.0.4.4
f66ff000-f6700000 rw-p 00026000 08:01 76459 /usr/lib/libvorbis.so.0.4.4
f6700000-f670b000 r-xp 00000000 08:01 73934 /usr/lib/libvorbisenc.so.2.0.3
f670b000-f67fb000 rw-p 0000a000 08:01 73934 /usr/lib/libvorbisenc.so.2.0.3
f67fb000-f684d000 r-xp 00000000 08:01 33564 /usr/lib/libFLAC.so.8.2.0
f684d000-f684e000 rw-p 00052000 08:01 33564 /usr/lib/libFLAC.so.8.2.0
f684e000-f684f000 rw-p 00000000 00:00 0
f684f000-f6862000 r-xp 00000000 08:01 188310 /lib/libnsl-2.10.2.so
f6862000-f6863000 r--p 00012000 08:01 188310 /lib/libnsl-2.10.2.so
f6863000-f6864000 rw-p 00013000 08:01 188310 /lib/libnsl-2.10.2.so
f6864000-f6866000 rw-p 00000000 00:00 0
f6866000-f6874000 r-xp 00000000 08:01 73643 /usr/lib/libXext.so.6.4.0
f6874000-f6875000 rw-p 0000d000 08:01 73643 /usr/lib/libXext.so.6.4.0
f6875000-f6891000 r-xp 00000000 08:01 187444 /lib/libgcc_s.so.1
f6891000-f6892000 rw-p 0001c000 08:01 187444 /lib/libgcc_s.so.1
f6892000-f6977000 r-xp 00000000 08:01 75548 /usr/lib/libstdc++.so.6.0.13
f6977000-f697b000 r--p 000e5000 08:01 75548 /usr/lib/libstdc++.so.6.0.13
f697b000-f697c000 rw-p 000e9000 08:01 75548 /usr/lib/libstdc++.so.6.0.13
f697c000-f6983000 rw-p 00000000 00:00 0
f6983000-f7459000 r--p 00000000 08:01 77368 /usr/lib/libicudata.so.38.1
f7459000-f745a000 rw-p 00ad5000 08:01 77368 /usr/lib/libicudata.so.38.1
f745a000-f745b000 rw-p 00000000 00:00 0
f745b000-f745e000 r-xp 00000000 08:01 80750 /usr/lib/libgpg-error.so.0.4.0
f745e000-f745f000 rw-p 00002000 08:01 80750 /usr/lib/libgpg-error.so.0.4.0
f745f000-f746e000 r-xp 00000000 08:01 73126 /usr/lib/libtasn1.so.3.1.9
f746e000-f746f000 rw-p 0000e000 08:01 73126 /usr/lib/libtasn1.so.3.1.9
f746f000-f7471000 r-xp 00000000 08:01 187510 /lib/libkeyutils-1.2.so
f7471000-f7472000 rw-p 00001000 08:01 187510 /lib/libkeyutils-1.2.so
f7472000-f7478000 r-xp 00000000 08:01 76511 /usr/lib/libkrb5support.so.0.1
Program received signal SIGABRT, Aborted.
Here's what gdb has to say:
(gdb) bt
#0 0xf7aa0906 in raise () from /lib/libc.so.6
#1 0xf7aa3e05 in abort () from /lib/libc.so.6
#2 0xf7ad778d in ?? () from /lib/libc.so.6
#3 0xf7ae1905 in ?? () from /lib/libc.so.6
#4 0xf7ae31a3 in ?? () from /lib/libc.so.6
#5 0xf7ae622d in free () from /lib/libc.so.6
#6 0x0806ed6f in kvm_get_dirty_pages_range (kvm=0x8466c6c, phys_addr=0,
len=4294967295, opaque=0x0, cb=0x806ceb0 <kvm_get_dirty_bitmap_cb>)
at qemu-kvm.c:695
#7 0x0806ee53 in kvm_update_dirty_pages_log () at qemu-kvm.c:2483
#8 0x08050fb6 in ram_save_block (f=0x85b0008) at vl.c:3003
#9 0x0805114b in ram_save_live (mon=0x84df9b0, f=0x85b0008, stage=1, opaque=0x0) at vl.c:3105
#10 0x0810e090 in qemu_savevm_state_begin (mon=0x84df9b0, f=0x85b0008,
blk_enable=0, shared=0) at savevm.c:1323
#11 0x08105ee9 in migrate_fd_connect (s=0x8573f40) at migration.c:366
#12 0x08106cce in tcp_wait_for_connect (opaque=0x8573f40) at migration-tcp.c:72
#13 0x08051d28 in main_loop_wait (timeout=<value optimized out>)
at vl.c:4006
#14 0x0806d3c4 in kvm_main_loop ()
at qemu-kvm.c:2126
#15 0x08054bbd in main_loop (argc=7, argv=0xffffdba4, envp=Cannot access memory at address 0x6ea)
The kvm command line is trivial:
kvm -drive lin0.{raw,qcow2},boot=on,if={virtio,ide} -monitor stdio
Ie, I tried both virtio and ide interface with the same result,
and both raw and qcow2 formats, also with the exact same result.
The above backtrace was with virtio and raw.
I understand that the above traces arent' that useful, because
it's some memory corruption. But the problem does not happen
when run with -no-kvm or under valgrind.
But what does happen is -- it works all right in 64bit version.
Ie, kvm-0.12.4 compiled as 64bit binary migrates guests fine.
But when compiled as 32bit binary, and run on 32bit kernel, it
crashes.
I've also seen another version of the same failure. On the
receiving side, there are 2 messages printed:
BUG: kvm_dirty_pages_log_disable_slot: invalid parameters
BUG: kvm_dirty_pages_log_enable_slot: invalid parameters
after which the guest appears to run fine on the receiving
side, but the (cirrus text-only) sdl window output is all
garbled and new output does not appear.
And yet another variation of the same, the receiving end says
qemu: warning: error while loading state for instance 0x0 of device 'ram'
load of migration failed
and it fails, the guest continues running on the sending side.
But this version - i've seen it only once so far, but tried
various guests and systems and kvm versions.
So far, it appears that all 0.12 versions are affected. 0.11
migrates the same guests under the same host kernels just fine
and quite reliable, be it 32 or 64bits. Also, all variations
of 64bit 0.12 versions I tried also works just fine.
The same findings were confirmed by other people on IRC, and I
tested numerous versions on different systems I've access to,
so the bug's here, trivial to reproduce.
But I'm not sure if it's kvm or qemu issue... ;)
Thanks!
/mjt
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
