>>>>> "CC" == saint <saint@xxxxxx> writes: CC> I can livemigrate w/o problems between any two from red9, red10 CC> and red11, I can livemigrate from either red9, red10 or red11 to CC> red3 but I can't livemigrate to any other blade from red3. That's a selection of lines from red9 log when red10 succeeds in migrating (live) one-71 and red3 fails to migrate one-73: Success from red10: 19:15:25.120: debug : virConnectOpen:1103 : name=qemu:///system 19:15:25.120: debug : do_open:973 : name "qemu:///system" to URI components: scheme qemu opaque (null) authority (null) server (null) user (null) port 0 path /system 19:15:25.120: debug : do_open:983 : trying driver 0 (Test) ... 19:15:25.120: debug : do_open:989 : driver 0 Test returned DECLINED 19:15:25.120: debug : do_open:983 : trying driver 1 (Xen) ... 19:15:25.120: debug : do_open:989 : driver 1 Xen returned DECLINED 19:15:25.120: debug : do_open:983 : trying driver 2 (remote) ... 19:15:25.120: debug : do_open:989 : driver 2 remote returned DECLINED 19:15:25.120: debug : do_open:983 : trying driver 3 (QEMU) ... 19:15:25.120: debug : do_open:989 : driver 3 QEMU returned SUCCESS 19:15:25.120: debug : do_open:1009 : network driver 0 Test returned DECLINED 19:15:25.120: debug : do_open:1009 : network driver 1 remote returned DECLINED 19:15:25.120: debug : do_open:1009 : network driver 2 Network returned SUCCESS 19:15:25.120: debug : do_open:1031 : storage driver 0 Test returned DECLINED 19:15:25.120: debug : do_open:1031 : storage driver 1 remote returned DECLINED 19:15:25.120: debug : do_open:1031 : storage driver 2 storage returned SUCCESS 19:15:25.120: debug : do_open:1052 : node driver 0 Test returned DECLINED 19:15:25.120: debug : do_open:1052 : node driver 1 remote returned DECLINED 19:15:25.120: debug : do_open:1052 : node driver 2 halDeviceMonitor returned SUCCESS 19:15:25.122: debug : virDrvSupportsFeature:1232 : conn=0x2aaaac000970, feature=3 19:15:25.125: debug : virDomainMigratePrepare2:3071 : dconn=0x2aaaac000970, cookie=0x43624eb8, cookielen=0x43624ec4, uri_in=(null), uri_out=0x18e54a80,flags=1, dname=(null), bandwidth=0, dom_xml=<domain type='kvm' id='7'> <name>one-71</name> <uuid>346d40a2-10f7-d39a-f969-7f1c901db6ca</uuid> <memory>262144</memory> <currentMemory>262144</currentMemory> <vcpu>1</vcpu> <os> <type arch='x86_64' machine='rhel5.4.0'>hvm</type> <boot dev='hd'/> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <source file='/srv/cloud/one/var/images/71/images/disk.0'/> <target dev='hda' bus='ide'/> </disk> <disk type='file' device='cdrom'> <source file='/srv/cloud/one/var/images/71/images/disk.1'/> <target dev='hdb' bus='ide'/> <readonly/> </disk> <interface type='bridge'> <mac address='02:00:a1:1b:d6:d5'/> <source bridge='bridge0.334'/> <target dev='vnet0'/> </interface> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5971' autoport='no' listen='0.0.0.0'/> </devices> </domain> 19:15:25.126: debug : virExecWithHook:546 : /usr/bin/kvm -help 19:15:25.133: info : Received unexpected signal 17 19:15:25.147: debug : virExecWithHook:546 : /usr/bin/kvm -S -M rhel5.4.0 -m 256 -smp 1 -name one-71 -uuid 346d40a2-10f7-d39a-f969-7f1c901db6ca -no-kvm-pit-reinjection -monitor pty -pidfile /var/run/libvirt/qemu//one-71.pid -no-acpi -boot c -drive file=/srv/cloud/one/var/images/71/images/disk.0,if=ide,index=0,boot=on -drive file=/srv/cloud/one/var/images/71/images/disk.1,if=ide,media=cdrom,index=1 -net nic,macaddr=02:00:a1:1b:d6:d5,vlan=0 -net tap,fd=16,script=,vlan=0,ifname=vnet3 -serial none -parallel none -usb -vnc 0.0.0.0:71 -incoming tcp:0.0.0.0:49154 19:15:25.148: info : Received unexpected signal 17 19:15:25.624: debug : virConnectOpen:1103 : name=qemu:///system 19:15:25.624: debug : do_open:973 : name "qemu:///system" to URI components: scheme qemu opaque (null) authority (null) server (null) user (null) port 0 path /system 19:15:25.624: debug : do_open:983 : trying driver 0 (Test) ... 19:15:25.624: debug : do_open:989 : driver 0 Test returned DECLINED 19:15:25.624: debug : do_open:983 : trying driver 1 (Xen) ... 19:15:25.624: debug : do_open:989 : driver 1 Xen returned DECLINED 19:15:25.624: debug : do_open:983 : trying driver 2 (remote) ... 19:15:25.624: debug : do_open:989 : driver 2 remote returned DECLINED 19:15:25.624: debug : do_open:983 : trying driver 3 (QEMU) ... 19:15:25.624: debug : do_open:989 : driver 3 QEMU returned SUCCESS 19:15:25.624: debug : do_open:1009 : network driver 0 Test returned DECLINED 19:15:25.624: debug : do_open:1009 : network driver 1 remote returned DECLINED 19:15:25.624: debug : do_open:1009 : network driver 2 Network returned SUCCESS 19:15:25.624: debug : do_open:1031 : storage driver 0 Test returned DECLINED 19:15:25.624: debug : do_open:1031 : storage driver 1 remote returned DECLINED 19:15:25.624: debug : do_open:1031 : storage driver 2 storage returned SUCCESS 19:15:25.624: debug : do_open:1052 : node driver 0 Test returned DECLINED 19:15:25.624: debug : do_open:1052 : node driver 1 remote returned DECLINED 19:15:25.624: debug : do_open:1052 : node driver 2 halDeviceMonitor returned SUCCESS 19:15:25.625: debug : virNodeGetInfo:3165 : conn=0x18e790b0, info=0x42c23e80 19:15:25.626: debug : qemudDispatchClientFailure:1407 : Deregistering to relay remote events 19:15:25.626: debug : virConnectDomainEventDeregister:7760 : conn=0x18e790b0, cb=0x4143d0 19:15:26.178: debug : qemudDomainSetMemoryBalloon:2719 : one-71: balloon reply: balloon 256 19:15:26.179: debug : virConnectClose:1171 : conn=0x18e790b0 19:15:26.179: debug : virUnrefConnect:210 : unref connection 0x18e790b0 1 19:15:26.179: debug : virReleaseConnect:171 : release connection 0x18e790b0 19:15:28.709: debug : virDomainMigrateFinish2:3119 : dconn=0x2aaaac000970, dname=one-71, cookie=(nil), cookielen=0, uri=tcp:itpsmenc044.engicloud.psmidc.com:49154, flags=1, retcode=0 19:15:28.710: debug : virGetDomain:271 : New hash entry 0x2aaaac000a80 19:15:28.729: debug : qemudDispatchClientFailure:1407 : Deregistering to relay remote events 19:15:28.729: debug : virConnectDomainEventDeregister:7760 : conn=0x2aaaac000970, cb=0x4143d0 19:15:28.729: debug : virConnectClose:1171 : conn=0x2aaaac000970 19:15:28.729: debug : virUnrefConnect:210 : unref connection 0x2aaaac000970 2 failure from red3: 18:03:55.054: debug : virConnectOpen:1103 : name=qemu:///system 18:03:55.054: debug : do_open:973 : name "qemu:///system" to URI components: scheme qemu opaque (null) authority (null) server (null) user (null) port 0 path /system 18:03:55.054: debug : do_open:983 : trying driver 0 (Test) ... 18:03:55.054: debug : do_open:989 : driver 0 Test returned DECLINED 18:03:55.054: debug : do_open:983 : trying driver 1 (Xen) ... 18:03:55.054: debug : do_open:989 : driver 1 Xen returned DECLINED 18:03:55.054: debug : do_open:983 : trying driver 2 (remote) ... 18:03:55.054: debug : do_open:989 : driver 2 remote returned DECLINED 18:03:55.054: debug : do_open:983 : trying driver 3 (QEMU) ... 18:03:55.054: debug : do_open:989 : driver 3 QEMU returned SUCCESS 18:03:55.054: debug : do_open:1009 : network driver 0 Test returned DECLINED 18:03:55.054: debug : do_open:1009 : network driver 1 remote returned DECLINED 18:03:55.054: debug : do_open:1009 : network driver 2 Network returned SUCCESS 18:03:55.054: debug : do_open:1031 : storage driver 0 Test returned DECLINED 18:03:55.054: debug : do_open:1031 : storage driver 1 remote returned DECLINED 18:03:55.054: debug : do_open:1031 : storage driver 2 storage returned SUCCESS 18:03:55.054: debug : do_open:1052 : node driver 0 Test returned DECLINED 18:03:55.054: debug : do_open:1052 : node driver 1 remote returned DECLINED 18:03:55.054: debug : do_open:1052 : node driver 2 halDeviceMonitor returned SUCCESS 18:03:55.057: debug : virDrvSupportsFeature:1232 : conn=0x18e7f280, feature=3 18:03:55.060: debug : virDomainMigratePrepare2:3071 : dconn=0x18e7f280, cookie=0x42222eb8, cookielen=0x42222ec4, uri_in=(null), uri_out=0x18e46a00,flags=1, dname=(null), bandwidth=0, dom_xml=<domain type='kvm' id='1'> <name>one-73</name> <uuid>1095202e-6042-d2f7-28bf-ad745dd341da</uuid> <memory>262144</memory> <currentMemory>262144</currentMemory> <vcpu>1</vcpu> <os> <type arch='x86_64' machine='rhel5.4.0'>hvm</type> <boot dev='hd'/> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <source file='/srv/cloud/one/var/images/73/images/disk.0'/> <target dev='hda' bus='ide'/> </disk> <disk type='file' device='cdrom'> <source file='/srv/cloud/one/var/images/73/images/disk.1'/> <target dev='hdb' bus='ide'/> <readonly/> </disk> <interface type='bridge'> <mac address='02:00:a1:1b:d6:d7'/> <source bridge='bridge0.334'/> <target dev='vnet4'/> </interface> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5973' autoport='no' listen='0.0.0.0'/> </devices> </domain> 18:03:55.061: debug : virExecWithHook:546 : /usr/bin/kvm -help 18:03:55.067: info : Received unexpected signal 17 18:03:55.078: debug : virExecWithHook:546 : /usr/bin/kvm -S -M rhel5.4.0 -m 256 -smp 1 -name one-73 -uuid 1095202e-6042-d2f7-28bf-ad745dd341da -no-kvm-pit-reinjection -monitor pty -pidfile /var/run/libvirt/qemu//one-73.pid -no-acpi -boot c -drive file=/srv/cloud/one/var/images/73/images/disk.0,if=ide,index=0,boot=on -drive file=/srv/cloud/one/var/images/73/images/disk.1,if=ide,media=cdrom,index=1 -net nic,macaddr=02:00:a1:1b:d6:d7,vlan=0 -net tap,fd=19,script=,vlan=0,ifname=vnet4 -serial none -parallel none -usb -vnc 0.0.0.0:73 -incoming tcp:0.0.0.0:49153 18:03:55.080: info : Received unexpected signal 17 18:03:56.106: debug : qemudDomainSetMemoryBalloon:2719 : one-73: balloon reply: balloon 256 18:03:56.114: debug : virDomainMigrateFinish2:3119 : dconn=0x18e7f280, dname=one-73, cookie=(nil), cookielen=0, uri=tcp:itpsmenc044.engicloud.psmidc.com:49153, flags=1, retcode=-1 18:03:56.114: debug : qemudShutdownVMDaemon:1684 : Shutting down VM 'one-73' 18:03:56.115: debug : qemudDispatchClientFailure:1407 : Deregistering to relay remote events 18:03:56.115: debug : virConnectDomainEventDeregister:7760 : conn=0x18e7f280, cb=0x4143d0 18:03:56.115: debug : virConnectClose:1171 : conn=0x18e7f280 18:03:56.115: debug : virUnrefConnect:210 : unref connection 0x18e7f280 1 18:03:56.115: debug : virReleaseConnect:171 : release connection 0x18e7f280 18:04:10.666: debug : virConnectOpen:1103 : name=qemu:///system 18:04:10.666: debug : do_open:973 : name "qemu:///system" to URI components: scheme qemu opaque (null) authority (null) server (null) user (null) port 0 path /system 18:04:10.666: debug : do_open:983 : trying driver 0 (Test) ... 18:04:10.666: debug : do_open:989 : driver 0 Test returned DECLINED 18:04:10.666: debug : do_open:983 : trying driver 1 (Xen) ... 18:04:10.666: debug : do_open:989 : driver 1 Xen returned DECLINED 18:04:10.666: debug : do_open:983 : trying driver 2 (remote) ... 18:04:10.666: debug : do_open:989 : driver 2 remote returned DECLINED 18:04:10.666: debug : do_open:983 : trying driver 3 (QEMU) ... 18:04:10.666: debug : do_open:989 : driver 3 QEMU returned SUCCESS 18:04:10.666: debug : do_open:1009 : network driver 0 Test returned DECLINED 18:04:10.666: debug : do_open:1009 : network driver 1 remote returned DECLINED 18:04:10.666: debug : do_open:1009 : network driver 2 Network returned SUCCESS 18:04:10.666: debug : do_open:1031 : storage driver 0 Test returned DECLINED 18:04:10.666: debug : do_open:1031 : storage driver 1 remote returned DECLINED 18:04:10.666: debug : do_open:1031 : storage driver 2 storage returned SUCCESS 18:04:10.666: debug : do_open:1052 : node driver 0 Test returned DECLINED 18:04:10.666: debug : do_open:1052 : node driver 1 remote returned DECLINED 18:04:10.666: debug : do_open:1052 : node driver 2 halDeviceMonitor returned SUCCESS 18:04:10.667: debug : virNodeGetInfo:3165 : conn=0x18e73df0, info=0x42222e80 18:04:10.668: debug : qemudDispatchClientFailure:1407 : Deregistering to relay remote events 18:04:10.668: debug : virConnectDomainEventDeregister:7760 : conn=0x18e73df0, cb=0x4143d0 18:04:10.668: debug : virConnectClose:1171 : conn=0x18e73df0 18:04:10.668: debug : virUnrefConnect:210 : unref connection 0x18e73df0 1 18:04:10.668: debug : virReleaseConnect:171 : release connection 0x18e73df0 18:04:14.646: debug : virConnectOpenReadOnly:1126 : name=qemu:///system 18:04:14.646: debug : do_open:973 : name "qemu:///system" to URI components: scheme qemu opaque (null) authority (null) server (null) user (null) port 0 path /system 18:04:14.646: debug : do_open:983 : trying driver 0 (Test) ... 18:04:14.646: debug : do_open:989 : driver 0 Test returned DECLINED 18:04:14.646: debug : do_open:983 : trying driver 1 (Xen) ... 18:04:14.646: debug : do_open:989 : driver 1 Xen returned DECLINED 18:04:14.646: debug : do_open:983 : trying driver 2 (remote) ... 18:04:14.646: debug : do_open:989 : driver 2 remote returned DECLINED 18:04:14.646: debug : do_open:983 : trying driver 3 (QEMU) ... 18:04:14.646: debug : do_open:989 : driver 3 QEMU returned SUCCESS 18:04:14.646: debug : do_open:1009 : network driver 0 Test returned DECLINED 18:04:14.646: debug : do_open:1009 : network driver 1 remote returned DECLINED 18:04:14.646: debug : do_open:1009 : network driver 2 Network returned SUCCESS 18:04:14.646: debug : do_open:1031 : storage driver 0 Test returned DECLINED 18:04:14.646: debug : do_open:1031 : storage driver 1 remote returned DECLINED 18:04:14.646: debug : do_open:1031 : storage driver 2 storage returned SUCCESS 18:04:14.646: debug : do_open:1052 : node driver 0 Test returned DECLINED 18:04:14.646: debug : do_open:1052 : node driver 1 remote returned DECLINED 18:04:14.646: debug : do_open:1052 : node driver 2 halDeviceMonitor returned SUCCESS 18:04:14.647: debug : virDomainLookupByName:1771 : conn=0x18e1e6b0, name=one-71 18:04:14.647: debug : virGetDomain:271 : New hash entry 0x18e717c0 18:04:14.647: debug : virDomainFree:1859 : domain=0x18e717c0 18:04:14.647: debug : virUnrefDomain:345 : unref domain 0x18e717c0 one-71 1 18:04:14.647: debug : virReleaseDomain:303 : release domain 0x18e717c0 one-71 18:04:14.647: debug : virReleaseDomain:315 : unref connection 0x18e1e6b0 2 18:04:14.648: debug : virGetDomain:271 : New hash entry 0x18e717c0 18:04:14.648: debug : virDomainGetOSType:2438 : domain=0x18e717c0 18:04:14.648: debug : virDomainFree:1859 : domain=0x18e717c0 18:04:14.648: debug : virUnrefDomain:345 : unref domain 0x18e717c0 one-71 1 18:04:14.648: debug : virReleaseDomain:303 : release domain 0x18e717c0 one-71 18:04:14.648: debug : virReleaseDomain:315 : unref connection 0x18e1e6b0 2 18:04:14.648: debug : virGetDomain:271 : New hash entry 0x18e717c0 18:04:14.648: debug : virDomainGetInfo:2622 : domain=0x18e717c0, info=0x42c23ea0 18:04:14.649: debug : qemudGetProcessInfo:2149 : Got status for 26133/0 user=4 sys=22 cpu=0 18:04:14.649: debug : qemudDomainGetMemoryBalloon:2665 : one-71: balloon reply: 'info balloon balloon: actual=256 ' 18:04:14.649: debug : virDomainFree:1859 : domain=0x18e717c0 18:04:14.649: debug : virUnrefDomain:345 : unref domain 0x18e717c0 one-71 1 18:04:14.649: debug : virReleaseDomain:303 : release domain 0x18e717c0 one-71 18:04:14.649: debug : virReleaseDomain:315 : unref connection 0x18e1e6b0 2 18:04:14.650: debug : virGetDomain:271 : New hash entry 0x18e717c0 18:04:14.650: debug : virDomainGetAutostart:3959 : domain=0x18e717c0, autostart=0x44026040 18:04:14.650: debug : virDomainFree:1859 : domain=0x18e717c0 18:04:14.650: debug : virUnrefDomain:345 : unref domain 0x18e717c0 one-71 1 18:04:14.650: debug : virReleaseDomain:303 : release domain 0x18e717c0 one-71 18:04:14.650: debug : virReleaseDomain:315 : unref connection 0x18e1e6b0 2 18:04:14.652: debug : qemudDispatchClientFailure:1407 : Deregistering to relay remote events 18:04:14.652: debug : virConnectDomainEventDeregister:7760 : conn=0x18e1e6b0, cb=0x4143d0 18:04:14.652: debug : virConnectClose:1171 : conn=0x18e1e6b0 18:04:14.652: debug : virUnrefConnect:210 : unref connection 0x18e1e6b0 1 18:04:14.652: debug : virReleaseConnect:171 : release connection 0x18e1e6b0 libsdl-console-dev libsdl-console libsdl-sge libsdl-sound1.2 Comparing the two sets of lines, the differences start after the second unexpected SIGCHLD (17) that happened at 19:15:25.148 in the successful migrate and at 18:03:55.080 during the failing attempt. As usual, any wisdom & advice about what I could have done wrong would be really appreciated. -- ing. Gian Uberto Lauri Ricercatore / Reasearcher Divisione Ricerca ed Innovazione / Research & Innovation Division GianUberto.Lauri@xxxxxx Engineering Ingegneria Informatica spa Corso Stati Uniti 23/C, 35127 Padova (PD) Tel. +39-049.8283.538 | main(){printf(&unix["\021%six\012\0"], Fax +39-049.8283.569 | (unix)["have"]+"fun"-0x60);} Skype: gian.uberto.lauri | David Korn, AT&T Bell Labs http://www.eng.it | ioccc best One Liner, 1987 P.S. I apologize for any inconvenience generated by my poor English ...