On 02/12/2014 01:03 PM, Chris Friesen wrote: > On 02/11/2014 04:45 PM, Cole Robinson wrote: >> On 02/10/2014 06:46 PM, Chris Friesen wrote: >>> Hi, >>> >>> We've run into a problem with libvirt 1.1.2 and are looking for some comments >>> on whether this is a bug or design intent. >>> >>> We're trying to use migrateToURI() but we're using a few things (numatune, >>> vcpu mask, etc.) that may need adjustment during the migration. We found that >>> migrateToURI2() mostly works if we use XML created by copying the domain XML >>> from the running instance and modifying the appropriate sections. >>> >>> The problem that we're seeing is that the serial console checking in >>> libvirt/src/conf/domain_conf.c::virDomainDefCheckABIStability() is failing >>> even though we haven't touched the serial console XML: > >> I think we will have to see the XML your passing before and after edit to >> figure out what's going wrong. Also, showing all the arguments you are >> invoking the migration APIs with. > > > Okay, this is going to be a long message. :) > > We're calling it as: > > dom.migrateToURI2(duri, muri, xml_str, logical_sum, None, CONF.live_migration_bandwidth) > > Here are the supplied arguments: > duri = u'qemu+tcp://compute-0/system' > muri = u'tcp://compute-0' > flaglist = ['VIR_MIGRATE_UNDEFINE_SOURCE', 'VIR_MIGRATE_PEER2PEER', 'VIR_MIGRATE_LIVE'] > logical_sum = 19, (logical OR of flaglist) > CONF.live_migration_bandwidth = 0 > > Grabbing original XML prior to migration: > > dom_xml = etree.fromstring(dom.XMLDesc(0)) > xml_str = etree.tostring(dom_xml) > > xml_str (BEFORE) = > <domain type="qemu" id="2"> > <name>instance-00000006</name> > <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid> > <dpdk> > <process type="secondary"/> > <file prefix="vs"/> > <cpu list="0"/> > <memory channels="4"/> > </dpdk> > <memory unit="KiB">262144</memory> > <currentMemory unit="KiB">262144</currentMemory> > <memoryBacking> > <hugepages/> > </memoryBacking> > <vcpu placement="static" cpuset="1-2">2</vcpu> > <cputune> > <vcpupin vcpu="0" cpuset="1"/> > <vcpupin vcpu="1" cpuset="2"/> > <emulatorpin cpuset="1-2"/> > </cputune> > <numatune> > <memory mode="preferred" nodeset="0"/> > </numatune> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">OpenStack Foundation</entry> > <entry name="product">OpenStack Nova</entry> > <entry name="version">2013.2.2</entry> > <entry name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry> > <entry name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry> > </system> > </sysinfo> > <os> > <type arch="x86_64" machine="pc-i440fx-1.4">hvm</type> > <boot dev="hd"/> > <smbios mode="sysinfo"/> > </os> > <features> > <acpi/> > <apic/> > </features> > <clock offset="utc"/> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>destroy</on_crash> > <devices> > <emulator>/usr/bin/qemu-system-x86_64</emulator> > <disk type="file" device="disk"> > <driver name="qemu" type="qcow2" cache="none"/> > <source file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/> > <target dev="vda" bus="virtio"/> > <alias name="virtio-disk0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/> > </disk> > <controller type="usb" index="0"> > <alias name="usb0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/> > </controller> > <controller type="pci" index="0" model="pci-root"> > <alias name="pci.0"/> > </controller> > <interface type="vswitch"> > <mac address="fa:16:3e:fb:34:6b"/> > <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/> > <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/> > <model type="e1000"/> > <alias name="net0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/> > </interface> > <interface type="vswitch"> > <mac address="fa:16:3e:d5:b9:50"/> > <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/> > <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/> > <model type="e1000"/> > <alias name="net1"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/> > </interface> > <serial type="file"> > <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/> > <target port="0"/> > <alias name="serial0"/> > </serial> > <serial type="pty"> > <source path="/dev/pts/1"/> > <target port="1"/> > <alias name="serial1"/> > </serial> > <console type="file"> > <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/> > <target type="serial" port="0"/> > <alias name="serial0"/> > </console> > <input type="tablet" bus="usb"> > <alias name="input0"/> > </input> > <input type="mouse" bus="ps2"/> > <graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0" keymap="en-us"> > <listen type="address" address="0.0.0.0"/> > </graphics> > <video> > <model type="cirrus" vram="9216" heads="1"/> > <alias name="video0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/> > </video> > <memballoon model="virtio"> > <alias name="balloon0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/> > </memballoon> > </devices> > <seclabel type="none"/> > </domain> > > > > This is the modified XML, specifically updating vcpu, cputune, numatune. > In this case they actually have the same values as above but there are > whitespace differences: > > > xml_str (MODIFIED) = > <domain type="qemu" id="2"> > <name>instance-00000006</name> > <uuid>2e13869b-edc0-40dc-92e3-1f2ffde60cce</uuid> > <dpdk> > <process type="secondary"/> > <file prefix="vs"/> > <cpu list="0"/> > <memory channels="4"/> > </dpdk> > <memory unit="KiB">262144</memory> > <currentMemory unit="KiB">262144</currentMemory> > <memoryBacking> > <hugepages/> > </memoryBacking> > <vcpu placement="static" cpuset="1,2">2</vcpu> > <cputune> > <vcpupin vcpu="0" cpuset="1"/><vcpupin vcpu="1" cpuset="2"/><emulatorpin cpuset="1,2"/></cputune> > <numatune> > <memory mode="preferred" nodeset="0"/></numatune> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">OpenStack Foundation</entry> > <entry name="product">OpenStack Nova</entry> > <entry name="version">2013.2.2</entry> > <entry name="serial">771a1b10-d2f7-4ebb-b21e-4040560732f5</entry> > <entry name="uuid">2e13869b-edc0-40dc-92e3-1f2ffde60cce</entry> > </system> > </sysinfo> > <os> > <type arch="x86_64" machine="pc-i440fx-1.4">hvm</type> > <boot dev="hd"/> > <smbios mode="sysinfo"/> > </os> > <features> > <acpi/> > <apic/> > </features> > <clock offset="utc"/> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>destroy</on_crash> > <devices> > <emulator>/usr/bin/qemu-system-x86_64</emulator> > <disk type="file" device="disk"> > <driver name="qemu" type="qcow2" cache="none"/> > <source file="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/disk"/> > <target dev="vda" bus="virtio"/> > <alias name="virtio-disk0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/> > </disk> > <controller type="usb" index="0"> > <alias name="usb0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/> > </controller> > <controller type="pci" index="0" model="pci-root"> > <alias name="pci.0"/> > </controller> > <interface type="vswitch"> > <mac address="fa:16:3e:fb:34:6b"/> > <source network="4db95d31-64f1-4f10-9138-5fe68c0d77ac"/> > <target dev="7bc76b0e-b12a-4153-8ce8-d3d145c51e6f"/> > <model type="e1000"/> > <alias name="net0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/> > </interface> > <interface type="vswitch"> > <mac address="fa:16:3e:d5:b9:50"/> > <source network="8512742c-af3e-4787-b2d8-b0b9ab160297"/> > <target dev="33ab1061-7fa4-4250-99a4-dd7eb4029aa1"/> > <model type="e1000"/> > <alias name="net1"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/> > </interface> > <serial type="file"> > <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/> > <target port="0"/> > <alias name="serial0"/> > </serial> > <serial type="pty"> > <source path="/dev/pts/1"/> > <target port="1"/> > <alias name="serial1"/> > </serial> > <console type="file"> > <source path="/etc/nova/instances/2e13869b-edc0-40dc-92e3-1f2ffde60cce/console.log"/> > <target type="serial" port="0"/> > <alias name="serial0"/> > </console> > <input type="tablet" bus="usb"> > <alias name="input0"/> > </input> > <input type="mouse" bus="ps2"/> > <graphics type="vnc" port="5900" autoport="yes" listen="0.0.0.0" keymap="en-us"> > <listen type="address" address="0.0.0.0"/> > </graphics> > <video> > <model type="cirrus" vram="9216" heads="1"/> > <alias name="video0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/> > </video> > <memballoon model="virtio"> > <alias name="balloon0"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/> > </memballoon> > </devices> > <seclabel type="none"/> > </domain> > > > > Calling migrateToURI2 then gives the following error : > 2014-02-12 03:37:17.738 1251 ERROR nova.virt.libvirt.driver [-] [instance: 2e13869b-edc0-40dc-92e3-1f2ffde60cce] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0 > > > If we remove the this section : > <serial type="pty"> > <source path="/dev/pts/1"/> > <target port="1"/> > <alias name="serial1"/> > </serial> > > Then we get the following error : > 2014-02-12 04:19:12.689 1278 ERROR nova.virt.libvirt.driver [-] [instance: 90335c32-a7ae-4c99-8e5a-cd177f6b3c22] Live Migration failure: unsupported configuration: Target domain serial port count 1 does not match source 2 > > If we remove just the following line: > <source path="/dev/pts/1"/> > > so the modified portion looks like this, > <serial type="file"> > <source path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/> > <target port="0"/> > <alias name="serial0"/> > </serial> > <serial type="pty"> > <target port="1"/> > <alias name="serial1"/> > </serial> > <console type="file"> > <source path="/etc/nova/instances/00b99f94-436e-406b-8459-83e77758f154/console.log"/> > <target type="serial" port="0"/> > <alias name="serial0"/> > </console> > <input type="tablet" bus="usb"> > > Then we get this error : > > 2014-02-12 04:29:06.402 5175 ERROR nova.virt.libvirt.driver [-] [instance: 00b99f94-436e-406b-8459-83e77758f154] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0 > > If we modify it with empty <serial type='pty'> element like this: > > <serial type="file"> > <source path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/> > <target port="0"/> > <alias name="serial0"/> > </serial> > <serial type="pty"> > </serial> > <console type="file"> > <source path="/etc/nova/instances/b1103349-8c9f-4315-9144-03fc22f09be1/console.log"/> > <target type="serial" port="0"/> > <alias name="serial0"/> > </console> > > Then we get this error: > > 2014-02-12 04:36:45.686 5358 ERROR nova.virt.libvirt.driver [-] [instance: b1103349-8c9f-4315-9144-03fc22f09be1] Live Migration failure: unsupported configuration: Target domain console count 1 does not match source 0 > > If we modify it with empty <serial type='pty'> element and also remove > <console type='file'> section completely like this : > > <serial type="file"> > <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/> > <target port="0"/> > <alias name="serial0"/> > </serial> > <serial type="pty"> > </serial> > > Then this successfully migrates! > > However, it then fails on subsequent migration back to the original host. Prior to migrating > back the domain XML serial/console section looks like: > > <serial type="file"> > <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/> > <target port="0"/> > <alias name="serial0"/> > </serial> > <serial type="pty"> > <source path="/dev/pts/1"/> > <target port="1"/> > <alias name="serial1"/> > </serial> > <console type="file"> > <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/> > <target type="serial" port="0"/> > <alias name="serial0"/> > </console> > > After modifying empty <serial type='pty'> element and removing the <console type='file'> section > it looks like this : > > <serial type="file"> > <source path="/etc/nova/instances/8cccb4ad-2abc-4513-b8fd-bfb811301677/console.log"/> > <target port="0"/> > <alias name="serial0"/> > </serial> > <serial type="pty"> > </serial> > > But this give the following error when migrating back: > > 2014-02-12 06:00:19.831 7125 ERROR nova.virt.libvirt.driver [-] [instance: 8cccb4ad-2abc-4513-b8fd-bfb811301677] Live Migration failure: unsupported configuration: Target domain console count 0 does not match source 1 > Thanks for the detailed info. Can you put that into an upstream libvirt bug report? http://libvirt.org/bugs.html Does this sound familiar to any other libvirt devs? - Cole -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list