Hi everyone
I'm using Ubuntu 12.04.3 LTS with the Xen hypervisor
installed (package version: 4.1.2-2ubuntu2.10),
and Libvirt v0.10.2.2 (that interacts with Xen through
the libxl driver).
I performed some adjustments to have the required
features:
- Switch from Xend to libxl:
I created upstart jobs for xenstored and xenconsoled
so that necessary daemons are started earlier enough
and Libvirt can contact the hypervisor through libxl.
- Add support for Open vSwitch:
I added the script 'vif-openvswitch', taken from
the master branch of the GIT repository, to the
/etc/xen/scripts directory in order to plug network
interfaces to Open vSwitch switches.
I know that the support for libxl is experimental
in Xen 4.1, but I'm wondering if someone has a solution
to the following issues:
1) On domain destroy, the host configuration is not clean:
if the flag 'force' is set to 0 some entries in the
xenstored database are not removed (console, ...)
if the flag 'force' is set to 1 (as it is done
in Libvirt), the event 'online' associated to the
network interface is not triggered and, thus, the
vif is not removed from the Open vSwitch switch.
To fix this issue, I temporarily added a call
to vif-openvswitch into the xen-hotplug-cleanup script.
2) When a domain is started from Libvirt, the console
source path does not appear in the output of the
command 'virsh dumpxml instance-00000381'
3) Sometimes, a domU is not started at all. If I start
a virtual machine from Libvirt through virsh or directly
through xl, these tools report that the virtual machine
is running but I see no messages coming from the console
and the CPU time remains fixed.
I'm attaching some files that may be useful to understand
the problem:
- libvirt.xml: domain specifications (Libvirt format)
- domain.xl: domain specifications (Xen format)
- libvirt_virsh_dumpxml.log: output of the command
'virsh dumpxml instance-00000381'
- xl_create.log: output of the command 'xl create -d domain.xl'
- xl_info.log: output of the command 'xl info'
- qemu-dm-instance-00000381.log: qemu-dm log
- syslog_vif6_0.log: vif attach/detach logs from /var/log/syslog
- xenstore-db-not-clean.log: xenstore db entries not removed
after 'xl destroy'
- libvirt_libxl.log: log from /var/log/libvirt/libxl.log
after the execution of the command 'virsh create libvirt.xml'
Lastly, I want to add that the image I'm using for the tests
is Ubuntu Precise 12.04.3 LTS, taken from the URL:
http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
and the kernel is taken from the URL:
http://cloud-images.ubuntu.com/precise/current/unpacked/precise-server-cloudimg-amd64-vmlinuz-virtual
Thanks
Roberto Sassu
<domain type="xen">
<uuid>a65c782d-05fe-4c54-9911-8e7e4cb33244</uuid>
<name>instance-00000381</name>
<memory>524288</memory>
<vcpu>1</vcpu>
<os>
<type>xen</type>
<kernel>/var/lib/nova/instances/instance-00000381/kernel</kernel>
<cmdline>ro root=/dev/xvda1</cmdline>
<root>/dev/xvda1</root>
</os>
<features>
<acpi/>
</features>
<clock offset="utc"/>
<devices>
<disk type="file" device="disk">
<driver name="tap" type="raw" cache="none"/>
<source file="/var/lib/nova/instances/instance-00000381/disk"/>
<target bus="xen" dev="xvda"/>
</disk>
<interface type="bridge">
<mac address="fa:16:3e:55:5d:e0"/>
<script path="/etc/xen/scripts/vif-openvswitch"/>
<source bridge="br-tvd-91970" bridgetype="openvswitch"/>
<virtualport type="openvswitch">
<parameters interfaceid="67d41c55-2849-4bc6-b138-c685ae9d3d5f"/>
</virtualport>
</interface>
<console type="pty"/>
<graphics type="vnc" autoport="yes" keymap="en-us" listen="node-110"/>
</devices>
</domain>
name = "instance-00000381"
uuid = "a65c782d-05fe-4c54-9911-8e7e4cb33244"
maxmem = 512
memory = 512
vcpus = 1
kernel = "/var/lib/nova/instances/instance-00000381/kernel"
extra = "ro root=/dev/xvda1"
localtime = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "destroy"
vfb = [ "type=vnc,vncunused=1,vnclisten=node-110,keymap=en-us" ]
disk = [ "tap:raw:/var/lib/nova/instances/instance-00000381/disk,xvda,w" ]
vif = [ "mac=fa:16:3e:55:5d:e0,bridge=br-tvd-91970,script=vif-bridge" ]
<domain type='xen' id='2'>
<name>instance-00000381</name>
<uuid>a65c782d-05fe-4c54-9911-8e7e4cb33244</uuid>
<memory unit='KiB'>524288</memory>
<currentMemory unit='KiB'>524288</currentMemory>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='x86_64' machine='xenpv'>linux</type>
<kernel>/var/lib/nova/instances/instance-00000381/kernel</kernel>
<cmdline>ro root=/dev/xvda1</cmdline>
<root>/dev/xvda1</root>
</os>
<features>
<acpi/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<disk type='file' device='disk'>
<driver name='tap' type='raw' cache='none'/>
<source file='/var/lib/nova/instances/instance-00000381/disk'/>
<target dev='xvda' bus='xen'/>
</disk>
<interface type='bridge'>
<mac address='fa:16:3e:55:5d:e0'/>
<source bridge='br-tvd-91970' bridgetype='openvswitch'/>
<virtualport type='openvswitch'>
<parameters interfaceid='67d41c55-2849-4bc6-b138-c685ae9d3d5f'/>
</virtualport>
<script path='/etc/xen/scripts/vif-openvswitch'/>
</interface>
<console type='pty'>
<target type='xen' port='0'/>
</console>
<input type='mouse' bus='xen'/>
<graphics type='vnc' port='5900' autoport='yes' listen='node-110' keymap='en-us'>
<listen type='address' address='node-110'/>
</graphics>
<video>
<model type='xen' vram='4096' heads='1'/>
</video>
<memballoon model='xen'/>
</devices>
</domain>
Parsing config file domain.xl
(domain
(domid -1)
(create_info)
(hvm 0)
(hap 1)
(oos 1)
(ssidref 0)
(name instance-00000381)
(uuid <unknown>)
(cpupool Pool-0)
(xsdata (null))
(platformdata (null))
(build_info)
(max_vcpus 1)
(tsc_mode 0)
(max_memkb 524288)
(target_memkb 524288)
(nomigrate 0)
(image
(linux 0)
(kernel /var/lib/nova/instances/instance-00000381/kernel)
(cmdline ro root=/dev/xvda1)
(ramdisk (null))
)
)
(device
(tap
(backend_domid 0)
(frontend_domid 0)
(physpath /var/lib/nova/instances/instance-00000381/disk)
(phystype 2)
(virtpath xvda)
(unpluggable 0)
(readwrite 1)
(is_cdrom 0)
)
)
(device
(vif
(backend_domid 0)
(frontend_domid 0)
(devid 0)
(mtu 1492)
(model rtl8139)
(mac fa163e555de0)
)
)
(device
(vfb
(backend_domid 0)
(frontend_domid 0)
(devid 0)
(vnc 1)
(vnclisten node-110)
(vncdisplay 0)
(vncunused 1)
(keymap en-us)
(sdl 0)
(opengl 0)
(display (null))
(xauthority (null))
)
)
)
Daemon running with PID 3658
host : node-110
release : 3.2.0-40-ima
version : #64 SMP Tue Apr 9 14:21:15 CEST 2013
machine : x86_64
nr_cpus : 2
nr_nodes : 1
cores_per_socket : 2
threads_per_core : 1
cpu_mhz : 2128
hw_caps : bfebfbff:20100800:00000000:00000940:0000e3bd:00000000:00000001:00000000
virt_caps : hvm
total_memory : 2023
free_memory : 151
free_cpus : 0
xen_major : 4
xen_minor : 1
xen_extra : .2
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline : placeholder
cc_compiler : gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
cc_compile_by : roberto.sassu
cc_compile_domain : polito.it
cc_compile_date : Fri Jul 26 11:33:14 CEST 2013
xend_config_format : 4
domid: 7
Warning: vlan 0 is not connected to host network
-videoram option does not work with cirrus vga device model. Videoram set to 4M.
char device redirected to /dev/pts/3
xs_read(): target get error. /local/domain/7/target.
xs_read(): vncpasswd get error. /vm/a65c782d-05fe-4c54-9911-8e7e4cb33244/vncpasswd.
Sep 11 13:42:08 node-110 logger: /etc/xen/scripts/vif-openvswitch: online type_if=vif XENBUS_PATH=backend/vif/6/0
Sep 11 13:42:08 node-110 ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl --timeout=30 -- --if-exists del-port vif6.0 -- add-port br-tvd-91970 vif6.0 -- set interface vif6.0 "external-ids:\"xen-vm-name\"=\"instance-00000381\"" -- set interface vif6.0 "external-ids:\"xen-vm-uuid\"=\"a65c782d-05fe-4c54-9911-8e7e4cb33244\"" -- set interface vif6.0 "external-ids:\"attached-mac\"=\"fa:16:3e:55:5d:e0\""
Sep 11 13:42:08 node-110 kernel: [ 243.392972] device vif6.0 entered promiscuous mode
Sep 11 13:42:08 node-110 kernel: [ 243.399111] ADDRCONF(NETDEV_UP): vif6.0: link is not ready
Sep 11 13:42:08 node-110 logger: /etc/xen/scripts/vif-openvswitch: Successful vif-openvswitch online for vif6.0.
Sep 11 13:42:08 node-110 logger: /etc/xen/scripts/vif-openvswitch: Writing backend/vif/6/0/hotplug-status connected to xenstore.
Sep 11 13:42:09 node-110 kernel: [ 244.678444] ADDRCONF(NETDEV_CHANGE): vif6.0: link becomes ready
Sep 11 13:42:12 node-110 kernel: [ 247.923648] vif vif-6-0: 2 reading script
Sep 11 13:42:12 node-110 kernel: [ 247.927140] device vif6.0 left promiscuous mode
Sep 11 13:42:12 node-110 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/qdisk/6/51712
Sep 11 13:42:12 node-110 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vif/6/0
Sep 11 13:42:12 node-110 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vfb/6/0
Sep 11 13:42:12 node-110 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/console/6/0
Sep 11 13:42:12 node-110 logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vkbd/6/0
Sep 11 13:42:13 node-110 logger: /etc/xen/scripts/vif-openvswitch: offline type_if=vif XENBUS_PATH=backend/vif/6/0
Sep 11 13:42:13 node-110 ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl --timeout=30 -- --if-exists del-port vif6.0
Sep 11 13:42:13 node-110 logger: /etc/xen/scripts/vif-openvswitch: ip link set vif6.0 down failed
Sep 11 13:42:13 node-110 logger: /etc/xen/scripts/vif-openvswitch: Successful vif-openvswitch offline for vif6.0.
tool = ""
xenstored = ""
local = ""
domain = ""
0 = ""
name = "Domain-0"
memory = ""
target = "1387864"
static-max = "4294967292"
freemem-slack = "153544"
backend = ""
qdisk = ""
1 = ""
51712 = ""
frontend = "/local/domain/1/device/vbd/51712"
params = "aio:/var/lib/nova/instances/instance-00000381/disk"
frontend-id = "1"
online = "0"
removable = "0"
bootable = "1"
state = "5"
dev = "xvda"
type = "tap"
mode = "w"
feature-barrier = "1"
info = "0"
sector-size = "512"
sectors = "4194304"
hotplug-status = "connected"
vfb = ""
1 = ""
0 = ""
frontend = "/local/domain/1/device/vfb/0"
frontend-id = "1"
online = "0"
state = "5"
domain = "instance-00000381"
vnc = "1"
vnclisten = "node-110"
vncdisplay = "0"
vncunused = "1"
sdl = "0"
opengl = "0"
feature-resize = "1"
hotplug-status = "connected"
request-update = "1"
vkbd = ""
1 = ""
0 = ""
frontend = "/local/domain/1/device/vkbd/0"
frontend-id = "1"
online = "0"
state = "5"
domain = "instance-00000381"
feature-abs-pointer = "1"
hotplug-status = "connected"
console = ""
1 = ""
0 = ""
frontend = "/local/domain/1/console"
frontend-id = "1"
online = "0"
state = "5"
domain = "instance-00000381"
protocol = "vt100"
hotplug-status = "connected"
device-model = ""
vm = ""
00000000-0000-0000-0000-000000000000 = ""
memory = "1355"
domainbuilder: detail: xc_dom_allocate: cmdline="ro root=/dev/xvda1", features="(null)"
domainbuilder: detail: xc_dom_kernel_file: filename="/var/lib/nova/instances/instance-00000381/kernel"
domainbuilder: detail: xc_dom_malloc_filemap : 4849 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.1, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
domainbuilder: detail: xc_dom_malloc : 18250 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x4b35c9 -> 0x11d2aa0
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x1000000 memsz=0xad6000
xc: detail: elf_parse_binary: phdr: paddr=0x1c00000 memsz=0xe30e0
xc: detail: elf_parse_binary: phdr: paddr=0x1ce4000 memsz=0x143c0
xc: detail: elf_parse_binary: phdr: paddr=0x1cf9000 memsz=0x35f000
xc: detail: elf_parse_binary: memory: 0x1000000 -> 0x2058000
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff81cf9200
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff81001000
xc: detail: elf_xen_parse_note: FEATURES = "!writable_page_tables|pae_pgdir_above_4gb"
xc: detail: elf_xen_parse_note: PAE_MODE = "yes"
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_parse_note: HV_START_LOW = 0xffff800000000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail: virt_base = 0xffffffff80000000
xc: detail: elf_paddr_offset = 0x0
xc: detail: virt_offset = 0xffffffff80000000
xc: detail: virt_kstart = 0xffffffff81000000
xc: detail: virt_kend = 0xffffffff82058000
xc: detail: virt_entry = 0xffffffff81cf9200
xc: detail: p2m_base = 0xffffffffffffffff
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0xffffffff81000000 -> 0xffffffff82058000
domainbuilder: detail: xc_dom_mem_init: mem 512 MB, pages 0x20000 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x20000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc : 1024 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment: kernel : 0xffffffff81000000 -> 0xffffffff82058000 (pfn 0x1000 + 0x1058 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x1000+0x1058 at 0x7f42bafa8000
xc: detail: elf_load_binary: phdr 0 at 0x139924581548032 -> 0x139924592910336
xc: detail: elf_load_binary: phdr 1 at 0x139924594130944 -> 0x139924595060960
xc: detail: elf_load_binary: phdr 2 at 0x139924595064832 -> 0x139924595147712
xc: detail: elf_load_binary: phdr 3 at 0x139924595150848 -> 0x139924596039680
domainbuilder: detail: xc_dom_alloc_segment: phys2mach : 0xffffffff82058000 -> 0xffffffff82158000 (pfn 0x2058 + 0x100 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x2058+0x100 at 0x7f42ef6ba000
domainbuilder: detail: xc_dom_alloc_page : start info : 0xffffffff82158000 (pfn 0x2158)
domainbuilder: detail: xc_dom_alloc_page : xenstore : 0xffffffff82159000 (pfn 0x2159)
domainbuilder: detail: xc_dom_alloc_page : console : 0xffffffff8215a000 (pfn 0x215a)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000 -> 0xffffffff823fffff, 18 table(s)
domainbuilder: detail: xc_dom_alloc_segment: page tables : 0xffffffff8215b000 -> 0xffffffff82170000 (pfn 0x215b + 0x15 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x215b+0x15 at 0x7f42ef800000
domainbuilder: detail: xc_dom_alloc_page : boot stack : 0xffffffff82170000 (pfn 0x2170)
domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0xffffffff82171000
domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0xffffffff82400000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x20000
domainbuilder: detail: clear_page: pfn 0x215a, mfn 0x4adad
domainbuilder: detail: clear_page: pfn 0x2159, mfn 0x4adae
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x2158+0x1 at 0x7f42ef7ff000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff81001000 pfn=0x1001
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail: allocated
domainbuilder: detail: malloc : 19381 kB
domainbuilder: detail: anon mmap : 0 bytes
domainbuilder: detail: mapped
domainbuilder: detail: file mmap : 4849 kB
domainbuilder: detail: domU mmap : 17848 kB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0x4bb68
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x215b mfn 0x4adac
domainbuilder: detail: launch_vm: called, ctxt=0x7f42eb556300
domainbuilder: detail: xc_dom_release: called
--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list