Am Freitag, 6. September 2013, 12:10:04 schrieb Daniel P. Berrange: > On Tue, Aug 27, 2013 at 09:09:25AM +0200, David Weber wrote: > > Hi, > > > > we try to use vcpu pinning on a 2 socket server with Intel Xeon E5620 > > cpus, HT enabled and 2*6*16GiB Ram but experience problems if we try to > > start a guest on the second socket: > > error: Failed to start domain test > > error: internal error: process exited while connecting to monitor: > > kvm_init_vcpu failed: Cannot allocate memory > > > > Libvirt version 1.1.1 > > Linux 3.11-rc7 > > > > Because I coudn't find any other service which allowed a 7M file upload, I > > put the log file and everything else which could perhabs be relevant into > > a github repository: > > https://github.com/David-Weber/vcpu-pinning > > > > When we try to start a guest on the first node it runs fine: > > <vcpu placement='static' cpuset='0-3,8-11'>4</vcpu> > > <numatune> > > > > <memory mode='strict' nodeset='0'/> > > > > </numatune> > > > > Starting it on the second node fails > > > > <vcpu placement='static' cpuset='4-7,12-15'>4</vcpu> > > <numatune> > > > > <memory mode='strict' nodeset='1'/> > > > > </numatune> > > > > Even more strange, starting it with the CPUs of the second node and the > > memory> > > of the first node works: > > <vcpu placement='static' cpuset='4-7,12-15'>4</vcpu> > > <numatune> > > > > <memory mode='strict' nodeset='0'/> > > > > </numatune> > > > > The log file contains these three cases. > > > > Using the placement='auto' parameter leads to the same problem. If numad > > return the second node, the guest won't start. > > > > Is this a configuration, a libvirt or a cgroup problem? :) > > With mode='strict' you are telling QEMU that if it can't allocate > memory from the requested node, it should fail. Is it possible > that some of your Numa nodes have insufficient memory free ? > > The combination of 'virsh capabilities' output and the results > of 'virsh freecell NODENUM' for each NUMA node will give an > indication of the allocation state. > > > Daniel Thank you for your response. There should be plenty of free memory be available. # virsh capabilities <capabilities> <host> <uuid>91ecf0d9-c821-c9ca-c7fa-00259064e5c6</uuid> <cpu> <arch>x86_64</arch> <model>Westmere</model> <vendor>Intel</vendor> <topology sockets='1' cores='4' threads='2'/> <feature name='rdtscp'/> <feature name='pdpe1gb'/> <feature name='dca'/> <feature name='pcid'/> <feature name='pdcm'/> <feature name='xtpr'/> <feature name='tm2'/> <feature name='est'/> <feature name='smx'/> <feature name='vmx'/> <feature name='ds_cpl'/> <feature name='monitor'/> <feature name='dtes64'/> <feature name='pclmuldq'/> <feature name='pbe'/> <feature name='tm'/> <feature name='ht'/> <feature name='ss'/> <feature name='acpi'/> <feature name='ds'/> <feature name='vme'/> </cpu> <power_management> <suspend_mem/> <suspend_disk/> <suspend_hybrid/> </power_management> <migration_features> <live/> <uri_transports> <uri_transport>tcp</uri_transport> </uri_transports> </migration_features> <topology> <cells num='2'> <cell id='0'> <memory unit='KiB'>99003008</memory> <cpus num='8'> <cpu id='0' socket_id='0' core_id='0' siblings='0,8'/> <cpu id='1' socket_id='0' core_id='1' siblings='1,9'/> <cpu id='2' socket_id='0' core_id='9' siblings='2,10'/> <cpu id='3' socket_id='0' core_id='10' siblings='3,11'/> <cpu id='8' socket_id='0' core_id='0' siblings='0,8'/> <cpu id='9' socket_id='0' core_id='1' siblings='1,9'/> <cpu id='10' socket_id='0' core_id='9' siblings='2,10'/> <cpu id='11' socket_id='0' core_id='10' siblings='3,11'/> </cpus> </cell> <cell id='1'> <memory unit='KiB'>99088544</memory> <cpus num='8'> <cpu id='4' socket_id='1' core_id='0' siblings='4,12'/> <cpu id='5' socket_id='1' core_id='1' siblings='5,13'/> <cpu id='6' socket_id='1' core_id='9' siblings='6,14'/> <cpu id='7' socket_id='1' core_id='10' siblings='7,15'/> <cpu id='12' socket_id='1' core_id='0' siblings='4,12'/> <cpu id='13' socket_id='1' core_id='1' siblings='5,13'/> <cpu id='14' socket_id='1' core_id='9' siblings='6,14'/> <cpu id='15' socket_id='1' core_id='10' siblings='7,15'/> </cpus> </cell> </cells> </topology> <secmodel> <model>none</model> <doi>0</doi> </secmodel> <secmodel> <model>dac</model> <doi>0</doi> </secmodel> </host> <guest> <os_type>hvm</os_type> <arch name='i686'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-x86_64</emulator> <machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine> <machine maxCpus='255'>pc-q35-1.4</machine> <machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine> <machine maxCpus='1'>isapc</machine> <machine maxCpus='255'>pc-0.10</machine> <machine maxCpus='255'>pc-0.11</machine> <machine maxCpus='255'>pc-0.12</machine> <machine maxCpus='255'>pc-0.13</machine> <machine maxCpus='255'>pc-0.14</machine> <machine maxCpus='255'>pc-0.15</machine> <machine maxCpus='255'>pc-1.0</machine> <machine maxCpus='255'>pc-1.1</machine> <machine maxCpus='255'>pc-1.2</machine> <machine maxCpus='255'>pc-1.3</machine> <machine maxCpus='255'>pc-i440fx-1.4</machine> <machine maxCpus='1'>none</machine> <domain type='qemu'> </domain> <domain type='kvm'> <emulator>/usr/bin/qemu-kvm</emulator> <machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine> <machine maxCpus='255'>pc-q35-1.4</machine> <machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine> <machine maxCpus='1'>isapc</machine> <machine maxCpus='255'>pc-0.10</machine> <machine maxCpus='255'>pc-0.11</machine> <machine maxCpus='255'>pc-0.12</machine> <machine maxCpus='255'>pc-0.13</machine> <machine maxCpus='255'>pc-0.14</machine> <machine maxCpus='255'>pc-0.15</machine> <machine maxCpus='255'>pc-1.0</machine> <machine maxCpus='255'>pc-1.1</machine> <machine maxCpus='255'>pc-1.2</machine> <machine maxCpus='255'>pc-1.3</machine> <machine maxCpus='255'>pc-i440fx-1.4</machine> <machine maxCpus='1'>none</machine> </domain> </arch> <features> <cpuselection/> <deviceboot/> <acpi default='on' toggle='yes'/> <apic default='on' toggle='no'/> <pae/> <nonpae/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='x86_64'> <wordsize>64</wordsize> <emulator>/usr/bin/qemu-system-x86_64</emulator> <machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine> <machine maxCpus='255'>pc-q35-1.4</machine> <machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine> <machine maxCpus='1'>isapc</machine> <machine maxCpus='255'>pc-0.10</machine> <machine maxCpus='255'>pc-0.11</machine> <machine maxCpus='255'>pc-0.12</machine> <machine maxCpus='255'>pc-0.13</machine> <machine maxCpus='255'>pc-0.14</machine> <machine maxCpus='255'>pc-0.15</machine> <machine maxCpus='255'>pc-1.0</machine> <machine maxCpus='255'>pc-1.1</machine> <machine maxCpus='255'>pc-1.2</machine> <machine maxCpus='255'>pc-1.3</machine> <machine maxCpus='255'>pc-i440fx-1.4</machine> <machine maxCpus='1'>none</machine> <domain type='qemu'> </domain> <domain type='kvm'> <emulator>/usr/bin/qemu-kvm</emulator> <machine canonical='pc-i440fx-1.5' maxCpus='255'>pc</machine> <machine maxCpus='255'>pc-q35-1.4</machine> <machine canonical='pc-q35-1.5' maxCpus='255'>q35</machine> <machine maxCpus='1'>isapc</machine> <machine maxCpus='255'>pc-0.10</machine> <machine maxCpus='255'>pc-0.11</machine> <machine maxCpus='255'>pc-0.12</machine> <machine maxCpus='255'>pc-0.13</machine> <machine maxCpus='255'>pc-0.14</machine> <machine maxCpus='255'>pc-0.15</machine> <machine maxCpus='255'>pc-1.0</machine> <machine maxCpus='255'>pc-1.1</machine> <machine maxCpus='255'>pc-1.2</machine> <machine maxCpus='255'>pc-1.3</machine> <machine maxCpus='255'>pc-i440fx-1.4</machine> <machine maxCpus='1'>none</machine> </domain> </arch> <features> <cpuselection/> <deviceboot/> <acpi default='on' toggle='yes'/> <apic default='on' toggle='no'/> </features> </guest> </capabilities> # virsh freecell 0 0: 86071624 KiB # virsh freecell 1 1: 75258628 KiB # virsh edit test <domain type='kvm'> <name>test</name> <uuid>08cdc389-78bf-450c-89f4-b4728edabdbf</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static' cpuset='4-7'>1</vcpu> <numatune> <memory mode='strict' nodeset='1'/> </numatune> <os> <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/bin/qemu-kvm</emulator> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'/> <video> <model type='cirrus' vram='9216' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </memballoon> </devices> </domain> # virsh start test error: Failed to start domain test error: internal error: process exited while connecting to monitor: kvm_init_vcpu failed: Cannot allocate memory Allocating memory on this node with numactl works fine # numactl --cpubind=1 --membind=1 -- dd if=/dev/zero of=/dev/null bs=2G count=1 0+1 records in 0+1 records out 2147479552 bytes (2.1 GB) copied, 0.60816 s, 3.5 GB/s David -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list