<quote who="Erik Brakkee"> > <quote who="David Cruz"> >> Other optimizations people are testing out there. >> >> - use "nohz=off" in the kernel loading line y menu.lst >> - Disable Cgroups completely. Using cgclear, and turning off cgred >> cg-config daemons. I also tried this option but it did not have a significant effect and degraded performance a bit, although it is difficult to tell because of the variations in the measured times. On a positive note, I have managed to get a significant performance improvement by checking the BIOS. When we got the server, it was configured to use the Performance Per Watt profile. All current tests were run with CPU scaling enabled (and also recognized by the linux kernel). What I did to get increased performance was to enable the 'Performance' profile instead and let the hardware do the CPU scaling (this is what I understand that the option means). Now the measured times have dropped from a consistent 19 seconds on the physical host to a consistent 10 seconds on the physical host. On the virtual host, the variable times from 21-27 seconds have now turned into consistent times of approximately 13.5 seconds. This is with the nohz=off kernel option. With the nohz=off option removed, I get consistent times that are mostly a bit under 13 seconds (12.8 upwards). These results are for a VM with 4 virtual CPUs that are also pinned to cores of the first CPU. >From the looks of it, the kernel we are using does not fully understand the CPU it is dealing with. Also, I am doubtful that we ever saw the actual CPU frequences when doing 'cat /proc/cpuinfo'. Also, positively, increasing the VCPU count from 4 to 8 and also not pinning them does not deteriorate performance. In fact, it looks like it increases performance even a bit, leading to an overhead of 20% approximately. All in all, the performance results are acceptable now looking at the absolute figures. In fact, I am quite happy with this because it means we can continue to take this setup in production. However, there still appears to be a 20% penalty for virtualization in our specific case. Do you have any ideas on how to get to the bottom of this? Being able to squeeze the last 20% out of it would really make us even more happy (my day is already made with this BIOS setting). Any ideas? > However, I got a performance degradation of running the same test with > MySQL and JBoss colocated and increasing the CPU count from 4 to 8. Then I > did various tricks to improve performance. Pinning the processes did not > have a significant effect. What appeared to have an effect was to remove > memory ballooning. > > But now I get inconsistent results, varying from 21 seconds to 27 seconds. > The 21 figure is acceptable but the 27 figure is not. The tests are being > done on new hardware (see previous posts), with basically the only thing > running the virtual machine. > > Summary of configuration: > - host and guest: centos 6.3, transparent hugepages > - vm: cpu mode is host-passthrough, pinning to the cores of one processor, > removed tablet, sound, and USB devices > - host: hyperthreading switched off in BIOS > > Do you have any idea what this could be? I expect it is somehow NUMA > related, but how would I troubleshoot this? Ideally I would like to make > sure the entire VM runs on one CPU and allocates memory from that CPU and > never moves (or both CPU and memory move together). I saw some > presentations on the internet about NUMA work being done for linux. Do you > have any suggestions? > > My domain.xml is given below: > > <domain type='kvm' id='37'> > <name>master-data05-v50</name> > <uuid>79ddd84d-937e-357b-8e57-c7f487dc3464</uuid> > <memory unit='KiB'>8388608</memory> > <currentMemory unit='KiB'>8388608</currentMemory> > <vcpu placement='static'>8</vcpu> > <cputune> > <vcpupin vcpu='0' cpuset='0'/> > <vcpupin vcpu='1' cpuset='2'/> > <vcpupin vcpu='2' cpuset='4'/> > <vcpupin vcpu='3' cpuset='6'/> > <vcpupin vcpu='4' cpuset='8'/> > <vcpupin vcpu='5' cpuset='10'/> > <vcpupin vcpu='6' cpuset='12'/> > <vcpupin vcpu='7' cpuset='14'/> > </cputune> > <os> > <type arch='x86_64' machine='rhel6.3.0'>hvm</type> > <boot dev='cdrom'/> > <boot dev='hd'/> > </os> > <features> > <acpi/> > <apic/> > </features> > <cpu mode='host-passthrough'> > </cpu> > <clock offset='utc'/> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>restart</on_crash> > <devices> > <emulator>/usr/libexec/qemu-kvm</emulator> > <disk type='file' device='cdrom'> > <driver name='qemu' type='raw'/> > <target dev='hdc' bus='ide'/> > <readonly/> > <alias name='ide0-1-0'/> > <address type='drive' controller='0' bus='1' target='0' unit='0'/> > </disk> > <disk type='block' device='disk'> > <driver name='qemu' type='raw' cache='none' io='native'/> > <source dev='/dev/raid5/v50disk1'/> > <target dev='vda' bus='virtio'/> > <alias name='virtio-disk0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x05' > function='0x0'/> > </disk> > <disk type='block' device='disk'> > <driver name='qemu' type='raw' cache='none' io='native'/> > <source dev='/dev/vg_system/v50disk2'/> > <target dev='vdb' bus='virtio'/> > <alias name='virtio-disk1'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x08' > function='0x0'/> > </disk> > <disk type='block' device='disk'> > <driver name='qemu' type='raw' cache='none' io='native'/> > <source dev='/dev/raid5/v50disk3'/> > <target dev='vdc' bus='virtio'/> > <alias name='virtio-disk2'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09' > function='0x0'/> > </disk> > <disk type='file' device='disk'> > <driver name='qemu' type='raw' cache='none'/> > <source file='/var/mydata/images/configdisks/v50/configdisk.img'/> > <target dev='vdz' bus='virtio'/> > <alias name='virtio-disk25'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x07' > function='0x0'/> > </disk> > <controller type='usb' index='0'> > <alias name='usb0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x2'/> > </controller> > <controller type='ide' index='0'> > <alias name='ide0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x01' > function='0x1'/> > </controller> > <interface type='bridge'> > <mac address='52:54:00:00:01:50'/> > <source bridge='br0'/> > <target dev='vnet0'/> > <model type='virtio'/> > <alias name='net0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x0'/> > </interface> > <serial type='pty'> > <source path='/dev/pts/1'/> > <target port='0'/> > <alias name='serial0'/> > </serial> > <console type='pty' tty='/dev/pts/1'> > <source path='/dev/pts/1'/> > <target type='serial' port='0'/> > <alias name='serial0'/> > </console> > <input type='mouse' bus='ps2'/> > <graphics type='vnc' port='5900' autoport='yes'/> > <video> > <model type='cirrus' vram='9216' heads='1'/> > <alias name='video0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x0'/> > </video> > </devices> > <seclabel type='none'/> > </domain> > > The output of virsh capabilities: > > <capabilities> > > <host> > <uuid>44454c4c-4600-1031-8043-b9c04f47354a</uuid> > <cpu> > <arch>x86_64</arch> > <model>SandyBridge</model> > <vendor>Intel</vendor> > <topology sockets='1' cores='8' threads='1'/> > <feature name='pdpe1gb'/> > <feature name='osxsave'/> > <feature name='tsc-deadline'/> > <feature name='dca'/> > <feature name='pdcm'/> > <feature name='xtpr'/> > <feature name='tm2'/> > <feature name='est'/> > <feature name='smx'/> > <feature name='vmx'/> > <feature name='ds_cpl'/> > <feature name='monitor'/> > <feature name='dtes64'/> > <feature name='pbe'/> > <feature name='tm'/> > <feature name='ht'/> > <feature name='ss'/> > <feature name='acpi'/> > <feature name='ds'/> > <feature name='vme'/> > </cpu> > <power_management> > <suspend_disk/> > </power_management> > <migration_features> > <live/> > <uri_transports> > <uri_transport>tcp</uri_transport> > </uri_transports> > </migration_features> > <topology> > <cells num='2'> > <cell id='0'> > <cpus num='8'> > <cpu id='0'/> > <cpu id='2'/> > <cpu id='4'/> > <cpu id='6'/> > <cpu id='8'/> > <cpu id='10'/> > <cpu id='12'/> > <cpu id='14'/> > </cpus> > </cell> > <cell id='1'> > <cpus num='8'> > <cpu id='1'/> > <cpu id='3'/> > <cpu id='5'/> > <cpu id='7'/> > <cpu id='9'/> > <cpu id='11'/> > <cpu id='13'/> > <cpu id='15'/> > </cpus> > </cell> > </cells> > </topology> > </host> > > <guest> > <os_type>hvm</os_type> > <arch name='i686'> > <wordsize>32</wordsize> > <emulator>/usr/libexec/qemu-kvm</emulator> > <machine>rhel6.3.0</machine> > <machine canonical='rhel6.3.0'>pc</machine> > <machine>rhel6.2.0</machine> > <machine>rhel6.1.0</machine> > <machine>rhel6.0.0</machine> > <machine>rhel5.5.0</machine> > <machine>rhel5.4.4</machine> > <machine>rhel5.4.0</machine> > <domain type='qemu'> > </domain> > <domain type='kvm'> > <emulator>/usr/libexec/qemu-kvm</emulator> > </domain> > </arch> > <features> > <cpuselection/> > <deviceboot/> > <pae/> > <nonpae/> > <acpi default='on' toggle='yes'/> > <apic default='on' toggle='no'/> > </features> > </guest> > > <guest> > <os_type>hvm</os_type> > <arch name='x86_64'> > <wordsize>64</wordsize> > <emulator>/usr/libexec/qemu-kvm</emulator> > <machine>rhel6.3.0</machine> > <machine canonical='rhel6.3.0'>pc</machine> > <machine>rhel6.2.0</machine> > <machine>rhel6.1.0</machine> > <machine>rhel6.0.0</machine> > <machine>rhel5.5.0</machine> > <machine>rhel5.4.4</machine> > <machine>rhel5.4.0</machine> > <domain type='qemu'> > </domain> > <domain type='kvm'> > <emulator>/usr/libexec/qemu-kvm</emulator> > </domain> > </arch> > <features> > <cpuselection/> > <deviceboot/> > <acpi default='on' toggle='yes'/> > <apic default='on' toggle='no'/> > </features> > </guest> > > </capabilities> > > The output of virsh vcpuinfo master-data05-v50: > > VCPU: 0 > CPU: 0 > State: running > CPU time: 119.4s > CPU Affinity: y--------------- > > VCPU: 1 > CPU: 2 > State: running > CPU time: 78.9s > CPU Affinity: --y------------- > > VCPU: 2 > CPU: 4 > State: running > CPU time: 74.4s > CPU Affinity: ----y----------- > > VCPU: 3 > CPU: 6 > State: running > CPU time: 61.6s > CPU Affinity: ------y--------- > > VCPU: 4 > CPU: 8 > State: running > CPU time: 91.4s > CPU Affinity: --------y------- > > VCPU: 5 > CPU: 10 > State: running > CPU time: 67.4s > CPU Affinity: ----------y----- > > VCPU: 6 > CPU: 12 > State: running > CPU time: 52.8s > CPU Affinity: ------------y--- > > VCPU: 7 > CPU: 14 > State: running > CPU time: 66.8s > CPU Affinity: --------------y- > > I also checked x2apic using 'grep -i x2apic /var/log/messages*'. this > shows x2apic is not working on the host but it is working on the guest. > Should I enable it on the host as well as suggested by the output? > > On the host I get: > > /var/log/messages-20130210:Feb 8 08:51:00 localhost kernel: Your BIOS has > requested that x2apic be disabled. > /var/log/messages-20130210:Feb 8 08:51:00 localhost kernel: Use > 'intremap=no_x2apic_optout' to override BIOS request. > /var/log/messages-20130210:Feb 8 08:51:00 localhost kernel: x2apic not > enabled, IRQ remapping is in xapic mode > > On the guest I get: > > /var/log/messages-20130213:Feb 13 02:00:19 localhost kernel: Setting APIC > routing to physical x2apic > /var/log/messages-20130213:Feb 13 02:08:11 localhost kernel: Enabling > x2apic > /var/log/messages-20130213:Feb 13 02:08:11 localhost kernel: Enabled > x2apic > /var/log/messages-20130213:Feb 13 02:08:11 localhost kernel: Setting APIC > routing to physical x2apic > >> >> David >> > > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
