<quote who="Gleb Natapov">
> On Thu, Feb 07, 2013 at 04:41:31PM +0100, Erik Brakkee wrote:
>> Hi,
>>
>>
>> We have been benchmarking a java server application (java 6 update 29)
>> that requires a mysql database. The scenario is quite simple. We open a
>> web page which displays a lot of search results. To get the content of
>> the
>> page one big query is done with many smaller queries to retrieve the
>> data.
>> The test from the java side is single threaded.
>>
>> We have used the following deployment scenarios:
>> 1. JBoss in VM, MySql in separate VM
>> 2. JBoss in VM, MySQL native
>> 3. JBoss native, MySQL in vm.
>> 4. JBoss native and MySQL native on the same physical machine
>> 5. JBoss and MySQL virtualized on the same VM.
>>
>> What we see is that the performance (time to execute) is practically the
>> same for all scenarios (approx. 30 seconds), except for scenario 4 that
>> takes approx. 21 seconds. This difference is quite large and contrasts
>> many other test on the internet and other benchmarks we did previously.
>>
>> We have tried pinning the VMs, turning hyperthreading off, varying the
>> CPU
>> model (including host-passthrough), but this did not have any
>> significant
>> impact.
>>
>> The hardware on which we are running is a dual socket E5-2650 machine
>> with
>> 64 GB memory. The server is a Dell poweredge R720 server with SAS disks,
>> RAID controller with battery backup (writeback cache). Transparent huge
>> pages is turned on.
>>
>> We are at a loss to explain the differences in the test. In particular,
>> we
>> would have expected the least performance when both were running
>> virtualized and we would have expected a better performance when JBoss
>> and
>> MySQL were running virtualized in the same VM as compared to JBoss and
>> MySQL both running in different virtual machines. It looks like we are
>> dealing with multiple issues here and not just one.
>>
>> Right now we have a 30% penalty for running virtualized which is too
>> much
>> for us; 10% would be allright. What would you suggest to do to
>> troubleshoot this further?
>>
>
> What is you kernel/qemu versions and command line you are using to start
> a VM?
Centos 6.3, 2.6.32-279.22.1.el6.x86_64
> rpm -qf /usr/libexec/qemu-kvm
qemu-kvm-0.12.1.2-2.295.el6_3.10.x86_64
The guest is also running centos 6.3 with the same settings. Settings that
can influence Java performance (such as transparent huge pages) are turned
on on both the host and guest (see the remark on hugepages below).
The command-line is as follows:
/usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 8192 -mem-prealloc
-mem-path /hugepages/libvirt/qemu -smp 4,sockets=4,cores=1,threads=1 -name
master-data05-v50 -uuid 79ddd84d-937e-357b-8e57-c7f487dc3464 -nodefconfig
-nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/master-data05-v50.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-drive
file=/dev/raid5/v50disk1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
-drive
file=/dev/vg_raid1/v50disk2,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/dev/raid5/v50disk3,if=none,id=drive-virtio-disk2,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk2,id=virtio-disk2
-drive
file=/var/data/images/configdisks/v50/configdisk.img,if=none,id=drive-virtio-disk25,format=raw,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk25,id=virtio-disk25
-netdev tap,fd=21,id=hostnet0,vhost=on,vhostfd=22 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:00:01:50,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
This virtual machine has three virtio disks, and one file based disk. The
last disk is about 100MB in size and is used only during startup (contains
configurationd data for initializing the vm) and is only read and never
written. It has one CDrom which is not used. It also uses old-style
hugepages. Apparently this did not have any significant effect on
performance over transparent hugepages (as would be expected). We
configured these old style hugepages just to rule out any issue with
transparent hugepages.
Initially we got 30% performance penalty with 16 processors, but in the
current setting of using 4 processors we see a reduced performance penalty
of 15-20%. Also on the physical host, we are not running the numad daemon
at the moment. Also, we tried disabling hyperthreading in the host's BIOS
but the measurements do not change significantly.
The IO scheduler on the host and on the guest is CFS. We also tried with
deadline scheduler on the host but this did not make any measurable
difference. We did not try no-op on the host. Additionally, the test is a
read-only test. It queries the database for data. We are excluding the
first measurement, so disk IO cannot be a problem as far as the database
is concerned. For the application (running on JBoss), the IO is basically
limited to network IO of a single web page and it writes a few lines in a
server log file.
This means that we can effectively rule out disk IO as a problem I think,
which limits it to CPU and memory issues. I also epxiermented with the cpu
mode 'host-passthrough' but that also did not result in an improvement.
Other things I could try is to remove the 'memballoon' configuration from
the VM.
Do you have any idea what could be the cause of this? Do you think it is a
numa problem? Memory related perhaps? Do you have any suggestions on
things to configure and/or measure?
Cheers
Erik
>
> --
> Gleb.
>
<domain type='kvm' id='1'>
<name>master-data05-v50</name>
<uuid>79ddd84d-937e-357b-8e57-c7f487dc3464</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement='static'>4</vcpu>
<os>
<type arch='x86_64' machine='rhel6.3.0'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<alias name='ide0-1-0'/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/raid5/v50disk1'/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/vg_raid1/v50disk2'/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/raid5/v50disk3'/>
<target dev='vdc' bus='virtio'/>
<alias name='virtio-disk2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/var/data/images/configdisks/v50/configdisk.img'/>
<target dev='vdz' bus='virtio'/>
<alias name='virtio-disk25'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='ide' index='0'>
<alias name='ide0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:00:01:50'/>
<source bridge='br0'/>
<target dev='vnet0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target port='0'/>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='5900' autoport='yes'/>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</memballoon>
</devices>
<seclabel type='none'/>
</domain>
