----- Message from Simon Wilson <simon@xxxxxxxxxxxxxxxx> ---------
Date: Fri, 25 Nov 2011 20:34:47 +1000
From: Simon Wilson <simon@xxxxxxxxxxxxxxxx>
Subject: Improving RAID5 write performance in a KVM VM
To: kvm@xxxxxxxxxxxxxxx
Good afternoon list.
I have a CentOS 6 server running six KVM VMs, and am struggling with
poor disk write performance in the VMs.
Setup summary:
Physical server has 2 x 500GB SATA drives in a RAID1 mirror, and 3 x
2TB SATA drives in a RAID5 stripeset. The 2 x 500GB drives host /,
/boot, and the VM .img files.
The .img files are mounted as per the following example in virsh xml:
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/vm/pool/server05.img'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
Each virtual server also has a slice of the RAID5 array for data,
including a CentOS 5.7 Samba server that has a 3TB piece.
The Samba drive is mounted in virsh xml as follows:
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/md2'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</disk>
Mount parameters in the VM are:
/dev/vdb1 /mnt/data ext3 suid,dev,exec,noatime 0 0
When I first built the server performance in the VMs was terribly
bad with constant CPU IOWAIT, but after a lot of searching the
addition of io=native significantly improved it.
It is copying file data to the Samba share that is causing me
issues. Some files I use are many GBs in size. Copies start at
110MBps across the network, but once the Samba server's network copy
cache fills up (the server has 4GB RAM allocated, and the cache
seems to fill after about 2.5 to 3GB copied from the network) they
hit a wall and crawl to 35MBps.
Allocating the Samba server more RAM tricks Windows into thinking
the copy is finished sooner (although it still hits a brick wall
once the larger cache is full), but of course the underlying data
write continues for quite some time after Windows is done.
I have done a fair bit of analysis with dstat and using dd copies,
and file writes onto the VM's RAID5 slice sit between 35 and 40MBps
consistently.
Using Samba on the physical server (not my normal practice, but for
troubleshooting) using a different partition on the same RAID5
array, I get a consistent 75MBps overall write speed.
Reads from the partitions are comparable speed from both physical
and virtual servers (~100MBps).
Are there any hints from anyone please about how I can improve the
write performance in the VM? I realise it will likely never achieve
the bare metal speed, but closer than 50% would be good...
Thanks
Simon.
--
Simon Wilson
M: 0400 12 11 16
Hi all - there's been a lot of activity on the list over the last few
days since I posted this, but no responses to my query.
The Mailing List web page indicates that user support is welcomed in
the list, but if this is not an appropriate channel for getting
assistance could someone please point me in the right direction?
Thanks
Simon.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
