Avi Kivity wrote:
On 05/20/2012 08:29 PM, Erik Brakkee wrote:
Avi Kivity wrote:
On 05/20/2012 08:02 PM, Erik Brakkee wrote:
[...]
Thanks for this information. Unfortunately, io="native" in domain.xml
is not supported by opensuse 11.3. It is supported in 12.1 so it
appears that the version of KVM I have on the server is too old. I
tried it on a system running the newer version and indeed, as you say
the load disappears completely when using io="native".
I am going to update the host now (probably to centos 6.2) to get rid
of this problem.
To be clear: it's not a problem. It's completely normal, and doesn't
affect anything.
The only problem with it is that it leads to high workload spikes,
which is normally a reason to have a good look at what is going on. In
this case, the newer version of KVM should help eliminate these
spikes, so that the next time I see a spike in the workload I know
that I have to look into something.
In the mean time I have migrated the host machine to run centos 6.2 and
later 6.3. There I could use the aio=native option together with
cache=none and I saw a significant drop in thw workload while doing a
lot of IO on the guest.
However, after upgrading to centos 6.4, I now see huge workload spikes
again, but the virtual machine configuration is the same. Also, I am
seeing a high usage of swap on the host machine. It almost looks as if
the aio=native option is being ignored or could it be a different problem?
Here is one of the processes:
qemu 3026 4.6 17.9 4751536 4419208 ? Sl Mar27 136:21
/usr/libexec/qemu-kvm -name sparrow -S -M rhel6.2.0 -enable-kvm -m 4096
-smp 2,sockets=2,cores=1,threads=1 -uuid
1389eb3f-8f26-686e-93c4-21267a66ec52 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/sparrow.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/dev/bootdisks/sparrow,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/dev/sparrow/disk,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=26 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:01:01:03,bus=pci.0,addr=0x5
-netdev tap,fd=27,id=hostnet1,vhost=on,vhostfd=28 -device
virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:01:01:04,bus=pci.0,addr=0x7
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:2 -vga cirrus
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
I can also see that the swap space is exclusively used by qemu-kvm:
[root@falcon bin]# ./examineswap
...
Overall swap used: 1264924 kB
========================================
kB pid name
========================================
575092 3026 qemu-kvm
391468 2882 qemu-kvm
298364 2953 qemu-kvm
(see at the end of the examineswap script).
This snapshot of the swap space is taken shortly after enabling swap and
it continues to fill up the entire swap space, with still a lot of
memory free
[root@falcon bin]# free
total used free shared buffers cached
Mem: 24662972 24460920 202052 0 10904048 40712
-/+ buffers/cache: 13516160 11146812
Swap: 2097144 1508516 588628
Also, I have set the swappiness to 0 but it does not help:
[root@falcon bin]# sysctl vm.swappiness
vm.swappiness = 0
Details on the OS and KVM:
[root@falcon ~]# uname -a
Linux falcon.fritz.box 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Mar 13
00:26:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[root@falcon ~]# cat /etc/issue
CentOS release 6.4 (Final)
Kernel \r on an \m
[root@falcon ~]# rpm -qa | grep kvm
qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.2.x86_64
I am also getting a workload on the host of approximately 20 now whereas
the workload on the VM doing the IO is just 6. What I am doing at this
time is creating a backup over iSCSI with a target on the physical host
and an initiator on the VM. The VM has created snapshot logical volumes
and is copying data to the iSCSI target. More spcifically, I am using
this script that I developed myself: http://wamblee.org/snapshot.html
Do you have any idea what this could be?
Cheers
Erik
PS. The examineswap script I downloaded
#!/bin/bash
# find-out-what-is-using-your-swap.sh
# -- Get current swap usage for all running processes
# --
# -- rev.0.3, 2012-09-03, Jan Smid - alignment and
intendation, sorting
# -- rev.0.2, 2012-08-09, Mikko Rantalainen - pipe the output to
"sort -nk3" to get sorted output
# -- rev.0.1, 2011-05-27, Erik Ljungstrom - initial version
SCRIPT_NAME=`basename $0`;
SORT="kb"; # {pid|kB|name} as first parameter, [default: kb]
[ "$1" != "" ] && { SORT="$1"; }
[ ! -x `which mktemp` ] && { echo "ERROR: mktemp is not available!"; exit; }
MKTEMP=`which mktemp`;
TMP=`${MKTEMP} -d`;
[ ! -d "${TMP}" ] && { echo "ERROR: unable to create temp dir!"; exit; }
>${TMP}/${SCRIPT_NAME}.pid;
>${TMP}/${SCRIPT_NAME}.kb;
>${TMP}/${SCRIPT_NAME}.name;
SUM=0;
OVERALL=0;
echo "${OVERALL}" > ${TMP}/${SCRIPT_NAME}.overal;
for DIR in `find /proc/ -maxdepth 1 -type d -regex "^/proc/[0-9]+"`;
do
PID=`echo $DIR | cut -d / -f 3`
PROGNAME=`ps -p $PID -o comm --no-headers`
for SWAP in `grep Swap $DIR/smaps 2>/dev/null| awk '{ print $2 }'`
do
let SUM=$SUM+$SWAP
done
if (( $SUM > 0 ));
then
echo -n ".";
echo -e "${PID}\t${SUM}\t${PROGNAME}" >> ${TMP}/${SCRIPT_NAME}.pid;
echo -e "${SUM}\t${PID}\t${PROGNAME}" >> ${TMP}/${SCRIPT_NAME}.kb;
echo -e "${PROGNAME}\t${SUM}\t${PID}" >>
${TMP}/${SCRIPT_NAME}.name;
fi
let OVERALL=$OVERALL+$SUM
SUM=0
done
echo "${OVERALL}" > ${TMP}/${SCRIPT_NAME}.overal;
echo;
echo "Overall swap used: ${OVERALL} kB";
echo "========================================";
case "${SORT}" in
name )
echo -e "name\tkB\tpid";
echo "========================================";
cat ${TMP}/${SCRIPT_NAME}.name|sort -r;
;;
kb )
echo -e "kB\tpid\tname";
echo "========================================";
cat ${TMP}/${SCRIPT_NAME}.kb|sort -rh;
;;
pid | * )
echo -e "pid\tkB\tname";
echo "========================================";
cat ${TMP}/${SCRIPT_NAME}.pid|sort -rh;
;;
esac
rm -fR "${TMP}/";
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
