On Mon, Mar 21, 2011 at 5:28 PM, Ben Nagy <ben@xxxxxxxx> wrote: > Bit of a status update: [...] > Since the underlying configs have changed so much, I will set up, run > all the tests again and then followup if things are still locky. Which they are. I will need to get the git sources in order to build with disable-strip as Avi suggested, which I can start doing tomorrow, but in the meantime, I brought up 96 guests. Command: 00:02:47 /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name fbn-95 -uuid 2c7bfda3-525f-9a4f-901c-7e139dfc236f -nographic -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/fbn-95.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=localtime -boot cd -drive file=/mnt/big/bigfiles/kvm_disks/eax/fbn-95.ovl,if=none,id=drive-virtio-disk0,boot=on,format=qed -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-0-1,readonly=on,format=raw -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=16,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:01:41:1c,bus=pci.0,addr=0x3 -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -snapshot perf report output (huge, sorry) http://paste.ubuntu.com/583311/ perf top output http://paste.ubuntu.com/583313/ lock_stat contents http://paste.ubuntu.com/583314/ The guests are xpsp3, ACPI Uniprocessor HAL, no /usepmtimer switch anymore (standard boot.ini), fully updated, with version 1.1.16 of the redhat virtio drivers for NIC and block Major changes I made - disabled usb - changed from cirrus to std-vga - changed .ovl format to qed - changed backing disk format to raw The machines now boot much MUCH faster, I can bring up 96 machines in a few minutes instead of hours, but the system is at around 43% system time, doing nothing, and each guest reports ~25% CPU from top. If anybody can see anything from the stripped perf report output that was be great, or if anyone has any other suggestions as to things I can try to debug, I'm all ears. :) My existing plans for further investigation are to build with disable-strip and retest, and also to test with similarly configured linux guests. Thanks a lot, ben -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html