> > You mentioned using qcow; if you're using qcow, than the userspace > qemu/kvm process will need to do its own locking to manage its own > space management file. > I tried raw files too, there is not much difference in performance. > In general, if you need to do allocation management, either because > you are writing to a sparse file, and ext4 has to do block allocation, > or if you are writing to a fallocated file, and ext4 has to keep track > of whether a block has been written to that location before (and if > not, do a journalled transaction to clear the unwritten bit), there > will be extra work that has to be done at the qcow or ext4 layer that > doesn't have to be done at the LVM layer. Of course, this work is > providing extra services (such as space management and/or not > revealing previously written block contents from another customer to > your current customer's VM, which might make your local data > protection authorities cranky). > The performance results which I posted in my original post are after all the block allocation is done. In other words: on the testing machine we do a first fio run which writes and allocates the testing file (during this first run the fio performance is actually much worse that what I reported, single VM performance is about 700 IOPS) and then we do second fio run and we take the benchmark numbers from this second run. > So the devil is very much in the details of how you set up the > hypervisor, and "except for the argument for the virtual machine disk" > is leaving an awful lot unspecified. Whether you preallocated and > pre-zeroed the file can make a difference, whether you are using > buffered or direct I/O based on the cache parameter makes a > difference, etc., etc. Here is the command line of the qemu when using raw files on EXT4: usr/bin/qemu-system-x86_64 -name instance-00000327 -S -machine pc-i440fx-utopic,accel=kvm,usb=off -cpu Haswell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -object memory-backend-ram,size=2048M,id=ram-node0,host-nodes=1,policy=bind -numa node,nodeid=0,cpus=0,memdev=ram-node0 -uuid 9426ece6-6c1f-403d-a7bf-fa8fb975b321 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2015.1.2,serial=31333937-3136-5a43-4a35-333130485632,uuid=9426ece6-6c1f-403d-a7bf-fa8fb975b321 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000327.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/9426ece6-6c1f-403d-a7bf-fa8fb975b321/disk,if=none,id=drive-virtio-disk0,format=raw,cache=none,iops_rd=20000,iops_wr=20000 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=44 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:ea:40:eb,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/9426ece6-6c1f-403d-a7bf-fa8fb975b321/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:16 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on Here is the command line when using LVM: /usr/bin/qemu-system-x86_64 -name instance-0000033b -S -machine pc-i440fx-utopic,accel=kvm,usb=off -cpu Haswell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -object memory-backend-ram,size=2048M,id=ram-node0,host-nodes=0,policy=bind -numa node,nodeid=0,cpus=0,memdev=ram-node0 -uuid d84d39c7-beba-4895-83ac-d2718fdee3f3 -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2015.1.2,serial=31333937-3136-5a43-4a35-333130485632,uuid=d84d39c7-beba-4895-83ac-d2718fdee3f3 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000033b.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/ssdGroup1/d84d39c7-beba-4895-83ac-d2718fdee3f3_disk,if=none,id=drive-virtio-disk0,format=raw,cache=none,iops_rd=20000,iops_wr=20000 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:4c:1a:66,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/d84d39c7-beba-4895-83ac-d2718fdee3f3/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on root 18449 0.0 0.0 11748 2184 pts/3 S+ 15:09 0:00 grep --color=auto qemu The virtual disk part for raw file: -drive file=/var/lib/nova/instances/9426ece6-6c1f-403d-a7bf-fa8fb975b321/disk,if=none,id=drive-virtio-disk0,format=raw,cache=none,iops_rd=20000,iops_wr=20000 The virtual disk part for LVM: -drive file=/dev/ssdGroup1/d84d39c7-beba-4895-83ac-d2718fdee3f3_disk,if=none,id=drive-virtio-disk0,format=raw,cache=none,iops_rd=20000,iops_wr=20000 Cheers, Prema -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
