Hi, do you have tried to enable cache=writeback when you use librbd ? Could be interesting to see performance with using /dev/rbd0 in your vm, instead mounting a qcow2 inside. ----- Mail original ----- De: "Bill WONG" <wongahshuen@xxxxxxxxx> À: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> Envoyé: Vendredi 28 Octobre 2016 10:24:50 Objet: RBD Block performance vs rbd mount as filesystem Hi All, we have build a Ceph with 72 OSDs, replica 3, all working fine. we have done some performance testing. we found a very interesting issue. we have a KVM + Libvirt + Ceph setup Case 1. KVM + Libvirt + Ceph w/ rbd backend The kvm hypervisor node create a VM and use rbd block device as storage backend. we do dd fdatasync, it got ~500MB/s Case 2. KVM + Libvirt + Ceph The KVM hypervisor node mount the Ceph storage pool directly as local partition eg. mount /dev/rbd0 /mnt/VM_pool, we we create VM with format qcow2 and place the vm disk under that partition, and do the same dd with fdatasync, it got ~850MB/s it is tested within same hypervisor node with same VM configurations. why the directory mount rbd0 to hypervisor filesystem as partition will be much good performance?..any idea on this ? thank you! ---- KVM Ceph VM disk setting -- <disk type='network' device='disk'> <source protocol='rbd' name='VM_pool/VM1.img' > <host name='mon1' port='6789'/> <host name='mon2' port='6789'/> <host name='mon3' port='6789'/> </source> <auth username='libvirt' type='ceph'> <secret type='ceph' uuid='856b660e-ce4e-4a91-a7be-f17e469024c5'/> </auth> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> -- KVM VM disk created at Ceph rbd0 partition <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/mnt/VM_Pool/CentOS1.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </disk> ================================= Bill _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |