RBD Block performance vs rbd mount as filesystem

[Bill WONG <wongahshuen@xxxxxxxxx>]

Hi All,

we have build a Ceph with 72 OSDs, replica 3, all working fine. we have done some performance testing. we found a very interesting issue.

we have a KVM + Libvirt + Ceph setup

Case 1. KVM + Libvirt + Ceph w/ rbd backend

The kvm hypervisor node create a VM and use rbd block device as storage backend. we do dd fdatasync, it got ~500MB/s

Case 2. KVM + Libvirt + Ceph
The KVM hypervisor node mount the Ceph storage pool directly as local partition eg. mount /dev/rbd0 /mnt/VM_pool, we we create VM with format qcow2 and place the vm disk under that partition, and do the same dd with fdatasync, it got ~850MB/s

it is tested within same hypervisor node with same VM configurations. why the directory mount rbd0 to hypervisor filesystem as partition will be much good performance?..any idea on this ?

thank you!

---- KVM Ceph VM disk setting --

    <disk type='network' device='disk'>

        <source protocol='rbd' name='VM_pool/VM1.img' >

                <host name='mon1' port='6789'/>

                <host name='mon2' port='6789'/>

                <host name='mon3' port='6789'/>

        </source>

        <auth username='libvirt' type='ceph'>

                <secret type='ceph' uuid='856b660e-ce4e-4a91-a7be-f17e469024c5'/>

        </auth>

      <target dev='vda' bus='virtio'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

    </disk>

-- KVM VM disk created at Ceph rbd0 partition

    <disk type='file' device='disk'>

      <driver name='qemu' type='qcow2'/>

      <source file='/mnt/VM_Pool/CentOS1.qcow2'/>

      <target dev='vda' bus='virtio'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

    </disk>

=================================

Bill

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com