rbd read write very slow for heavy I/O operations

[johnu <johnugeorge109@xxxxxxxxx>]

Hi,

     I have an openstack cluster which runs on ceph . I tried running hadoop inside VM's and I noticed that map tasks take long time to complete with time and finally it fails. RDB read/writes are getting slower with time. Is it because of too many objects in ceph per volume?

I have 8 node cluster with 24* 1TB disk for each node.

master : mon

slave1: 1 osd per disk, ie 23

slave 2: 1 osd per disk ie 23
.....
.....

slave 7 : 1 osd per isk  ie 23

replication factor :2

pg nums in default pool: 128

In openstack,I have 14 instances. 14  5TB volumes are created and each one is attached to an instance.I am using default stripe settings.

rpd -p volumes info volume-1

      size 5000GB in 1280000 objects

      order 22(4096kb objects)

1.I couldn't find the documentation for stripe settings which can be used for volume creation  in openstack. Can it be exposed through any configuration files? . (http://ceph.com/docs/master/rbd/rbd-openstack/).  Like 64MB default block size in hdfs, how do we set layout for the objects,?.Can we change it after volume creation ?. Will this affect performance for huge I/O applications like MapReduce?

2. How can RBD caching improve the performance?

3. Like in hdfs which gives priority over localized writes, how can we implement the same feature because rbd volumes are striped across the cluster. I am not sure of crush rulesets which can help this situation

Can someone give me debug points and ideas related to this?. I have not used cephfs for now .

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com