How to use libradostriper to improve I/O bandwidth?

[Jialin Liu <jalnliu@xxxxxxx>]

Hello Ceph Community, 

I used libradosstriper api to test the striping feature, it doesn't seem to improve the performance at all, can anyone advise what's wrong with my settings:

The rados object store  testbed at my center has

osd: 48

oss: 4

monitor:2

pg number: 1024

replicated size: 3

I have implemented a benchmark code  with libradosstriper api. 

I then used 1 processes 1 thread to do the test,  by varying a few settings:

• stripe count from 1 to 48, 
• and object size from 1MB to 128 MB (with stripe size 1MB, stripe size needs to be smaller than the rados object size), 
• and file size from 100MB to 1.6GB, 

The peak bandwidth among all tests is only 130MB/s, no difference in different tests. 

 I suspect that the IO got serialized in the rados layer, with some uncertain evidence in the libradosstriper source code (note the for loop): 

...
·Striper::file_to_extents(cct(), format.c_str(), &l, off, len, 0, extents);
·for (vector<ObjectExtent>::iterator p = extents.begin(); p != extents.end(); ++p) {
r = m_ioCtx.aio_write(p->oid.name, rados_completion, oid_bl,
                                        p->length, p->offset);
             }
...

Could you please correct me if I misused or misunderstood any things? Thanks much. 

Best,

Jialin

NERSC/LBNL

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com