My cluster has 3 MON nodes & 6 DATA nodes, all nodes have 2Gbps connectivity (bonding).
Each Data node has 14 SATA HDD (osd), each journal on the same disk as OSD.
Each MON node run RadosGW too.
On Oct 15, 2013 12:34 AM, "Kyle Bader" <kyle.bader@xxxxxxxxx> wrote:
I've personally saturated 1Gbps links on multiple radosgw nodes on a large cluster, if I remember correctly, Yehuda has tested it up into the 7Gbps range with 10Gbps gear. Could you describe your clusters hardware and connectivity?On Mon, Oct 14, 2013 at 3:34 AM, Chu Duc Minh <chu.ducminh@xxxxxxxxx> wrote:Hi sorry, i missed this mail.No, CPU stay the same & very low (< 10%)
> During writes, does the CPU usage on your RadosGW node go way up?When upload small files(300KB/file) over RadosGW:- using 1 process: upload bandwidth ~ 3MB/s- using 100 processes: upload bandwidth ~ 15MB/s
When upload big files(3GB/file) over RadosGW:- using 1 process: upload bandwidth ~ 70MB/s(Therefore i don't upload big files using multi-processes any more :D)
Maybe, RadosGW have a problem when write many smail files. Or it's a problem of CEPH when simultaneously write many smail files into a bucket, that already have millions files?On Wed, Sep 25, 2013 at 7:24 PM, Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote:
On 09/25/2013 02:49 AM, Chu Duc Minh wrote:
I have a CEPH cluster with 9 nodes (6 data nodes & 3 mon/mds nodes)_*Problem*_: i can pass limit 50MB/s when put file over Rados-GW,
And i setup 4 separate nodes to test performance of Rados-GW:
- 2 node run Rados-GW
- 2 node run multi-process put file to [multi] Rados-GW
Result:
a) When i use 1 RadosGW node & 1 upload-node, speed upload = 50MB/s
/upload-node, Rados-GW input/output speed = 50MB/s
b) When i use 2 RadosGW node & 1 upload-node, speed upload = 50MB/s
/upload-node; each RadosGW have input/output = 25MB/s ==> sum
input/ouput of 2 Rados-GW = 50MB/s
c) When i use 1 RadosGW node & 2 upload-node, speed upload = 25MB/s
/upload-node ==> sum output of 2 upload-node = 50MB/s, RadosGW have
input/output = 50MB/s
d) When i use 2 RadosGW node & 2 upload-node, speed upload = 25MB/s
/upload-node ==> sum output of 2 upload-node = 50MB/s; each RadosGW have
input/output = 25MB/s ==> sum input/ouput of 2 Rados-GW = 50MB/s
regardless of the number Rados-GW nodes and upload-nodes.
When i use this CEPH cluster over librados (openstack/kvm), i can easily
achieve > 300MB/s
I don't know why performance of RadosGW is so low. What's bottleneck?
During writes, does the CPU usage on your RadosGW node go way up?
If this is a test cluster, you might want to try the wip-6286 build from our gitbuilder site. There is a fix that depending on the size of your objects, could have a big impact on performance. We're currently investigating some other radosgw performance issues as well, so stay tuned. :)
Mark
Thank you very much!
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Kyle
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
