Re: [Ceph] 10Gb network support

Kuo Hugo <tonytkdk@xxxxxxxxx> · Fri, 13 Sep 2013 10:02:50 +0800

Hi Gregory, 

For the full command of rados bench like this : 
$> rados bench 100  write -t 100 -p .rgw.buckets --block-size 40485760

The network bandwidth between rados client to OSDs is 10Gb,  192.168.2.51 --> 192.168.2.61 (one of storage node includes 10 OSDs)

[  3] local 192.168.2.51 port 52256 connected with 192.168.2.61 port 5001

[  3]  0.0-10.0 sec  10.7 GBytes  9.19 Gbits/sec 

Would you please to explain more about *lower-level* tests ?  is it something like single disk I/O performance test in your mind?  If so, no.  But it's impossible that all drives are fucked-up tho. 

In my knowledge of the data path of Rados, When I upload an object from the Rados Client (assumes replicas=3).  It should be : 
Rados Client --> MON.0 (get CRUSH map) --> Three of 30 OSDs

Ideally, to upload 10~20 objects simultaneously could filled up all the bandwidth of 10Gb network. 

Thanks

+Hugo Kuo+
(+886) 935004793

2013/9/13 Gregory Farnum <greg@xxxxxxxxxxx>

What command did you use to get those results? Have you tried increasing parallelism? What bandwidth do you have between that machine and your OSDs? Have you run lower-level tests on individual disks and nodes to make sure they're performing as you expect?

-GregSoftware Engineer #42 @ http://inktank.com | http://ceph.com

On Thu, Sep 12, 2013 at 7:47 AM, Kuo Hugo <tonytkdk@xxxxxxxxx> wrote:

Hi folks, 

I deployed a Ceph cluster with 10Gb network devices. But the max bandwidth usage only 100MB/sec
Do I need to enable or setup anything for 10Gb support ? 

My Rados Bench 

Total time run:         101.265252
Total writes made:      236

Write size:             40485760
Bandwidth (MB/sec):     89.982

Stddev Bandwidth:       376.238

Max bandwidth (MB/sec): 3822.41
Min bandwidth (MB/sec): 0

Average Latency:        33.9225
Stddev Latency:         12.8661

Max latency:            43.6013
Min latency:            1.03948

I check the network bandwidth between nodes by iperf . 

[Iperf]

>From BM to RadosGW  
local 192.168.2.51 port 5001 connected with 192.168.2.40 port 39421

0.0-10.0 sec  10.1 GBytes  8.69 Gbits/sec

From RadosGW to Rados nodes

[  3] local 192.168.2.51 port 52256 connected with 192.168.2.61 port 5001

[  3]  0.0-10.0 sec  10.7 GBytes  9.19 Gbits/sec 

[  3] local 192.168.2.51 port 52256 connected with 192.168.2.62 port 5001

[  3]  0.0-10.0 sec  9.2 GBytes  8.1 Gbits/sec 

[  3] local 192.168.2.51 port 51196 connected with 192.168.2.63 port 5001

[  3]  0.0-10.0 sec  10.7 GBytes  9.21 Gbits/sec

All OSDs are listening on 192.168.2.x 

My OSD dump : 

2013-09-12 07:43:42.556501 7f026a66b780 -1 asok(0x1c9d510) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph': (98) Address already in use

epoch 237
fsid 6e05675c-f545-4d88-9784-ea56ceda750e

created 2013-09-06 00:16:50.324070
modified 2013-09-09 19:56:55.395161
flags

pool 0 'data' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45

pool 1 'metadata' rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0
pool 2 'rbd' rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0

pool 13 '.rgw.gc' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 213 owner 18446744073709551615

pool 14 '.rgw.control' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 214 owner 18446744073709551615
pool 15 '.users.uid' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 216 owner 18446744073709551615

pool 16 '.users' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 218 owner 18446744073709551615

pool 17 '.rgw' rep size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 400 pgp_num 400 last_change 231 owner 0
pool 18 '.rgw.buckets' rep size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 400 pgp_num 400 last_change 235 owner 0

pool 19 '.users.swift' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 236 owner 18446744073709551615

max_osd 30
osd.0 up   in  weight 1 up_from 222 up_thru 233 down_at 221 last_clean_interval [202,219) 192.168.2.61:6809/9155 192.168.2.61:6810/9155 192.168.2.61:6811/9155 exists,up e230eb86-8ed4-4ce6-90ef-60197cd4a6ad

.
.
osd.10 up   in  weight 1 up_from 223 up_thru 233 down_at 222 last_clean_interval [70,219) 192.168.2.62:6827/21018 192.168.2.62:6828/21018 192.168.2.62:6829/21018 exists,up 3bfc59f4-e11c-4acf-941d-b8fd0a789ce3

.
.
osd.20 up   in  weight 1 up_from 221 up_thru 233 down_at 220 last_clean_interval [118,219) 192.168.2.63:6801/19185 192.168.2.63:6807/19185 192.168.2.63:6810/19185 exists,up 62833dc4-79b4-4905-8432-a26a8d1aaf06

.
.

+Hugo Kuo+
(+886) 935004793

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com