Hi Shain,
Thanks for you advice.
After a rebuild. I drop the SSD as journal disk. The performance is as my expectation now.
That means to have a single journal SSD for 10 HDD is really not a good idea.
Appreciate.
Hugo
2013/12/19 Shain Miley <SMiley@xxxxxxx>
Three things come to my mind when looking at your setup/results:
1)The number of pg's. According to the documentation it should be:
(number of OSD * 100 ) / number of replicas.
Maybe playing with the number a bit would yield better results.
2) Although I am NOT using SSD's as journals, I am using 10GigE with 6 nodes and 72 OSD's, and I am able to get around 1100 MB/s using rados bench and the default 4 MB object size, the default number of threads (16) and 1 replica.
It seems like the consensus is that you should front around 4 to 6 OSD's total per SSD in order to get the best performance. Also you need to make sure that the SSD's that you are using can actually handle the combined write throughput from 10 OSD's.
Otherwise the SSD's become the bottleneck.
3) What is the output if you just use the default number of threads (instead of 100)?
For example:
'rados bench -p test3 30 write'
Shain
Shain Miley | Manager of Systems and Infrastructure, Digital Media | smiley@xxxxxxx | 202.513.3649
________________________________________
From: ceph-users-bounces@xxxxxxxxxxxxxx [ceph-users-bounces@xxxxxxxxxxxxxx] on behalf of Kuo Hugo [tonytkdk@xxxxxxxxx]
Sent: Thursday, December 19, 2013 7:17 AM
To: ceph-users@xxxxxxxxxxxxxx
Subject: [Rados Bench Result] Maximum bandwidth only 380MB/sec with 10Gb Network.
Hi folks,
I'm doing a test for Rados bench now. The cluster is deployed by ceph-deploy
Ceph version: Emperor
FS : XFS
I created a pool test3 with size1 :
pool 13 'test3' rep size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 2000 pgp_num 2000 last_change 166 owner 0
The Rados bench command issued from an independent benchmark machine.
Public network : 10GbE
Cluster network : 10GbE
Result : rados bench -p test3 30 write -t 100
Total time run: 30.275789
Total writes made: 2876
Write size: 4194304
Bandwidth (MB/sec): 379.974
Stddev Bandwidth: 90.2077
Max bandwidth (MB/sec): 420
Min bandwidth (MB/sec): 0
Average Latency: 1.03858
Stddev Latency: 0.112756
Max latency: 2.20545
Min latency: 0.10155
Why the bandwidth only 379MB/sec? I expected the speed suppose to be over 900MB/sec due to the pool size only 1.
There're 3 storage nodes here.
10 OSDs on each storage node + journals on a SSD with 10 partitions.
Any suggestions?
Thanks
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
