Each node has 48GB RAMs on it though.
+Hugo Kuo+
(+886) 935004793
2013/9/13 Gregory Farnum <greg@xxxxxxxxxxx>
Well, there's obviously something very wrong in your hardware or configuration. Looking at the rados bench results I see a pretty large standard deviation and very high latencies, which leads me to believe that probably something is wrong with your journal. Try running"ceph -w" in one window and then "ceph OSD tell \* bench" and wait and see what results come in.Oh -- did you mean to test with not-quite-40MB objects? That's at least 4GB of RAM on your client node so maybe you've just run it out of memory?-Greg
On Thursday, September 12, 2013, Kuo Hugo wrote:Hi Gregory,
For the full command of rados bench like this :The network bandwidth between rados client to OSDs is 10Gb, 192.168.2.51 --> 192.168.2.61 (one of storage node includes 10 OSDs)$> rados bench 100 write -t 100 -p .rgw.buckets --block-size 40485760Would you please to explain more about *lower-level* tests ? is it something like single disk I/O performance test in your mind? If so, no. But it's impossible that all drives are fucked-up tho.[ 3] local 192.168.2.51 port 52256 connected with 192.168.2.61 port 5001[ 3] 0.0-10.0 sec 10.7 GBytes 9.19 Gbits/sec
In my knowledge of the data path of Rados, When I upload an object from the Rados Client (assumes replicas=3). It should be :Rados Client --> MON.0 (get CRUSH map) --> Three of 30 OSDs
Ideally, to upload 10~20 objects simultaneously could filled up all the bandwidth of 10Gb network.Thanks+Hugo Kuo+2013/9/13 Gregory Farnum <greg@xxxxxxxxxxx>What command did you use to get those results? Have you tried increasing parallelism? What bandwidth do you have between that machine and your OSDs? Have you run lower-level tests on individual disks and nodes to make sure they're performing as you expect?-GregSoftware Engineer #42 @ http://inktank.com | http://ceph.comOn Thu, Sep 12, 2013 at 7:47 AM, Kuo Hugo <tonytkdk@xxxxxxxxx> wrote:Hi folks,I deployed a Ceph cluster with 10Gb network devices. But the max bandwidth usage only 100MB/secDo I need to enable or setup anything for 10Gb support ?My Rados BenchTotal time run: 101.265252Total writes made: 236Write size: 40485760Bandwidth (MB/sec): 89.982Stddev Bandwidth: 376.238Max bandwidth (MB/sec): 3822.41Min bandwidth (MB/sec): 0Average Latency: 33.9225Stddev Latency: 12.8661Max latency: 43.6013Min latency: 1.03948I check the network bandwidth between nodes by iperf .[Iperf]>From BM to RadosGWlocal 192.168.2.51 port 5001 connected with 192.168.2.40 port 394210.0-10.0 sec 10.1 GBytes 8.69 Gbits/secFrom RadosGW to Rados nodes[ 3] local 192.168.2.51 port 52256 connected with 192.168.2.61 port 5001[ 3] 0.0-10.0 sec 10.7 GBytes 9.19 Gbits/sec[ 3] local 192.168.2.51 port 52256 connected with 192.168.2.62 port 5001[ 3] 0.0-10.0 sec 9.2 GBytes 8.1 Gbits/sec[ 3] local 192.168.2.51 port 51196 connected with 192.168.2.63 port 5001[ 3] 0.0-10.0 sec 10.7 GBytes 9.21 Gbits/secAll OSDs are listening on 192.168.2.xMy OSD dump :2013-09-12 07:43:42.556501 7f026a66b780 -1 asok(0x1c9d510) AdminSocketConfigObs
--
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
