Hadoop/Ceph and DFS IO tests

[ker can <kercan74@xxxxxxxxx>]

hi Noah, 

while we're still on the hadoop topic ... I was also trying out the TestDFSIO tests ceph v/s hadoop.  The Read tests on ceph takes about 1.5x the hdfs time.  The write tests are worse about ... 2.5x the time on hdfs, but I guess we have additional journaling overheads for the writes on ceph. But there should be no such overheads for the read  ?

Is this something that you are already aware of ?   [ by the way these results are after applying the data locality patches ]

13/07/09 12:16:52 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read (For CephFS)
13/07/09 12:16:52 INFO fs.TestDFSIO:            Date & time: Tue Jul 09 12:16:52 PDT 2013
13/07/09 12:16:52 INFO fs.TestDFSIO:        Number of files: 300
13/07/09 12:16:52 INFO fs.TestDFSIO: Total MBytes processed: 458070
13/07/09 12:16:52 INFO fs.TestDFSIO:      Throughput mb/sec: 27.356665605451205
13/07/09 12:16:52 INFO fs.TestDFSIO: Average IO rate mb/sec: 30.80777931213379
13/07/09 12:16:52 INFO fs.TestDFSIO:  IO rate std deviation: 10.731377923459378
13/07/09 12:16:52 INFO fs.TestDFSIO:     Test exec time sec: 388.969
13/07/09 12:16:52 INFO fs.TestDFSIO:


13/07/09 04:50:34 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read   (For HDFS)
13/07/09 04:50:34 INFO fs.TestDFSIO:            Date & time: Tue Jul 09 04:50:34 PDT 2013
13/07/09 04:50:34 INFO fs.TestDFSIO:        Number of files: 300
13/07/09 04:50:34 INFO fs.TestDFSIO: Total MBytes processed: 456388
13/07/09 04:50:34 INFO fs.TestDFSIO:      Throughput mb/sec: 40.62858541727874
13/07/09 04:50:34 INFO fs.TestDFSIO: Average IO rate mb/sec: 48.635948181152344
13/07/09 04:50:34 INFO fs.TestDFSIO:  IO rate std deviation: 27.46651216689178
13/07/09 04:50:34 INFO fs.TestDFSIO:     Test exec time sec: 270.682
13/07/09 04:50:34 INFO fs.TestDFSIO:

Thanks

KC

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com