> Hi, > > > Correct, but that's the theoretical maximum I was referring to. If I calculate > that I should be able to get 50MB/second then 30MB/second is acceptable > but 500KB/second is not :) > > I have written a small benchmark for RBD : > > https://gist.github.com/smunaut/5433222 > > It uses the librbd API directly without kernel client and queue > requests long in advance and this should give an "upper" bound to what > you can get at best. > It reads and writes the whole image, so I usually just create a 1 or 2 > G image for testing. > > Using two OSDs on two distinct recent 7200rpm drives (with journal on > the same disk as data), I get : > > Read: 89.52 Mb/s (2147483648 bytes in 22877 ms) > Write: 10.62 Mb/s (2147483648 bytes in 192874 ms) > I like your benchmark tool! How many replicas? With two OSD's with xfs on ~3yo 1TB disks with two replicas I get: # ./a.out admin xen test Read: 111.99 Mb/s (1073741824 bytes in 9144 ms) Write: 29.68 Mb/s (1073741824 bytes in 34507 ms) Which means I forgot to drop caches on the OSD's so I'm seeing the limit on my public network (single gigabit interface). After dropping caches I consistently get: # ./a.out admin xen test Read: 39.98 Mb/s (1073741824 bytes in 25614 ms) Write: 23.11 Mb/s (1073741824 bytes in 44316 ms) Journal is on the same disk. Network is... confusing :) but is basically public on a single gigabit and cluster on a bonded pair of gigabit links. The whole network thing is shared with my existing drbd cluster so performance may vary over time. My read speed is consistently around 40MB/second, and my write speed is consistently around 22MB/second. I had expected better of read... While running, iostat on each osd reports a read rate of around 20MB/second (1/2 total on each) during read test and a rate of 40-60MB/second (~2x total on each) during write test, which is pretty much exactly right. iperf on the cluster network (pair of gigabits bonded) gives me about 1.97Gbits/second. iperf between osd and client is around 0.94Gbits/second. changing the scheduler on the harddisk doesn't seem to make any difference, even when I set it to cfq which normally really sucks. What ceph version are you using and what filesystem? Thanks James -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
