Le 13/05/2013 15:55, Mark Nelson a écrit :
On 05/13/2013 07:26 AM, Greg wrote:
Le 13/05/2013 07:38, Olivier Bonvalet a écrit :
Le vendredi 10 mai 2013 à 19:16 +0200, Greg a écrit :
Hello folks,
I'm in the process of testing CEPH and RBD, I have set up a small
cluster of hosts running each a MON and an OSD with both journal and
data on the same SSD (ok this is stupid but this is simple to
verify the
disks are not the bottleneck for 1 client). All nodes are connected
on a
1Gb network (no dedicated network for OSDs, shame on me :).
Summary : the RBD performance is poor compared to benchmark
A 5 seconds seq read benchmark shows something like this :
sec Cur ops started finished avg MB/s cur MB/s last lat
avg lat
0 0 0 0 0 0 - 0
1 16 39 23 91.9586 92 0.966117
0.431249
2 16 64 48 95.9602 100 0.513435
0.53849
3 16 90 74 98.6317 104 0.25631
0.55494
4 11 95 84 83.9735 40 1.80038
0.58712
Total time run: 4.165747
Total reads made: 95
Read size: 4194304
Bandwidth (MB/sec): 91.220
Average Latency: 0.678901
Max latency: 1.80038
Min latency: 0.104719
91MB read performance, quite good !
Now the RBD performance :
root@client:~# dd if=/dev/rbd1 of=/dev/null bs=4M count=100
100+0 records in
100+0 records out
419430400 bytes (419 MB) copied, 13.0568 s, 32.1 MB/s
There is a 3x performance factor (same for write: ~60M benchmark, ~20M
dd on block device)
The network is ok, the CPU is also ok on all OSDs.
CEPH is Bobtail 0.56.4, linux is 3.8.1 arm (vanilla release + some
patches for the SoC being used)
Can you show me the starting point for digging into this ?
You should try to increase read_ahead to 512K instead of the defaults
128K (/sys/block/*/queue/read_ahead_kb). I have seen a huge difference
on reads with that.
Olivier,
thanks a lot for pointing this out, it indeed makes a *huge*
difference !
# dd if=/mnt/t/1 of=/dev/zero bs=4M count=100
100+0 records in
100+0 records out
419430400 bytes (419 MB) copied, 5.12768 s, 81.8 MB/s
(caches dropped before each test of course)
Mark, this is probably something you will want to investigate and
explain in a "tweaking" topic of the documentation.
Regards,
Out of curiosity, has your rados bench performance improved as well?
We've also seen improvements for sequential read throughput when
increasing read_ahead_kb. (it may decrease random iops in some cases
though!) The reason I didn't think to mention it here though is
because I was just focused on the difference between rados bench and
rbd. It would be interesting to know if rbd has improved more
dramatically than rados bench.
Mark, the read ahead is set on the RBD block device (on the client), so
it doesn't improve benchmark results as the benchmark doesn't use the
block layer.
1 question remains : why did I have poor performance with 1 single
writing thread ?
Regards,
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
