Le vendredi 10 mai 2013 à 19:16 +0200, Greg a écrit : > Hello folks, > > I'm in the process of testing CEPH and RBD, I have set up a small > cluster of hosts running each a MON and an OSD with both journal and > data on the same SSD (ok this is stupid but this is simple to verify the > disks are not the bottleneck for 1 client). All nodes are connected on a > 1Gb network (no dedicated network for OSDs, shame on me :). > > Summary : the RBD performance is poor compared to benchmark > > A 5 seconds seq read benchmark shows something like this : > > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > > 0 0 0 0 0 0 - 0 > > 1 16 39 23 91.9586 92 0.966117 0.431249 > > 2 16 64 48 95.9602 100 0.513435 0.53849 > > 3 16 90 74 98.6317 104 0.25631 0.55494 > > 4 11 95 84 83.9735 40 1.80038 0.58712 > > Total time run: 4.165747 > > Total reads made: 95 > > Read size: 4194304 > > Bandwidth (MB/sec): 91.220 > > > > Average Latency: 0.678901 > > Max latency: 1.80038 > > Min latency: 0.104719 > > 91MB read performance, quite good ! > > Now the RBD performance : > > root@client:~# dd if=/dev/rbd1 of=/dev/null bs=4M count=100 > > 100+0 records in > > 100+0 records out > > 419430400 bytes (419 MB) copied, 13.0568 s, 32.1 MB/s > > There is a 3x performance factor (same for write: ~60M benchmark, ~20M > dd on block device) > > The network is ok, the CPU is also ok on all OSDs. > CEPH is Bobtail 0.56.4, linux is 3.8.1 arm (vanilla release + some > patches for the SoC being used) > > Can you show me the starting point for digging into this ? > > Thanks! > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > You should try to increase read_ahead to 512K instead of the defaults 128K (/sys/block/*/queue/read_ahead_kb). I have seen a huge difference on reads with that. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |