On 01/24/2014 09:31 AM, Christian Kauhaus wrote:
Hi, we're using Ceph to serve VM images via RBD and thus, RBD performance is important for us. I've prepared some write benchmarks using different object sizes. One time I use 'rados bench' directly and the other time 'rbd bench-write'. The results are interesting: Raw RADOS write rates are significantly better for large objects (>128k), RBD performs better for medium sized objects (>16k, <128k), but RBD is really slow for small writes. We have lot of small writes, so this is the pain point. I think latencies are dominant here. Our test setup consists of two Ceph servers running a MON and 9 OSDs (one OSD daemon per disk; ext4 filesystem) with journals on a shared SSD (one SSD partition per OSD). There are 2 GigE networks (storage frontend/backend) with approx 62µs RTT and jumbo frames enabled. See attached ceph.conf further details. Some parameters there are taken from the tuning recommendations at [1]. Note that I have to stick to ext4 on the OSDs. Is there anything we can do to improve latencies? I don't know where to start: * OSD setup? * Network setup? * ceph.conf parameter tuning? * Separate MONs? * Separate networks for MON access? A lot of options... so I would be grateful for hints what is worth looking at.
Are you using RBD cache? That can help quite a bit for small writes. More generally: How many IOPs per disk are you getting now and what are your goals? Like you said, there are many many different things to look at, and IOPS is often trickier to tune for.
Please refer to bitbucket[2] for benchmark scripts. TIA Christian [1] http://ceph.com/community/ceph-bobtail-jbod-performance-tuning/ [2] https://bitbucket.org/ckauhaus/ceph_performance _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
