Mark, thanks for your reply. I've tried increasing the readahead values to 4M (8192) for all osds as well as all vm block devices. This did not change performance much, the osd values are still in the region of 30 MB/s. All vms are using virtio drivers and I've got 3.8.0 kernel on the osd servers from Ubuntu repo. The osd fragmentation level of zfs is at 8% at the moment, not sure if this should impact the performance by this much. I will defrag it over night and check tomorrow to see if it makes the difference. Cheers Andrei ----- Original Message ----- From: "Mark Nelson" <mark.nelson@xxxxxxxxxxx> To: ceph-users@xxxxxxxxxxxxxx Sent: Tuesday, 25 March, 2014 12:27:21 PM Subject: Re: rbd + qemu osd performance On 03/25/2014 07:19 AM, Andrei Mikhailovsky wrote: > Hello guys, > > Was hoping someone could help me with strange read performance problems > on osds. I have a test setup of 4 kvm host servers which are running > about 20 test linux vms between them. The vms' images are stored in ceph > cluster and accessed via rbd. I also have 2 osd servers with replica of > 2. The physical specs are at the end of the email. > > I've been running some test to check concurrent read/write performance > from all vms. > > I've simply concurrently fired the following tests on each vm: > > "dd if=/dev/vda of=/dev/null bs=1M count=1000 iflag=direct" and after that > "dd if=/dev/vda of=/dev/null bs=4M count=1000 iflag=direct" > > > While the tests are running i've fired up iostat to monitor osd > performance and load. I've noticed that during the read tests each osd > is reading only about 25-30MB/s, which seems very poor to me. The osd > devices are capable of reading around 150-160MB/s when I fire up dd on > them. I am aware that osds "loose" about 40-50% throughput, so I was > expecting to see each osd doing around 70-80MB/s during my tests. So, > what could be the problem here? I'd first try increasing read ahead on the OSD disks from 128KB to something much larger. Anywhere from 512KB to 4MB may be appropriate. Setting it too large can potentially hurt small random reads, but you'll just have to try it out. You may also want to play with readahead on the RBD devices. Also, make sure you are on kernel 3.8+, and that you are using a virtio driver. Since these are sequential read tests, another thing to watch out for is underlying filesystem fragmentation. That can really wreck havoc. We've especially seen bad behaviour with btrfs when used with RBD for sequential reads. > > I've checked the networking for errors and I can't see any packet drops > or other problems with connectivity. I've also ran various networking > tests for several days bashing servers with high volume of traffic and > I've not had any problems/packet drops/disconnects. > > My infrastructure setup is as follows: > > 1. Software: Ubuntu 12.04.4 with Ceph version 0.72.2, Qemu 1.5 from > Ubuntu Cloud repo > 2. OSD servers: 2xIntel E5-2620 with 12 cores in total, 24GB Ram and 8 > SAS 7k rpm disks each > 3. Networking - QDR 40gbit/s Infiniband using IP over Infiniband. > > Many thanks > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |