On 08/05/2014 02:48 AM, Mark Kirkwood wrote: > On 05/08/14 03:52, Tregaron Bayly wrote: >> Does anyone have any insight on how we can tune librbd to perform closer >> to the level of the rbd kernel module? >> >> In our lab we have a four node cluster with 1GbE public network and >> 10GbE cluster network. A client node connects to the public network >> with 10GbE. >> >> When doing benchmarks on the client using the kernel module we get >> decent performance and can cause the OSD nodes to max out their 1GbE >> link at peak servicing the requests: >> >> tx rx >> max 833.66 Mbit/s | 639.44 Mbit/s >> max 938.06 Mbit/s | 707.35 Mbit/s >> max 846.78 Mbit/s | 702.04 Mbit/s >> max 790.66 Mbit/s | 621.92 Mbit/s >> >> However, using librbd we only get about 30% of performance and I can see >> that it doesn't seem to generate requests fast enough to max out the >> links on OSD nodes: >> >> max 309.74 Mbit/s | 196.77 Mbit/s >> max 300.15 Mbit/s | 154.38 Mbit/s >> max 263.06 Mbit/s | 154.38 Mbit/s >> max 368.91 Mbit/s | 234.38 Mbit/s >> >> I know that I can play with cache settings to help give the client >> better service on hits, but I'm wondering how I can soup up librbd so >> that it can take advantage of more of the speed available in the >> cluster. It seems like using librbd will leave a lot of the resources >> idle. > > > Hi Tregaron, > > I'm guessing that in the librbd case you are injecting the volume into a > VM before running your tests - might be interesting to see your libvirt > XML for the VM... in particular the 'cache' setting for the rbd volume. > If this are not set or is 'default' then changing to 'none' will > probably be significantly faster. In addition adding: > > io='native' > > may give a bit of a boost too! Oh, that reminds me, also make sure to use the virtio bus instead of ide or something else. That can make a very large performance difference. > > Regards > > Mark > > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |