On 08/05/2014 06:19 PM, Mark Kirkwood wrote: > On 05/08/14 23:44, Mark Nelson wrote: >> On 08/05/2014 02:48 AM, Mark Kirkwood wrote: >>> On 05/08/14 03:52, Tregaron Bayly wrote: >>>> Does anyone have any insight on how we can tune librbd to perform >>>> closer >>>> to the level of the rbd kernel module? >>>> >>>> In our lab we have a four node cluster with 1GbE public network and >>>> 10GbE cluster network. A client node connects to the public network >>>> with 10GbE. >>>> >>>> When doing benchmarks on the client using the kernel module we get >>>> decent performance and can cause the OSD nodes to max out their 1GbE >>>> link at peak servicing the requests: >>>> >>>> tx rx >>>> max 833.66 Mbit/s | 639.44 Mbit/s >>>> max 938.06 Mbit/s | 707.35 Mbit/s >>>> max 846.78 Mbit/s | 702.04 Mbit/s >>>> max 790.66 Mbit/s | 621.92 Mbit/s >>>> >>>> However, using librbd we only get about 30% of performance and I can >>>> see >>>> that it doesn't seem to generate requests fast enough to max out the >>>> links on OSD nodes: >>>> >>>> max 309.74 Mbit/s | 196.77 Mbit/s >>>> max 300.15 Mbit/s | 154.38 Mbit/s >>>> max 263.06 Mbit/s | 154.38 Mbit/s >>>> max 368.91 Mbit/s | 234.38 Mbit/s >>>> >>>> I know that I can play with cache settings to help give the client >>>> better service on hits, but I'm wondering how I can soup up librbd so >>>> that it can take advantage of more of the speed available in the >>>> cluster. It seems like using librbd will leave a lot of the resources >>>> idle. >>> >>> >>> Hi Tregaron, >>> >>> I'm guessing that in the librbd case you are injecting the volume into a >>> VM before running your tests - might be interesting to see your libvirt >>> XML for the VM... in particular the 'cache' setting for the rbd volume. >>> If this are not set or is 'default' then changing to 'none' will >>> probably be significantly faster. In addition adding: >>> >>> io='native' >>> >>> may give a bit of a boost too! >> >> Oh, that reminds me, also make sure to use the virtio bus instead of ide >> or something else. That can make a very large performance difference. >> > > Yes, good point Mark (man this plethora of Marks is confusing...). That > reminds me, we currently have some libvirt configs in the docs that use > > bus='ide' > > ...we should probably weed 'em out - or at least mention that vertio is > the preferred bus (e.g http://ceph.com/docs/master/rbd/libvirt/#summary) ugh, I thought we had gotten rid of all of those. Good catch. Mark > > > Cheers > > Mark > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |