On Wed, 6 Aug 2014, Mark Nelson wrote: > On 08/05/2014 06:19 PM, Mark Kirkwood wrote: > > On 05/08/14 23:44, Mark Nelson wrote: > > > On 08/05/2014 02:48 AM, Mark Kirkwood wrote: > > > > On 05/08/14 03:52, Tregaron Bayly wrote: > > > > > Does anyone have any insight on how we can tune librbd to perform > > > > > closer > > > > > to the level of the rbd kernel module? > > > > > > > > > > In our lab we have a four node cluster with 1GbE public network and > > > > > 10GbE cluster network. A client node connects to the public network > > > > > with 10GbE. > > > > > > > > > > When doing benchmarks on the client using the kernel module we get > > > > > decent performance and can cause the OSD nodes to max out their 1GbE > > > > > link at peak servicing the requests: > > > > > > > > > > tx rx > > > > > max 833.66 Mbit/s | 639.44 Mbit/s > > > > > max 938.06 Mbit/s | 707.35 Mbit/s > > > > > max 846.78 Mbit/s | 702.04 Mbit/s > > > > > max 790.66 Mbit/s | 621.92 Mbit/s > > > > > > > > > > However, using librbd we only get about 30% of performance and I can > > > > > see > > > > > that it doesn't seem to generate requests fast enough to max out the > > > > > links on OSD nodes: > > > > > > > > > > max 309.74 Mbit/s | 196.77 Mbit/s > > > > > max 300.15 Mbit/s | 154.38 Mbit/s > > > > > max 263.06 Mbit/s | 154.38 Mbit/s > > > > > max 368.91 Mbit/s | 234.38 Mbit/s > > > > > > > > > > I know that I can play with cache settings to help give the client > > > > > better service on hits, but I'm wondering how I can soup up librbd so > > > > > that it can take advantage of more of the speed available in the > > > > > cluster. It seems like using librbd will leave a lot of the resources > > > > > idle. > > > > > > > > > > > > Hi Tregaron, > > > > > > > > I'm guessing that in the librbd case you are injecting the volume into a > > > > VM before running your tests - might be interesting to see your libvirt > > > > XML for the VM... in particular the 'cache' setting for the rbd volume. > > > > If this are not set or is 'default' then changing to 'none' will > > > > probably be significantly faster. In addition adding: > > > > > > > > io='native' > > > > > > > > may give a bit of a boost too! > > > > > > Oh, that reminds me, also make sure to use the virtio bus instead of ide > > > or something else. That can make a very large performance difference. > > > > > > > Yes, good point Mark (man this plethora of Marks is confusing...). That > > reminds me, we currently have some libvirt configs in the docs that use > > > > bus='ide' > > > > ...we should probably weed 'em out - or at least mention that vertio is > > the preferred bus (e.g http://ceph.com/docs/master/rbd/libvirt/#summary) > > ugh, I thought we had gotten rid of all of those. Good catch. BTW, do we still need to use something != virtio in order for trim/discard? sage
 |