> From: "Logan Barfield" <lbarfield@xxxxxxxxxxxxx> > We've been running some tests to try to determine why our FreeBSD VMs > are performing much worse than our Linux VMs backed by RBD, especially > on writes. > > Our current deployment is: > - 4x KVM Hypervisors (QEMU 2.0.0+dfsg-2ubuntu1.6) > - 2x OSD nodes (8x SSDs each, 10Gbit links to hypervisors, pool has 2x > replication across nodes) > - Hypervisors have "rbd_cache enabled" > - All VMs use "cache=none" currently. If you don't have rbd cache writethrough until flush = true, this configuration is unsafe - with cache=none, qemu will not send flushes. > In testing we were getting ~30MB/s writes, and ~100MB/s reads on > FreeBSD 10.1. On Linux VMs we're seeing ~150+MB/s for writes and > reads (dd if=/dev/zero of=output bs=1M count=1024 oflag=direct). I'm not very familiar with FreeBSD, but I'd guess it's sending smaller I/Os for some reason. This could be due to trusting the sector size qemu reports (this can be changed, though I don't remember the syntax offhand), lower fs block size, or scheduler or block subsystem configurables. It could also be related to differences in block allocation strategies by whatever FS you're using in the guest and Linux filesystems. What FS are you using in each guest? You can check the I/O sizes seen by rbd by adding something like this to ceph.conf on a node running qemu: [client] debug rbd = 20 log file = /path/writeable/by/qemu.$pid.log This will show the offset and length of requests in lines containing aio_read and aio_write. If you're using giant you could instead gather a trace of I/O to rbd via lttng. > I tested several configurations on both RBD and local SSDs, and the > only time FreeBSD performance was comparable to Linux was with the > following configuration: > - Local SSD > - Qemu cache=writeback > - GPT journaling enabled > > We did see some performance improvement (~50MB/s writes instead of > 30MB/s) when using cache=writeback on RBD. > > I've read several threads regarding cache=none vs cache=writeback. > cache=none is apparently safer for live migration, but cache=writeback > is recommended by Ceph to prevent data loss. Apparently there was a > patch submitted for Qemu a few months ago to make cache=writeback > safer for live migrations as well: http://tracker.ceph.com/issues/2467 RBD caching is already safe with live migration without this patch. It just makes sure that it will continue to be safe in case of future QEMU changes. Josh _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |