On Thu, Jun 28, 2012 at 2:17 PM, Stefan Priebe <s.priebe@xxxxxxxxxxxx> wrote: > Am 28.06.2012 18:12, schrieb Josh Durgin: > >> On 06/28/2012 06:10 AM, Stefan Priebe - Profihost AG wrote: >>> >>> Hello list, >>> >>> my cluster is now pretty stable i'm just wondering about the sequential >>> write values. >>> >>> With rados bench command and 16 threads i get totally different values >>> than with KVM and rbd block device. >>> >>> rados -p kvmpool bench 60 write -t 16: >>> pool size 2: Bandwidth (MB/sec): 1137.294 >>> pool size 3: Bandwidth (MB/sec): 846.983 >>> >>> Inside KVM with fio: >>> >>> fio --filename=$DISK --direct=1 --rw=write --bs=4M --size=200G >>> --numjobs=16 --runtime=60 --group_reporting --name=file1: >> >> >> There are a number of differences between running that in a vm on rbd >> and rados bench. >> >> Keep in mind it's running on a filesystem, so requests go through the >> guest fs and block layer before getting into librbd. > > No it doesn't i'm testing directly the block device. > > >> If you don't use direct I/O, and you enable rbd writeback caching, >> librbd will be able to merge many of the smaller requests and >> you should see much better throughput. > > I'm already using rbd writeback and it works good for random 4k writes, But > it doesn't make sense for sequential 4M writes. I haven't been able to come up with a good explanation for this, but if you're interested in exploring it further, you can gather logging data that includes the messages KVM is sending out to the OSDs (and do the same for rados bench). Any differences we see there would be instructive. -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
