On 06/28/2012 06:10 AM, Stefan Priebe - Profihost AG wrote:
Hello list, my cluster is now pretty stable i'm just wondering about the sequential write values. With rados bench command and 16 threads i get totally different values than with KVM and rbd block device. rados -p kvmpool bench 60 write -t 16: pool size 2: Bandwidth (MB/sec): 1137.294 pool size 3: Bandwidth (MB/sec): 846.983 Inside KVM with fio: fio --filename=$DISK --direct=1 --rw=write --bs=4M --size=200G --numjobs=16 --runtime=60 --group_reporting --name=file1:
There are a number of differences between running that in a vm on rbd and rados bench. Keep in mind it's running on a filesystem, so requests go through the guest fs and block layer before getting into librbd. These two layers can break up those 4M writes, so you end up doing a bunch more small I/Os which degrades performance a bunch. Running those 16 processes in does not directly translate to 16 I/Os in flight from the guest kernel, like rados bench is doing. If you use blktrace on the guest, or just add --debug-ms 1, you can track the requests the guest is sending by looking at the lines with 'osd_op\(.*'. If you don't use direct I/O, and you enable rbd writeback caching, librbd will be able to merge many of the smaller requests and you should see much better throughput. Josh
pool size 2: write: io=32984MB, bw=562046KB/s, iops=137 , runt= 60094msec pool size 3: write: io=29124MB, bw=496024KB/s, iops=121 , runt= 60124msec Even when i change the pool size to 3 i get with fio 520MB/s. Any ideas? Is this expected? Greets Stefan
-- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
