There was a bug in the rbd CLI bench-write tool that would result in the same offset being re-written [1]. Since writeback cache is enabled (by default), in your example only 4MB would be written to the OSD at the conclusion of the test. The fix should have been scheduled for backport to Hammer but it looks like it was missed. I will open a new tracker ticket to start that process. [1] https://github.com/ceph/ceph/commit/333f3a01a9916c781f266078391c580efb81a0fc -- Jason Dillaman ----- Original Message ----- > From: "Emile Snyder" <emsnyder@xxxxxxxx> > To: ceph-users@xxxxxxxxxxxxxx > Sent: Monday, January 4, 2016 3:51:25 PM > Subject: rbd bench-write vs dd performance confusion > > Hi all, > > I'm trying to get comfortable with managing and benchmarking ceph clusters, > and I'm struggling to understan rbd bench-write results vs using dd against > mounted rbd images. > > I have a 6 node test cluster running version 0.94.5, 2 nodes per rack, 20 > OSDs per node. Write journals are on the same disk as their OSD. My rbd pool > is set for 3 replicas, with 2 on different hosts in a given rack, and 3rd on > some host in a different rack. > > > I created a test 100GB image with 4MB object size, created a VM client, and > mounted the image at /dev/rbd1. > > In a shell on one of my 6 storage nodes I have 'iostat 2' running. > > Now my confusion; If I run on the client: > > 'sudo dd if=/dev/zero of=/dev/rbd1 bs=4M count=1000 iflag=fullblock > oflag=direct' > > I see '4194304000 bytes (4.2 GB) copied, 18.5798 s, 226 MB/s' and the iostat > on the storage node shows almost all 20 disks sustaining 4-16MB/s writes. > > However, if I run > > 'rbd --cluster <clustername> bench-write test-4m-image --io-size 4000000 > --io-threads 1 --io-total 40000000000 --io-pattern rand' > > I see 'elapsed: 12 ops: 10000 ops/sec: 805.86 bytes/sec: > 3223441447.72' but the iostat shows the disks basically all at 0.00kb_wrtn/s > for the duration of the run. > > So that's bench-write reporting 3.2 GB/s with iostat showing *nothing* > happening, while dd writes 226 MB/s and iostat lights up. Am I > misunderstanding what rbd-bench is supposed to do? > > Thanks, > -Emile > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |