Re: Direct IO tests on RBD device vary significantly

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



You're not really testing only a RBD device there - you're testing
1) the O_DIRECT implementation in the kernel version you have (they differ)
- try different kernels in guest
2) cache implementation in qemu (and possibly virtio block driver) - if it's enabled
- disable it for this test completely (cache=none)
3) O_DIRECT implementation on the filesystem where your "test" file is - most important is preallocation!
- I'm not sure if you can "dd" into an existing file without truncating it, but you should first create the file with:
dd if=/dev/zero of=test bs=1M
It's better to create a new virtual drive and attach it to this machine, then test it directly (and while dd is good for "baseline" testing, I recommend you use fio)

Btw I find the 4MB test pretty consistent, it seems to oscillate about ~50MB/s. In the beginning the cluster was probably busy doing something else (scrubbing? cleanup of something? cron scripts?).

Jan

> On 07 Aug 2015, at 03:31, Steve Dainard <sdainard@xxxxxxxx> wrote:
> 
> Trying to get an understanding why direct IO would be so slow on my cluster.
> 
> Ceph 0.94.1
> 1 Gig public network
> 10 Gig public network
> 10 Gig cluster network
> 
> 100 OSD's, 4T disk sizes, 5G SSD journal.
> 
> As of this morning I had no SSD journal and was finding direct IO was
> sub 10MB/s so I decided to add journals today.
> 
> Afterwards I started running tests again and wasn't very impressed.
> Then for no apparent reason the write speeds increased significantly.
> But I'm finding they vary wildly.
> 
> Currently there is a bit of background ceph activity, but only my
> testing client has an rbd mapped/mounted:
>           election epoch 144, quorum 0,1,2 mon1,mon3,mon2
>     osdmap e181963: 100 osds: 100 up, 100 in
>            flags noout
>      pgmap v2852566: 4144 pgs, 7 pools, 113 TB data, 29179 kobjects
>            227 TB used, 135 TB / 363 TB avail
>                4103 active+clean
>                  40 active+clean+scrubbing
>                   1 active+clean+scrubbing+deep
> 
> Tests:
> 1M block size: http://pastebin.com/LKtsaHrd throughput has no consistency
> 4k block size: http://pastebin.com/ib6VW9eB thoughput is amazingly consistent
> 
> Thoughts?
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux