RADOS Bench slow write speed

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi all!!
I'm setting up a Ceph (version 0.80.6) cluster and I'mbenchmarking the infrastructure and Ceph itself. I've got 3 rack servers (Dell R630) each with it's own disks in enclosures. 
The cluster network bandwidth is of 10Gbps, the bandwidth between the RAID controller (Dell H830) and enclosure (MD1400) is of 48 Gbps and negotiated speed with each disk (configured as JBOD) is of 6 Gbps. 
We use 4 TB SAS disks, 7200 RPM with sustained bandwidth of 175 MB/s (confirmed with fio).

I tried to simulate a Ceph pattern by writing/reading on a single disk with XFS, a great number of 4 MB files with on instance of fio.

[writetest]
ioengine=libaio
directory=/var/local/xfs
filesize=4m
iodepth=256
rw= write
direct=1
numjobs=1
loops=1
nrfiles=18000
bs=4096k
 
For the other tests (sequential reads, random writes, random reads), only the "rw" changes.

I got the following bandwidth:
Sequential write speed: 137 MB/s
Sequential read speed:  144.5 MB/s
Radom write speed: 134 MB/s
Radom read speed: 144 MB/s

Ok, it has an overhead associated with writing a great amount of 4 MB files.

Now, I move on to bechmark with an OSD on top of the disk, so I've got the cluster with only 1 OSD (separate partitions for journal and data) and run rados bench.

1 Thread:
Writes: 27.7 MB/s
Reads: 90 MB/s
Random reads: 82 MB/s

16 Threads (default):
Writes: 37 MB/s
Reads: 79.2 MB/s
Random reads:  71.4 MB/s


As you can see, writes are awfully slow. What I have notice is very high latencies:

Total writes made:      16868
Write size:             4194304
Bandwidth (MB/sec):     37.449 
Stddev Bandwidth:       24.606
Max bandwidth (MB/sec): 120
Min bandwidth (MB/sec): 0
Average Latency:        1.70862
Stddev Latency:         1.13098
Max latency:            6.33929
Min latency:            0.20767

Is this throughput to be expected from 7.2K RPM disks with 4 TB size? Or is anything in the Ceph configuration that might be changed to decrease the observed latency? Any sugestions?

Appreciated,
Pedro Miranda.


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux