Replication strategy, write throughput

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

I'd like to understand how replication works.
In the paper [1] several replication strategies are described, and
according to a (bit old) mailing list post [2] primary-copy is used.
Therefore the primary OSD waits until the object is persisted and then
updates all replicas in parallel.

Current cluster setup:
Ceph jewel 10.2.3
6 storage nodes
24 HDDs each, journal on same disk [3]
Frontend network: 10 Gbit/s
Backend network: 2 x 10 Gbit/s bonded with layer3+4 hashing [4]
CephFS with striping: 1M stripe unit, 10 stripe count, 10M object size

My assumption was that there should be no difference whether I write
to replication 2 or 3, because each storage node can accept 10 Gbit/s
traffic from the frontend network and send 10 Gbit/s traffic
simultaneous to two other storage nodes.

Disk write capacity shouldn't be a problem either:
200 MB/s throughput * 6 nodes * 24 disks / 2 (journal) / 3 replicas = 4800 MB/s.

Results with 7 clients:
Replication 1: 5695.33 MB/s
Replication 2: 3337.09 MB/s
Replication 3: 1898.17 MB/s

Replication 2 is about 1/2 of Replication 1, and Replication 3 is
exact 1/3 of Replication 1.
Any hints what the bottleneck is in this case?


[1] http://ceph.com/papers/weil-rados-pdsw07.pdf
[2] http://www.spinics.net/lists/ceph-devel/msg02420.html
[3] Test with fio --name=job --ioengine=libaio --rw=write
--blocksize=1M --size=30G --direct=1 --sync=1 --iodepth=128
--filename=/dev/sdw gives about 200 MB/s (test for journal writes)
[4] Test with iperf3, 1 storage node connects to 2 other nodes to the
backend IP gives 10 Gbit/s throughput for each connection


Thanks,
Andreas
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux