|
We've built Ceph cluster:
3 mon nodes (one of them is combined with mds)
3 osd nodes (each one have 10 osd + 2 ssd for journaling)
switch 24 ports x 10G
10 gigabit - for public network
20 gigabit bonding - between osds
Ubuntu 12.04.05
Ceph 0.87.2
-----------------------------------------------------
Clients has:
10 gigabit for ceph-connection
CentOS 6.6 with kernel 3.19.8 equipped by cephfs-kmodule
====== fio-2.0.13 seqwrite, bs=1M, filesize=10G, parallel-jobs=16 =========== Single client: ++++++++++++++++++++++++++++ Starting 16 processes .....below is just 1 job info.... trivial-readwrite-grid01: (groupid=0, jobs=1): err= 0: pid=10484: Tue Jul 28 13:26:24 2015 write: io=10240MB, bw=78656KB/s, iops=76 , runt=133312msec slat (msec): min=1 , max=117 , avg=13.01, stdev=12.57 clat (usec): min=1 , max=68 , avg= 3.61, stdev= 1.99 lat (msec): min=1 , max=117 , avg=13.01, stdev=12.57 clat percentiles (usec): | 1.00th=[ 1], 5.00th=[ 2], 10.00th=[ 2], 20.00th=[ 2], | 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 3], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 6], | 99.00th=[ 9], 99.50th=[ 10], 99.90th=[ 23], 99.95th=[ 28], | 99.99th=[ 62] bw (KB/s) : min=35790, max=318215, per=6.31%, avg=78816.91, stdev=26397.76 lat (usec) : 2=1.33%, 4=54.43%, 10=43.54%, 20=0.56%, 50=0.11% lat (usec) : 100=0.03% cpu : usr=0.89%, sys=12.85%, ctx=58248, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=10240/d=0, short=r=0/w=0/d=0 ...what's above repeats 16 times... Run status group 0 (all jobs): WRITE: io=163840MB, aggrb=1219.8MB/s, minb=78060KB/s, maxb=78655KB/s, mint=133312msec, maxt=134329msec +++++++++++++++++++++++++++++++++ Two clients: +++++++++++++++++++++++++++++++++ ....below is just 1 job info.... trivial-readwrite-gridsrv: (groupid=0, jobs=1): err= 0: pid=10605: Tue Jul 28 14:05:59 2015 write: io=10240MB, bw=43154KB/s, iops=42 , runt=242984msec slat (usec): min=991 , max=285653 , avg=23716.12, stdev=23960.60 clat (usec): min=1 , max=65 , avg= 3.67, stdev= 2.02 lat (usec): min=994 , max=285664 , avg=23723.39, stdev=23962.22 clat percentiles (usec): | 1.00th=[ 2], 5.00th=[ 2], 10.00th=[ 2], 20.00th=[ 2], | 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 3], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 6], | 99.00th=[ 8], 99.50th=[ 10], 99.90th=[ 28], 99.95th=[ 37], | 99.99th=[ 56] bw (KB/s) : min=20630, max=276480, per=6.30%, avg=43328.34, stdev=21905.92 lat (usec) : 2=0.84%, 4=49.45%, 10=49.13%, 20=0.37%, 50=0.18% lat (usec) : 100=0.03% cpu : usr=0.49%, sys=5.68%, ctx=31428, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=10240/d=0, short=r=0/w=0/d=0 ...what's above repeats 16 times... Run status group 0 (all jobs): WRITE: io=163840MB, aggrb=687960KB/s, minb=42997KB/s, maxb=43270KB/s, mint=242331msec, maxt=243869msec --------- And almost the same(?!) aggregated result from the second client: --------- Run status group 0 (all jobs): WRITE: io=163840MB, aggrb=679401KB/s, minb=42462KB/s, maxb=42852KB/s, mint=244697msec, maxt=246941msec ----------------- If I'll summarize: --------------------- aggrb1 + aggrb2 = 687960KB/s + 679401KB/s = 1367MB/s it looks like the same bandwidth from just one client aggrb=1219.8MB/s and it was divided? why? Question: If I'll connect 12 clients nodes - each one can write just on 100MB/s? Perhaps, I need to scale out our ceph up to 15(how many?) OSD nodes - and it'll serve 2 clients on the 1.3GB/s (bw of 10gig nic), or not? ============================================================================ health HEALTH_OK monmap e1: 3 mons at {mon1=192.168.56.251:6789/0,mon2=192.168.56.252:6789/0,mon3=192.168.56.253:6789/0}, election epoch 140, quorum 0,1,2 mon1,mon2,mon3 mdsmap e12: 1/1/1 up {0=mon3=up:active} osdmap e832: 31 osds: 30 up, 30 in pgmap v106186: 6144 pgs, 3 pools, 2306 GB data, 1379 kobjects 4624 GB used, 104 TB / 109 TB avail 6144 active+clean Perhaps, I don't understand something in Ceph architecture? I thought, that: Each spindel-disk can write ~ 100MB/s , and we have 10 SAS disks on each node = aggregated write speed is ~ 900MB/s (because of striping etc.) And we have 3 OSD nodes, and objects are striped also on 30 osds - I thought it's also aggregateble and we'll get something around 2.5 GB/s, but not... ************************************************************************************ This footnote confirms that this email message has been scanned by PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses. ************************************************************************************ |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
