Hi, But my question is why speed is divided between clients? And how much OSDnodes, OSDdaemos, PGs, I have to add/remove to ceph, that each cephfs-client could write with his max network speed (10Gbit/s ~ 1.2GB/s)??? ________________________________________ From: Johannes Formann <mlmail@xxxxxxxxxx> Sent: Tuesday, July 28, 2015 12:46 PM To: Shneur Zalman Mattern Subject: Re: Did maximum performance reached? Hi, size=3 would decrease your performance. But with size=2 your results are not bad too: Math: size=2 means each write is written 4 times (2 copies, first journal, later disk). Calculating with 1.300MB/s „Client“ Bandwidth that means: 2 (size) * 1300 MB/s / 6 (SSD) => 433MB/s each SSD 2 (size) * 1300 MB/s / 30 (HDD) => 87MB/s each HDD greetings Johannes > Am 28.07.2015 um 11:41 schrieb Shneur Zalman Mattern <shzama@xxxxxxxxxxxx>: > > Hi, Johannes (that's my grandpa's name) > > The size is 2, do you really think that number of replicas can increase performance? > on the http://ceph.com/docs/master/architecture/ > written "Note: Striping is independent of object replicas. Since CRUSH replicates objects across OSDs, stripes get replicated automatically. " > > OK, I'll check it, > Regards, Shneur > ________________________________________ > From: Johannes Formann <mlmail@xxxxxxxxxx> > Sent: Tuesday, July 28, 2015 12:09 PM > To: Shneur Zalman Mattern > Cc: ceph-users@xxxxxxxxxxxxxx > Subject: Re: Did maximum performance reached? > > Hello, > > what is the „size“ parameter of your pool? > > Some math do show the impact: > size=3 means each write is written 6 times (3 copies, first journal, later disk). Calculating with 1.300MB/s „Client“ Bandwidth that means: > > 3 (size) * 1300 MB/s / 6 (SSD) => 650MB/s per SSD > 3 (size) * 1300 MB/s / 30 (HDD) => 130MB/s per HDD > > If you use size=3, the results are as good as one can expect. (Even with size=2 the results won’t be bad) > > greetings > > Johannes > >> Am 28.07.2015 um 10:53 schrieb Shneur Zalman Mattern <shzama@xxxxxxxxxxxx>: >> >> We've built Ceph cluster: >> 3 mon nodes (one of them is combined with mds) >> 3 osd nodes (each one have 10 osd + 2 ssd for journaling) >> switch 24 ports x 10G >> 10 gigabit - for public network >> 20 gigabit bonding - between osds >> Ubuntu 12.04.05 >> Ceph 0.87.2 >> ----------------------------------------------------- >> Clients has: >> 10 gigabit for ceph-connection >> CentOS 6.6 with kernel 3.19.8 equipped by cephfs-kmodule >> >> >> >> ====== fio-2.0.13 seqwrite, bs=1M, filesize=10G, parallel-jobs=16 =========== >> Single client: >> ++++++++++++++++++++++++++++ >> >> Starting 16 processes >> >> .....below is just 1 job info.... >> trivial-readwrite-grid01: (groupid=0, jobs=1): err= 0: pid=10484: Tue Jul 28 13:26:24 2015 >> write: io=10240MB, bw=78656KB/s, iops=76 , runt=133312msec >> slat (msec): min=1 , max=117 , avg=13.01, stdev=12.57 >> clat (usec): min=1 , max=68 , avg= 3.61, stdev= 1.99 >> lat (msec): min=1 , max=117 , avg=13.01, stdev=12.57 >> clat percentiles (usec): >> | 1.00th=[ 1], 5.00th=[ 2], 10.00th=[ 2], 20.00th=[ 2], >> | 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 3], 60.00th=[ 4], >> | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 6], >> | 99.00th=[ 9], 99.50th=[ 10], 99.90th=[ 23], 99.95th=[ 28], >> | 99.99th=[ 62] >> bw (KB/s) : min=35790, max=318215, per=6.31%, avg=78816.91, stdev=26397.76 >> lat (usec) : 2=1.33%, 4=54.43%, 10=43.54%, 20=0.56%, 50=0.11% >> lat (usec) : 100=0.03% >> cpu : usr=0.89%, sys=12.85%, ctx=58248, majf=0, minf=9 >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% >> issued : total=r=0/w=10240/d=0, short=r=0/w=0/d=0 >> >> ...what's above repeats 16 times... >> >> Run status group 0 (all jobs): >> WRITE: io=163840MB, aggrb=1219.8MB/s, minb=78060KB/s, maxb=78655KB/s, mint=133312msec, maxt=134329msec >> >> +++++++++++++++++++++++++++++++++ >> Two clients: >> +++++++++++++++++++++++++++++++++ >> ....below is just 1 job info.... >> trivial-readwrite-gridsrv: (groupid=0, jobs=1): err= 0: pid=10605: Tue Jul 28 14:05:59 2015 >> write: io=10240MB, bw=43154KB/s, iops=42 , runt=242984msec >> slat (usec): min=991 , max=285653 , avg=23716.12, stdev=23960.60 >> clat (usec): min=1 , max=65 , avg= 3.67, stdev= 2.02 >> lat (usec): min=994 , max=285664 , avg=23723.39, stdev=23962.22 >> clat percentiles (usec): >> | 1.00th=[ 2], 5.00th=[ 2], 10.00th=[ 2], 20.00th=[ 2], >> | 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 3], 60.00th=[ 4], >> | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 6], >> | 99.00th=[ 8], 99.50th=[ 10], 99.90th=[ 28], 99.95th=[ 37], >> | 99.99th=[ 56] >> bw (KB/s) : min=20630, max=276480, per=6.30%, avg=43328.34, stdev=21905.92 >> lat (usec) : 2=0.84%, 4=49.45%, 10=49.13%, 20=0.37%, 50=0.18% >> lat (usec) : 100=0.03% >> cpu : usr=0.49%, sys=5.68%, ctx=31428, majf=0, minf=9 >> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% >> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% >> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% >> issued : total=r=0/w=10240/d=0, short=r=0/w=0/d=0 >> >> ...what's above repeats 16 times... >> >> Run status group 0 (all jobs): >> WRITE: io=163840MB, aggrb=687960KB/s, minb=42997KB/s, maxb=43270KB/s, mint=242331msec, maxt=243869msec >> >> --------- And almost the same(?!) aggregated result from the second client: --------- >> >> Run status group 0 (all jobs): >> WRITE: io=163840MB, aggrb=679401KB/s, minb=42462KB/s, maxb=42852KB/s, mint=244697msec, maxt=246941msec >> >> ----------------- If I'll summarize: --------------------- >> aggrb1 + aggrb2 = 687960KB/s + 679401KB/s = 1367MB/s >> >> it looks like the same bandwidth from just one client aggrb=1219.8MB/s and it was divided? why? >> Question: If I'll connect 12 clients nodes - each one can write just on 100MB/s? >> Perhaps, I need to scale out our ceph up to 15(how many?) OSD nodes - and it'll serve 2 clients on the 1.3GB/s (bw of 10gig nic), or not? >> >> ============================================================================ >> >> health HEALTH_OK >> monmap e1: 3 mons at {mon1=192.168.56.251:6789/0,mon2=192.168.56.252:6789/0,mon3=192.168.56.253:6789/0}, election epoch 140, quorum 0,1,2 mon1,mon2,mon3 >> mdsmap e12: 1/1/1 up {0=mon3=up:active} >> osdmap e832: 31 osds: 30 up, 30 in >> pgmap v106186: 6144 pgs, 3 pools, 2306 GB data, 1379 kobjects >> 4624 GB used, 104 TB / 109 TB avail >> 6144 active+clean >> >> >> Perhaps, I don't understand something in Ceph architecture? I thought, that: >> >> Each spindel-disk can write ~ 100MB/s , and we have 10 SAS disks on each node = aggregated write speed is ~ 900MB/s (because of striping etc.) >> And we have 3 OSD nodes, and objects are striped also on 30 osds - I thought it's also aggregateble and we'll get something around 2.5 GB/s, but not... >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ************************************************************************************ >> This footnote confirms that this email message has been scanned by >> PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses. >> ************************************************************************************ >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > ************************************************************************************ > This footnote confirms that this email message has been scanned by > PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses. > ************************************************************************************ > > > > > > > ************************************************************************************ > This footnote confirms that this email message has been scanned by > PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses. > ************************************************************************************ > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ************************************************************************************ This footnote confirms that this email message has been scanned by PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses. ************************************************************************************ ************************************************************************************ This footnote confirms that this email message has been scanned by PineApp Mail-SeCure for the presence of malicious code, vandals & computer viruses. ************************************************************************************ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |