Hi Somnath,
Was this with HT enabled/disabled on both the cluster and the RBD nodes?
Mark
On 10/08/2014 07:50 PM, Somnath Roy wrote:
Hi Jiangang,
I managed to get some data for you but it's for a 3 node cluster. I will try to get data for single node as well.
Test config:
-------------
Cluster and rbd node config:
----------------------------------
"2x E5-2680 10C 2.8GHz 25M
8x 16GB RDIMM, dual rank x4 (128GB)
Mellanox MT27500 40 Gigabit Ethernet
LSI 9207 SAS HBA"
8 X 800 GB SSDs (Optimus Eco) per cluster node
3 cluster nodes + 3 rbd nodes
Total storage ~ 19 TB
We have total 24 OSDs running , each node has 8 OSDs/SSD
Configured 3 pools with 528 PGs/pool and 6 RBDs/pool . Each RBD image size is ~230G.
We have tried on 64K_RR_QD64 workload here.
HT_ENABLE
--------------
IOPS : 112500
Throughput (MB/S): 7012
Avg Resp.Time (m.sec): 17
Max Resp.Time (m.sec): 3184
HT_DISABLE
--------------
IOPS : 120864
Throughput (MB/S): 7530
Avg Resp.Time (m.sec): 11
Max Resp.Time (m.sec): 1056
So, ~7% iop increase but response time decrease is ~35% which is real good.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@xxxxxxxxx]
Sent: Wednesday, October 08, 2014 1:03 PM
To: Somnath Roy; Andreas Bluemle; ceph-devel@xxxxxxxxxxxxxxx
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Sound good. Thanks. -jiangang
-----Original Message-----
From: Somnath Roy [mailto:Somnath.Roy@xxxxxxxxxxx]
Sent: Wednesday, October 08, 2014 10:53 AM
To: Duan, Jiangang; Andreas Bluemle; ceph-devel@xxxxxxxxxxxxxxx
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi Jiangang,
Give me a day or two, I will gather all the data and share with community.
Thanks & Regards
Somnath
-----Original Message-----
From: Duan, Jiangang [mailto:jiangang.duan@xxxxxxxxx]
Sent: Wednesday, October 08, 2014 10:47 AM
To: Somnath Roy; Andreas Bluemle; ceph-devel@xxxxxxxxxxxxxxx
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Can you guys share the w/ HT and w/o HT data? I want to take a look at that to understand why.
-jiangang
-----Original Message-----
From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Somnath Roy
Sent: Wednesday, October 08, 2014 10:38 AM
To: Andreas Bluemle; ceph-devel@xxxxxxxxxxxxxxx
Subject: RE: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Thanks Andres for sharing this. I will try those out.
BTW, I am using Ubuntu 14.04 LTS and couldn't find any sysfs entry like 'cpufreq'..
root@stormeap-4:~# ll /sys/devices/system/cpu/cpu10/
cache/ crash_notes driver/ microcode/ online subsystem/ topology/
cpuidle/ crash_notes_size firmware_node/ node0/ power/ thermal_throttle/ uevent
I am using Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz.
Regards
Somnath
-----Original Message-----
From: ceph-devel-owner@xxxxxxxxxxxxxxx [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Andreas Bluemle
Sent: Wednesday, October 08, 2014 9:33 AM
To: ceph-devel@xxxxxxxxxxxxxxx
Subject: Re: 10/7/2014 Weekly Ceph Performance Meeting: kernel boot params
Hi,
as mentioned during today's meeting, here are the kernel boot parameters which I found to provide the basis for good performance results:
processor.max_cstate=0
intel_idle.max_cstate=0
I understand these to basically turn off any power saving modes of the CPU; the CPU's we are using are like
Intel(R) Xeon(R) CPU E5-2640 v2 @ 2.00GHz
Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
At the BIOS level, we
- turn off Hyperthraeding
- turn off Turbo mode (in order ot not leave the specifications)
- turn on frequency floor override
We also assert that
/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
is set to "performance"
Using above we see a constant frequency at the maximum level allowed by the CPU (except Turbo mode).
Best Regards
Andreas Bluemle
On Wed, 8 Oct 2014 02:51:21 +0200
Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote:
Hi All,
Just a remind that the weekly performance meeting is on Wednesdays at
8AM PST. Same bat time, same bat channel!
Etherpad URL:
http://pad.ceph.com/p/performance_weekly
To join the Meeting:
https://bluejeans.com/268261044
To join via Browser:
https://bluejeans.com/268261044/browser
To join with Lync:
https://bluejeans.com/268261044/lync
To join via Room System:
Video Conferencing System: bjn.vc -or- 199.48.152.152 Meeting ID:
268261044
To join via Phone:
1) Dial:
+1 408 740 7256
+1 888 240 2560(US Toll Free)
+1 408 317 9253(Alternate Number)
(see all numbers - http://bluejeans.com/numbers)
2) Enter Conference ID: 268261044
Mark
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel"
in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo
info at http://vger.kernel.org/majordomo-info.html
--
Andreas Bluemle mailto:Andreas.Bluemle@xxxxxxxxxxx
ITXperts GmbH http://www.itxperts.de
Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917
D-81541 Muenchen (Germany) Fax: (+49) 89 89044910
Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
