Hi, I went from using v0.88 to v0.90 and can confirm the that performance is similar between these two versions. I am using config settings similar to the values given by Somnath. There is one difference in my settings: where Somanth has disabled message throttling both for the number of messages and the amount of message data, I am using the settiings: "osd_client_message_size_cap": "524288000" (default) "osd_client_message_cap": "3000" (default is 100) With my test profile (small, i.e. 4k, random writes), the message size throttle is no problem - but the message count throttle is worth to look at: with my test profile I hit this throttle - but this seems to depend on the number of placement groups (or the distribution achieved by different placement group set). I have configured my cluster (3 nodes, 12 osds) with 3 rbd pools: rbd with 768 pgs (default) rbd2 with 3000 pgs rbd3 with 769 pgs I don't see any hits of message count throttle on rbd2 and rbd3 - but on rbd, I see the message throttle hits about 3.750 times during a test with a total of 262.144 client write request within a period of approx. 35 seconds. All of the hits of the message count throttle happen on a single osd; there is no hit of the message count throttle on any of the other 11 osds. Sage: yesterday, you had been asking for a value of the message count throttle where my tests start to run smoothly - and I can't give an answer. It depends on the distribution of I/O requests achieved by the specific set of pg's - and vice versa, a different pattern of I/O requests will change the behavior again. Regards Andreas Bluemle On Wed, 14 Jan 2015 22:44:01 +0000 Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote: > Stephen, > You may want to tweak the following parameter(s) in your ceph.conf > file and see if it is further improving your performance or not. > > debug_lockdep = 0/0 > debug_context = 0/0 > debug_crush = 0/0 > debug_buffer = 0/0 > debug_timer = 0/0 > debug_filer = 0/0 > debug_objecter = 0/0 > debug_rados = 0/0 > debug_rbd = 0/0 > debug_journaler = 0/0 > debug_objectcatcher = 0/0 > debug_client = 0/0 > debug_osd = 0/0 > debug_optracker = 0/0 > debug_objclass = 0/0 > debug_filestore = 0/0 > debug_journal = 0/0 > debug_ms = 0/0 > debug_monc = 0/0 > debug_tp = 0/0 > debug_auth = 0/0 > debug_finisher = 0/0 > debug_heartbeatmap = 0/0 > debug_perfcounter = 0/0 > debug_asok = 0/0 > debug_throttle = 0/0 > debug_mon = 0/0 > debug_paxos = 0/0 > debug_rgw = 0/0 > osd_op_num_threads_per_shard = 2 //You may want to try with 1 as well > osd_op_num_shards = 10 //Depends on your cpu util > ms_nocrc = true > cephx_sign_messages = false > cephx_require_signatures = false > ms_dispatch_throttle_bytes = 0 > throttler_perf_counter = false > > [osd] > osd_client_message_size_cap = 0 > osd_client_message_cap = 0 > osd_enable_op_tracker = false > > Also, run more clients (in your case rados bench) and see if it is > scaling or not (it should, till it saturates your cpu). > > But, your observation on RHEL7 vs UBUNTU 14.04 LTS is interesting ! > > Thanks & Regards > Somnath > -----Original Message----- > From: ceph-devel-owner@xxxxxxxxxxxxxxx > [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Blinick, > Stephen L Sent: Wednesday, January 14, 2015 2:32 PM To: Ceph > Development Subject: RE: Memstore performance improvements v0.90 vs > v0.87 > > I went back and grabbed 87 and built it on RHEL7 as well, and > performance is also similar (much better). I've also run it on a few > systems (Dual socket 10-core E5v2, Dual socket 6-core E5v3). So, > it's related to my switch to RHEL7, and not to the code changes > between v0.90 and v0.87. Will post when I get more data. > > Thanks, > > Stephen > > -----Original Message----- > From: ceph-devel-owner@xxxxxxxxxxxxxxx > [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Blinick, > Stephen L Sent: Wednesday, January 14, 2015 12:06 AM To: Ceph > Development Subject: Memstore performance improvements v0.90 vs v0.87 > > In the process of moving to a new cluster (RHEL7 based) I grabbed > v0.90, compiled RPM's and re-ran the simple local-node memstore test > I've run on .80 - .87. It's a single Memstore OSD and a single Rados > Bench client locally on the same node. Increasing queue depth and > measuring latency /IOPS. So far, the measurements have been > consistent across different hardware and code releases (with about a > 30% improvement with the OpWQ Sharding changes that came in after > Firefly). > > These are just very early results, but I'm seeing a very large > improvement in latency and throughput with v90 on RHEL7. Next I'm > working to get lttng installed and working in RHEL7 to determine > where the improvement is. On previous levels, these measurements > have been roughly the same using a real (fast) backend (i.e. NVMe > flash), and I will verify here as well. Just wondering if anyone > else has measured similar improvements? > > > 100% Reads or Writes, 4K Objects, Rados Bench > > ======================== > V0.87: Ubuntu 14.04LTS > > *Writes* > #Thr IOPS Latency(ms) > 1 618.80 1.61 > 2 1401.70 1.42 > 4 3962.73 1.00 > 8 7354.37 1.10 > 16 7654.67 2.10 > 32 7320.33 4.37 > 64 7424.27 8.62 > > *Reads* > #thr IOPS Latency(ms) > 1 837.57 1.19 > 2 1950.00 1.02 > 4 6494.03 0.61 > 8 7243.53 1.10 > 16 7473.73 2.14 > 32 7682.80 4.16 > 64 7727.10 8.28 > > > ======================== > V0.90: RHEL7 > > *Writes* > #Thr IOPS Latency(ms) > 1 2558.53 0.39 > 2 6014.67 0.33 > 4 10061.33 0.40 > 8 14169.60 0.56 > 16 14355.63 1.11 > 32 14150.30 2.26 > 64 15283.33 4.19 > > *Reads* > #Thr IOPS Latency(ms) > 1 4535.63 0.22 > 2 9969.73 0.20 > 4 17049.43 0.23 > 8 19909.70 0.40 > 16 20320.80 0.79 > 32 19827.93 1.61 > 64 22371.17 2.86 > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe > from this list: send the line "unsubscribe ceph-devel" in the body of > a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at > http://vger.kernel.org/majordomo-info.html > > ________________________________ > > PLEASE NOTE: The information contained in this electronic mail > message is intended only for the use of the designated recipient(s) > named above. If the reader of this message is not the intended > recipient, you are hereby notified that you have received this > message in error and that any review, dissemination, distribution, or > copying of this message is strictly prohibited. If you have received > this communication in error, please notify the sender by telephone or > e-mail (as shown above) immediately and destroy any and all copies of > this message in your possession (whether hard copies or > electronically stored copies). > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- Andreas Bluemle mailto:Andreas.Bluemle@xxxxxxxxxxx ITXperts GmbH http://www.itxperts.de Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 Company details: http://www.itxperts.de/imprint.htm -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
