On Thu, 15 Jan 2015, Andreas Bluemle wrote: > Hi, > > I went from using v0.88 to v0.90 and can confirm the > that performance is similar between these two versions. > > I am using config settings similar to the values given by > Somnath. > > There is one difference in my settings: where Somanth has disabled > message throttling both for the number of messages and the amount > of message data, I am using the settiings: > > "osd_client_message_size_cap": "524288000" (default) > "osd_client_message_cap": "3000" (default is 100) > > With my test profile (small, i.e. 4k, random writes), the message > size throttle is no problem - but the message count throttle > is worth to look at: > > with my test profile I hit this throttle - but this seems to depend > on the number of placement groups (or the distribution achieved > by different placement group set). > > I have configured my cluster (3 nodes, 12 osds) with 3 rbd pools: > rbd with 768 pgs (default) > rbd2 with 3000 pgs > rbd3 with 769 pgs > > I don't see any hits of message count throttle on rbd2 and rbd3 - but > on rbd, I see the message throttle hits about 3.750 times during a > test with a total of 262.144 client write request within a period > of approx. 35 seconds. All of the hits of the message count throttle > happen on a single osd; there is no hit of the message count throttle > on any of the other 11 osds. The difference between rbd and rbd3 is pretty surprising.. it makes me think that the rbd distribution is just a bit unlucky. Can you try this? ceph osd reweight-by-pg 110 or possibly 105 and see if that changes things? > Sage: yesterday, you had been asking for a value of the message > count throttle where my tests start to run smoothly - and I can't > give an answer. It depends on the distribution of I/O requests > achieved by the specific set of pg's - and vice versa, a different > pattern of I/O requests will change the behavior again. Understood. This is actually good news, I think.. it's not the throttle itself that's problematic but that the rbd distrbution is too inbalanced. Thanks! sage > > > > Regards > > Andreas Bluemle > > On Wed, 14 Jan 2015 22:44:01 +0000 > Somnath Roy <Somnath.Roy@xxxxxxxxxxx> wrote: > > > Stephen, > > You may want to tweak the following parameter(s) in your ceph.conf > > file and see if it is further improving your performance or not. > > > > debug_lockdep = 0/0 > > debug_context = 0/0 > > debug_crush = 0/0 > > debug_buffer = 0/0 > > debug_timer = 0/0 > > debug_filer = 0/0 > > debug_objecter = 0/0 > > debug_rados = 0/0 > > debug_rbd = 0/0 > > debug_journaler = 0/0 > > debug_objectcatcher = 0/0 > > debug_client = 0/0 > > debug_osd = 0/0 > > debug_optracker = 0/0 > > debug_objclass = 0/0 > > debug_filestore = 0/0 > > debug_journal = 0/0 > > debug_ms = 0/0 > > debug_monc = 0/0 > > debug_tp = 0/0 > > debug_auth = 0/0 > > debug_finisher = 0/0 > > debug_heartbeatmap = 0/0 > > debug_perfcounter = 0/0 > > debug_asok = 0/0 > > debug_throttle = 0/0 > > debug_mon = 0/0 > > debug_paxos = 0/0 > > debug_rgw = 0/0 > > osd_op_num_threads_per_shard = 2 //You may want to try with 1 as well > > osd_op_num_shards = 10 //Depends on your cpu util > > ms_nocrc = true > > cephx_sign_messages = false > > cephx_require_signatures = false > > ms_dispatch_throttle_bytes = 0 > > throttler_perf_counter = false > > > > [osd] > > osd_client_message_size_cap = 0 > > osd_client_message_cap = 0 > > osd_enable_op_tracker = false > > > > Also, run more clients (in your case rados bench) and see if it is > > scaling or not (it should, till it saturates your cpu). > > > > But, your observation on RHEL7 vs UBUNTU 14.04 LTS is interesting ! > > > > Thanks & Regards > > Somnath > > -----Original Message----- > > From: ceph-devel-owner@xxxxxxxxxxxxxxx > > [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Blinick, > > Stephen L Sent: Wednesday, January 14, 2015 2:32 PM To: Ceph > > Development Subject: RE: Memstore performance improvements v0.90 vs > > v0.87 > > > > I went back and grabbed 87 and built it on RHEL7 as well, and > > performance is also similar (much better). I've also run it on a few > > systems (Dual socket 10-core E5v2, Dual socket 6-core E5v3). So, > > it's related to my switch to RHEL7, and not to the code changes > > between v0.90 and v0.87. Will post when I get more data. > > > > Thanks, > > > > Stephen > > > > -----Original Message----- > > From: ceph-devel-owner@xxxxxxxxxxxxxxx > > [mailto:ceph-devel-owner@xxxxxxxxxxxxxxx] On Behalf Of Blinick, > > Stephen L Sent: Wednesday, January 14, 2015 12:06 AM To: Ceph > > Development Subject: Memstore performance improvements v0.90 vs v0.87 > > > > In the process of moving to a new cluster (RHEL7 based) I grabbed > > v0.90, compiled RPM's and re-ran the simple local-node memstore test > > I've run on .80 - .87. It's a single Memstore OSD and a single Rados > > Bench client locally on the same node. Increasing queue depth and > > measuring latency /IOPS. So far, the measurements have been > > consistent across different hardware and code releases (with about a > > 30% improvement with the OpWQ Sharding changes that came in after > > Firefly). > > > > These are just very early results, but I'm seeing a very large > > improvement in latency and throughput with v90 on RHEL7. Next I'm > > working to get lttng installed and working in RHEL7 to determine > > where the improvement is. On previous levels, these measurements > > have been roughly the same using a real (fast) backend (i.e. NVMe > > flash), and I will verify here as well. Just wondering if anyone > > else has measured similar improvements? > > > > > > 100% Reads or Writes, 4K Objects, Rados Bench > > > > ======================== > > V0.87: Ubuntu 14.04LTS > > > > *Writes* > > #Thr IOPS Latency(ms) > > 1 618.80 1.61 > > 2 1401.70 1.42 > > 4 3962.73 1.00 > > 8 7354.37 1.10 > > 16 7654.67 2.10 > > 32 7320.33 4.37 > > 64 7424.27 8.62 > > > > *Reads* > > #thr IOPS Latency(ms) > > 1 837.57 1.19 > > 2 1950.00 1.02 > > 4 6494.03 0.61 > > 8 7243.53 1.10 > > 16 7473.73 2.14 > > 32 7682.80 4.16 > > 64 7727.10 8.28 > > > > > > ======================== > > V0.90: RHEL7 > > > > *Writes* > > #Thr IOPS Latency(ms) > > 1 2558.53 0.39 > > 2 6014.67 0.33 > > 4 10061.33 0.40 > > 8 14169.60 0.56 > > 16 14355.63 1.11 > > 32 14150.30 2.26 > > 64 15283.33 4.19 > > > > *Reads* > > #Thr IOPS Latency(ms) > > 1 4535.63 0.22 > > 2 9969.73 0.20 > > 4 17049.43 0.23 > > 8 19909.70 0.40 > > 16 20320.80 0.79 > > 32 19827.93 1.61 > > 64 22371.17 2.86 > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > > info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe > > from this list: send the line "unsubscribe ceph-devel" in the body of > > a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at > > http://vger.kernel.org/majordomo-info.html > > > > ________________________________ > > > > PLEASE NOTE: The information contained in this electronic mail > > message is intended only for the use of the designated recipient(s) > > named above. If the reader of this message is not the intended > > recipient, you are hereby notified that you have received this > > message in error and that any review, dissemination, distribution, or > > copying of this message is strictly prohibited. If you have received > > this communication in error, please notify the sender by telephone or > > e-mail (as shown above) immediately and destroy any and all copies of > > this message in your possession (whether hard copies or > > electronically stored copies). > > > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > > in the body of a message to majordomo@xxxxxxxxxxxxxxx > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > > -- > Andreas Bluemle mailto:Andreas.Bluemle@xxxxxxxxxxx > ITXperts GmbH http://www.itxperts.de > Balanstrasse 73, Geb. 08 Phone: (+49) 89 89044917 > D-81541 Muenchen (Germany) Fax: (+49) 89 89044910 > > Company details: http://www.itxperts.de/imprint.htm > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
