Haomai/Dong Have you tried this with latest shardedpool/WQ model which is already in the Giant branch ? IOS will be going with this path in the latest code not with op_wq. Yes, we also saw encode/decode was consuming lot of cpu times and if I remember correctly profiler was pointing bufferlist::append in many of such cases. Thanks & Regards Somnath -----Original Message----- From: Dong Yuan [mailto:yuandong1222@xxxxxxxxx] Sent: Thursday, September 25, 2014 11:31 PM To: Haomai Wang Cc: Sage Weil; ceph-devel@xxxxxxxxxxxxxxx; Somnath Roy; Allen Samuels; dieter.kasper@xxxxxxxxxxxxxx; PVonStamwitz@xxxxxxxxxxxxxx; Shu, Xinxin; Stefan Priebe - Profihost AG; xiaoxi.chen@xxxxxxxxx; Milosz Tanski; zhiqiang.wang@xxxxxxxxx; jianpeng.ma@xxxxxxxxx; gdror@xxxxxxxxxxxx; vuhuong@xxxxxxxxxxxx; Mark Nelson Subject: Re: Weekly performance meeting Some data can support Haomai's points. > 1. encode/decode plays remarkable latency, especially in > ObjectStore::Transaction. I'm urgen in refactor ObjectStore API to > avoid encode/decode codes. It seemed has be signed in note(- remove > serialization from ObjectStore::Transaction (ymmv)) My environment, single OSD on a single SSD with filestore_blackhole = true. With All transaction encode, 10000 4K WriteFull operations by single thread need about 14.3s. While without transaction encode, the same test can be finished in about 11.5s. Considering the FileStore needs to decode the bufferlist too, encode/decode cost more than 20% time! Oprofile results can validate this problem too: methods used by encode/decode sometimes take 9 of the top 10. > 2. obvious latency for threadpool/workqueue model. Do we consider to > impl performance optimization workqueue to replace existing critical > workqueue such as op_wq in OSD.h and op_wq in FileStore.h. Now in my > AsyncMessenger impl, I will try to use custom and simple workqueue > impl to improve performance. When I analyze the latency of a 4K object WriteFull operation, I put static probes into codes to measure times used by OpWQ. I test 10000 4K object WriteFull operations and average the results. I found it spends 158us for the OpWQ for each IO, including 30us to enqueue, 108us in the queue, and 20us to dequeue. It takes more than 20% time of PG layer (not including msg and os layer) when encode is ignored. Maybe a more effective ThreadPool/WorkQueue Model is needed or at least some improvement for WorkQueues in the IO path to reduce the latency. On 26 September 2014 10:27, Haomai Wang <haomaiwang@xxxxxxxxx> wrote: > Thanks for sage! > > I'm on the flight at Oct 1. :-( > > Now my team is mainly worked on the performance of ceph, we have > observed these points: > > 1. encode/decode plays remarkable latency, especially in > ObjectStore::Transaction. I'm urgen in refactor ObjectStore API to > avoid encode/decode codes. It seemed has be signed in note(- remove > serialization from ObjectStore::Transaction (ymmv)) 2. obvious latency > for threadpool/workqueue model. Do we consider to impl performance > optimization workqueue to replace existing critical workqueue such as > op_wq in OSD.h and op_wq in FileStore.h. Now in my AsyncMessenger > impl, I will try to use custom and simple workqueue impl to improve > performance. > 3. Large lock in client library such as ObjectCacher > > > On Fri, Sep 26, 2014 at 2:27 AM, Sage Weil <sweil@xxxxxxxxxx> wrote: >> Hi everyone, >> >> A number of people have approached me about how to get more involved >> with the current work on improving performance and how to better >> coordinate with other interested parties. A few meetings have taken >> place offline with good results but only a few interested parties were involved. >> >> Ideally, we'd like to move as much of this dicussion into the public >> forums: ceph-devel@xxxxxxxxxxxxxxx and #ceph-devel. That isn't >> always sufficient, however. I'd like to also set up a regular weekly >> meeting using google hangouts or bluejeans so that all interested >> parties can share progress. There are a lot of things we can do >> during the Hammer cycle to improve things but it will require some coordination of effort. >> >> Among other things, we can discuss: >> >> - observed performance limitations >> - high level strategies for addressing them >> - proposed patch sets and their performance impact >> - anything else that will move us forward >> >> One challenge is timezones: there are developers in the US, China, >> Europe, and Israel who may want to join. As a starting point, how >> about next Wednesday, 15:00 UTC? If I didn't do my tz math wrong, >> that's >> >> 8:00 (PDT, California) >> 15:00 (UTC) >> 18:00 (IDT, Israel) >> 23:00 (CST, China) >> >> That is surely not the ideal time for everyone but it can hopefully >> be a starting point. >> >> I've also created an etherpad for collecting discussion/agenda items >> at >> >> http://pad.ceph.com/p/performance_weekly >> >> Is there interest here? Please let everyone know if you are actively >> working in this area and/or would like to join, and update the pad >> above with the topics you would like to discuss. >> >> Thanks! >> sage > > > > -- > Best Regards, > > Wheat > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > info at http://vger.kernel.org/majordomo-info.html -- Dong Yuan Email:yuandong1222@xxxxxxxxx ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). ��.n��������+%������w��{.n����z��u���ܨ}���Ơz�j:+v�����w����ޙ��&�)ߡ�a����z�ޗ���ݢj��w�f
