Hi Eric, We applied dmclock in ceph, but in actual environments only the limitation worked while the reservation and proportion had no real effect at all. After careful analysis, we find that the dmclock algrithm developed from the mclock algrithm, has great defects in theory. First of all , a corepremise of attaining reservation is that all requests which should be scheduled in reservation phase have been completed before it going into proportion phase.This means if the reservation heap has requests with R-tag smaller than current time, it should not process requests in proportion phase. It is very easy to get this point in single server. But in multi-server situation, some servers are in reservation phase while some in proportion phase. So the reservation can not be guaranteed at all. Second of all, a similar problem remains for proportion. Let's consider an extreme situation . There are 2 servers ,S1 and S2;2 clients C1,C2. All of requests from C1 go to S1 while all requests from C2 go to S2. In this case, proportion cannot be guaranteed . And requests are not equally distributed in real wolrd like ceph, which leads to the failure in proportion. So in conclusion,we think that dmclock can only work for limitation in theory. So what about your idea about our conclusion ? -----邮件原件----- 发件人: J. Eric Ivancich [mailto:ivancich@xxxxxxxxxx] 发送时间: 2017年6月13日 22:12 收件人: lijie 11803 (RD); Users, Ceph 主题: Re: About dmClock tests confusion after integrating dmClock QoS library into ceph codebase On 06/02/2017 03:49 AM, Lijie wrote: > Hi Eric, > > Our team has developed QOS feature on ceph using the dmclock library > from community. We treat a rbd as a dmclock client instead of pool as . > We tested our code and the result is confusing . > > > > Testing environment: single server with 16 cores , RAM of 32G, 8 > non-systerm disks,each runs one OSD. We set > osd_op_num_threads_per_shard=2, osd_op_num_shards=5 on each OSD. The > Size of our RBD is 100G. The qos params of RBD is （r:1K, p:100, l:2k）, > rbd_cache = false , allow_limit_break = false. > > > > Conclusion: we only got 1500 IOPS while the system serves much more > than the limit value 2000. > > We used fio and adjusted osd_op_num_shards .And we found that iops > runs bigger along as the growth of osd_op_num_shards. Finally it can > break the limit value .( fio iops = 2300 while osd_op_num_shards = 20) . > > > > So would you mind to provide your test environment and result about > dmclock library? And we would be appreciated if you offer some other > direction . It will really be a great help for us. Are you using src/common/mClockPriorityQueue.h? Because it makes sure allow_limit_break is true and asserts that as long as the dmclock queue is not empty, it will retrieve a request when pull_request is called (assert(pr.is_retn());). If allow_limit_break is false, that assert can fail. I'm guessing you've created your own interface to the dmclock library, because of the issue outlined above. The dmclock library has a simulator built on top of it, which you can use to design various scenarios and see how it works. You can run it, for example, by: $ git clone git@xxxxxxxxxx:ceph/dmclock.git $ cd dmclock/ $ mkdir build $ cd build $ cmake .. $ make dmclock-sims $ sim/dmc_sim -c ../sim/dmc_sim_100th.conf where the file dmc_sim_100th.conf describes the scenario being simulated. I'm not fully clear I understand your approach. If you can describe it in more detail, I'll try to address any questions. Eric ------------------------------------------------------------------------------------------------------------------------------------- 本邮件及其附件含有新华三技术有限公司的保密信息，仅限于发送给上面地址中列出 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 邮件！ This e-mail and its attachments contain confidential information from New H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |