Hi, all I've experimented for a while with mClock QoS and didn't get any visible impact of QoS on speed/bandwidth of operations of different types (client, recovery). I suppose It's enough to have single PC with single drive to check how it works. I ran my tests on PC with 500Gb SSD, 32Gb RAM, I7 CPU. 1) Changes in ceph.conf: # double replication in any case osd pool default size = 2 osd pool default min size = 2 # mclock queue, put client and recovery operations in one queue osd op queue = mclock_client osd op queue cut off = high # reserve iops and higher weight for recovery osd_op_queue_mclock_client_op_wgt = 1.0 osd_op_queue_mclock_osd_subop_wgt = 1.0 osd_op_queue_mclock_recov_res = 500.0 osd_op_queue_mclock_recov_wgt = 9.0 # lower number of shards will increase the impact of the mClock queues osd_op_num_shards = 1 osd_op_num_threads_per_shard = 1 # accelerate recovery operations osd max backfills = 1000 osd recovery max active = 1000 # some option to slow down storage filestore queue max ops = 10 bluestore_throttle_bytes = 100 bluestore_throttle_deferred_bytes = 1000 bluestore_throttle_cost_per_io = 100 2) Run Ceph with 4 OSD: RGW=1 OSD=4 ../src/vstart.sh -n -X -l --filestore # or --bluestore 3) Set minimal weight for half of OSDs: for i in 0 1; do ceph osd reweight $i 0.1; done 4) Create new pool and write some data: ceph osd pool create scbench 128 128 rados bench -p scbench 300 write --no-cleanup -b 100K -t 1000 5) Reweight all of OSDs, recovery will be started: for i in 0 1; do ceph osd reweight $i 1; done for i in 2 3; do ceph osd reweight $i 0.1; done 6) Start writing another part of data: rados bench -p scbench 300 write --no-cleanup -b 100K -t 1000 7) Check and compare speed of client and recovery IO operations: watch -n1 ceph -s During next 5-10 minutes speed of recovery and client ops changes dramatically. Sometimes recovery ops goes faster, sometimes slower than client ops. Sometimes recovery can stop at all. But in general client ops goes much faster than recovery ops despite the mclock setttings. Why speed/bandwidth of operations don't divides according to mclock settings? Probably, my test is not correct? How to check work of mClock QoS? Any help is appreciated. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
