I know FileStore.ondisk_finisher handle C_OSD_OpCommit , and from "journaled_completion_queue" to "op_commit" cost 3.6 seconds, maybe cost in the function of ReplicatedPG::op_commit . Through OpTracker , I find that ReplicatedPG::op_commit first lock pg, but it sometimes cost from 0.5 to 1 second , so the whole ondisk_finisher must wait 1 second. How can cancel pg lock in the ReplicatedPG::op_commit ? thanks baijiaruo at 126.com ???? Guang Yang ????? 2014-07-01 11:39 ???? baijiaruo ??? ceph-users ??? Re: [ceph-users] Ask a performance question for the RGW On Jun 30, 2014, at 3:59 PM, baijiaruo at 126.com wrote: > Hello, > thanks for you answer the question. > But when there are less than 50 thousand objects, and latency is very big . I see the write ops for the bucket index object.? from "journaled_completion_queue" to "op_commit" cost 3.6 seconds?this mean that from ?writing journal finish? to "op_commit" cost 3.6 seconds? > so I can't understand this and what happened? The operations updating the same bucket index object get serialized, one possibility is that those operation was hang there waiting other ops finishing their work. > > thanks > baijiaruo at 126.com > > ???? Guang Yang > ????? 2014-06-30 14:57 > ???? baijiaruo > ??? ceph-users > ??? Re: [ceph-users] Ask a performance question for the RGW > Hello, > There is a known limitation of bucket scalability, and there is a blueprint tracking it - https://wiki.ceph.com/Planning/Blueprints/Submissions/rgw%3A_bucket_index_scalability. > > At time being, I would recommend to do sharding at application level (create multiple buckets) to workaround this limitation. > > Thanks, > Guang > > On Jun 30, 2014, at 2:54 PM, baijiaruo at 126.com wrote: > > > > > hello, everyone! > > > > when I user rest bench test RGW performance and the cmd is: > > ./rest-bench --access-key=ak --secret=sk --bucket=bucket_name --seconds=600 -t 200 -b 524288 -no-cleanup write > > > > test result: > > Total time run: 362.962324 T > > otal writes made: 48189 > > Write size: 524288 > > Bandwidth (MB/sec): 66.383 > > Stddev Bandwidth: 40.7776 > > Max bandwidth (MB/sec): 173 > > Min bandwidth (MB/sec): 0 > > Average Latency: 1.50435 > > Stddev Latency: 0.910731 > > Max latency: 9.12276 > > Min latency: 0.19867 > > > > my environment is 4 host and 40 disk?osd?? but test result is very bad? average latency is 1.5 seconds ?and I find write obj metadate is very slowly?because it puts so many object to one bucket? we know writing object metadate can call method ?bucket_prepare_op?,and test find this op is very slowly? I find the osd which contain bucket-obj?and see the ?bucket_prepare_op?by dump_historic_ops : > > { "description": "osd_op(client.4742.0:87613 .dir.default.4243.3 [call rgw.bucket_prepare_op] 3.3670fe74 e317)", > > "received_at": "2014-06-30 13:35:55.409597", > > "age": "51.148026", > > "duration": "4.130137", > > "type_data": [ > > "commit sent; apply or cleanup", > > { "client": "client.4742", > > "tid": 87613}, > > [ > > { "time": "2014-06-30 13:35:55.409660", > > "event": "waiting_for_osdmap"}, > > { "time": "2014-06-30 13:35:55.409669", > > "event": "queue op_wq"}, > > { "time": "2014-06-30 13:35:55.896766", > > "event": "reached_pg"}, > > { "time": "2014-06-30 13:35:55.896793", > > "event": "started"}, > > { "time": "2014-06-30 13:35:55.896796", > > "event": "started"}, > > { "time": "2014-06-30 13:35:55.899450", > > "event": "waiting for subops from [40,43]"}, > > { "time": "2014-06-30 13:35:55.899757", > > "event": "commit_queued_for_journal_write"}, > > { "time": "2014-06-30 13:35:55.899799", > > "event": "write_thread_in_journal_buffer"}, > > { "time": "2014-06-30 13:35:55.899910", > > "event": "journaled_completion_queued"}, > > { "time": "2014-06-30 13:35:55.899936", > > "event": "journal first callback"}, > > { "time": "2014-06-30 13:35:55.899944", > > "event": "queuing ondisk"}, > > { "time": "2014-06-30 13:35:56.142104", > > "event": "sub_op_commit_rec"}, > > { "time": "2014-06-30 13:35:56.176950", > > "event": "sub_op_commit_rec"}, > > { "time": "2014-06-30 13:35:59.535301", > > "event": "op_commit"}, > > { "time": "2014-06-30 13:35:59.535331", > > "event": "commit_sent"}, > > { "time": "2014-06-30 13:35:59.539723", > > "event": "op_applied"}, > > { "time": "2014-06-30 13:35:59.539734", > > "event": "done"}]]}, > > > > so why from "journaled_completion_queued" to "op_commit" is very slowly, and what happened? > > thanks > > > > baijiaruo at 126.com > > _______________________________________________ > > ceph-users mailing list > > ceph-users at lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140701/d4b3ed3c/attachment.htm>
 |