On Tue, Nov 26, 2013 at 1:27 PM, Dominik Mostowiec <dominikmostowiec@xxxxxxxxx> wrote: > Hi, > We have 2 clusters with copy of objects. > On one of them we splited all large buckets (largest 17mln objects) to > 256 buckets each (shards) and we have added 3 extra servers (6->9). > Old bucket was created in ceph argonaut. > Now we have dumpling. > After this operation get avg time increased almost twice: > https://www.dropbox.com/s/lbrpk6ias6r4459/avg_get_time.PNG > I found that also increased 'get_initial_lat' > Sum of this parametr on all servers in cluster: > https://www.dropbox.com/s/l90a43vf9ivw639/get_initial_lat_sum.png > > Do you have an idea what is the reason? > Where can I start to search? > Are you running latest dumpling? There was some performance related issue that was fixed fairly recently. I'd start with trying to identify whether the issue is in the osd or in the gateway itself. Try turning on some logging (e.g., debug ms = 1, debug rgw = 20), and try to see where most of the request processing time is spent (within the gateway, or is it just waiting for the osd?). Yehuda -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
