On 02/04/2014 07:44 PM, Craig Lewis wrote:
On 2/4/14 17:06 , Craig Lewis wrote:
On 2/4/14 14:43 , Yehuda Sadeh wrote:
Does it ever catching up? You mentioned before that most of the writes
went to the same two buckets, so that's probably one of them. Note
that writes to the same bucket are being handled in-order by the
agent.
Yehuda
... I think so. This is what my graph looks like:
I think it's still catching up, despite the graph. radosgw-admin bucket
stats shows more objects are being created in the slave zone than the
master zone during the same time period. Not a large number though. If
it is catching up, it'll take months at this rate. It's not entirely
clear, because the slave zone trails the master, but it's been pretty
consistent for the past hour.
It's not doing all of the missing objects though. I have a bucket that
I stopped writing to a few days ago. The slave is missing ~500k
objects, and none have been added to the slave in the past hour.
You can run
$ radosgw-admin bilog list --bucket=<bucket> --marker=<id>
E.g.,
$ radosgw-admin bilog list --bucket=live-2 --marker=00000127871.328492.2
The entries there should have timestamp info.
Thanks! I'll see what I can figure out.
From the log it looks like you're hitting the default maximum number of
entries to be processed at once per shard. This was intended to prevent
one really busy shard from blocking progress on syncing other shards,
since the remainder will be synced the next time the shard is processed.
Perhaps the default is too low though, or the idea should be scrapped
altogether since you can sync other shards in parallel.
For your particular usage, since you're updating the same few buckets,
the max entries limit is hit constantly. You can increase it with
max-entries: 1000000 in the config file or --max-entries 10000000 on the
command line.
Josh
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
