Re: RGW Replication

[Craig Lewis <clewis@xxxxxxxxxxxxxxxxxx>]

 

On 2/4/14 20:02 , Josh Durgin wrote:

From the log it looks like you're hitting the default maximum number of
entries to be processed at once per shard. This was intended to prevent
one really busy shard from blocking progress on syncing other shards,
since the remainder will be synced the next time the shard is processed.
Perhaps the default is too low though, or the idea should be scrapped altogether since you can sync other shards in parallel.

For your particular usage, since you're updating the same few buckets,
the max entries limit is hit constantly. You can increase it with
max-entries: 1000000 in the config file or --max-entries 10000000 on the command line.

Josh

This doesn't appear to have any effect:
root@ceph1c:/etc/init.d# grep max-entries /etc/ceph/radosgw-agent.conf
max-entries: 1000000
root@ceph1c:/etc/init.d# egrep 'has [0-9]+ entries after' /var/log/ceph/radosgw-agent.log  | tail -1
2014-02-05T13:11:03.915 2743:INFO:radosgw_agent.worker:bucket instance "live-2:us-west-1.35026898.2" has 1000 entries after "00000206789.410535.3"

Neither does --max-entries 10000000:
root@ceph1c:/etc/init.d# ps auxww | grep radosgw-agent | grep max-entries
root     19710  6.0  0.0  74492 18708 pts/3    S    13:22   0:00 /usr/bin/python /usr/bin/radosgw-agent --incremental-sync-delay=10 --max-entries 10000000 -c /etc/ceph/radosgw-agent.conf
root@ceph1c:/etc/init.d# egrep 'has [0-9]+ entries after' /var/log/ceph/radosgw-agent.us-west-1.us-central-1.log  | tail -1
2014-02-05T13:22:58.577 21626:INFO:radosgw_agent.worker:bucket instance "live-2:us-west-1.35026898.2" has 1000 entries after "00000207788.411542.2"


I guess I'll look into that too, since I'll be in that area of the code.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com