On 6/12/19 11:49 AM, Peter Eisch wrote:
Hi,
Could someone be able to point me to a blog or documentation page which helps me resolve the issues noted below?
All nodes are Luminous, 12.2.12; one realm, one zonegroup (clustered haproxies fronting), two zones (three rgw in each); All endpoint references to each zone go are an haproxy.
In hoping to replace a swift config with RGW it has been interesting. Crafting a functional configuration from blog posts and documentation takes time. It was crucial to find and use http://docs.ceph.com/docs/luminous/radosgw/multisite/ instead of http://docs.ceph.com/docs/master/radosgw/config-ref/ except parts suggest incorrect configurations. I've submitted corrections to the former in #28517, for what it's worth.
Through this I'm now finding fewer resources to help explain the abundance of 404's in the gateway logs:
"GET /admin/log/?type=data&id=8&marker&extra-info=true&rgwx-zonegroup=<zg-guid> HTTP/1.1" 404 0 - -
"GET /admin/log/?type=data&id=8&marker&extra-info=true&rgwx-zonegroup=<zg-guid > HTTP/1.1" 404 0 - -
"GET /admin/log/?type=data&id=8&marker&extra-info=true&rgwx-zonegroup=<zg-guid > HTTP/1.1" 404 0 - -
"GET /admin/log/?type=data&id=8&marker&extra-info=true&rgwx-zonegroup=<zg-guid > HTTP/1.1" 404 0 - -
To the counts of hundreds of thousands. The site seems to work with just minimal testing so far. The 404's also seem to be limited to the data queries while the metadata queries are mostly more successful with 200's.
"GET /admin/log?type=metadata&id=55&period=58b43d07-03e2-48e4-b2dc-74d64ef7f0c9&max-entries=100&&rgwx-zonegroup=<zg-guid> HTTP/1.1" 200 0 - -
"GET /admin/log?type=metadata&id=45&period=58b43d07-03e2-48e4-b2dc-74d64ef7f0c9&max-entries=100&&rgwx-zonegroup==<zg-guid> HTTP/1.1" 200 0 - -
"GET /admin/log?type=metadata&id=4&period=58b43d07-03e2-48e4-b2dc-74d64ef7f0c9&max-entries=100&&rgwx-zonegroup==<zg-guid> HTTP/1.1" 200 0 - -
"GET /admin/log?type=metadata&id=35&period=58b43d07-03e2-48e4-b2dc-74d64ef7f0c9&max-entries=100&&rgwx-zonegroup==<zg-guid> HTTP/1.1" 200 0 - -
Q: How do I address the 404 events to help them succeed?
Hi Peter,
These 404s are not really failures. The replication logs for data and
metadata are spread over several objects (identified by '&id=' in the
requests). If no changes have occurred on a given shard id, that object
never gets created, so other zones will get a 404 when trying to read them.
Other log events which I cannot resolve are the tens of thousands (even while no reads or writes are requested) of:
... meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
... meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
... data sync: ERROR: failed to read remote data log info: ret=-2
... data sync: ERROR: failed to read remote data log info: ret=-2
... meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
... meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
... data sync: ERROR: failed to read remote data log info: ret=-2
... data sync: ERROR: failed to read remote data log info: ret=-2
... data sync: ERROR: failed to read remote data log info: ret=-2
... meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
... etc.
These seem to fire off every 30 seconds but doesn't seem to be managed by "rgw usage log tick interval" nor "rgw init timeout" values. Meanwhile the usage between the two zones matches for each bucket.
Q: What are these log events indicating?
These are the same non-fatal errors from above, where 404 errors from
other zones are converted to -ENOENT. The RGWBackoffControlCR will
continue to poll these objects for changes. These ERROR messages are
unnecessarily spammy though, so I'd be in favor of removing them.
Thanks,
peter
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
