Hi Brett,
These meta.log objects store the replication logs for metadata sync in
multisite. Log entries are trimmed automatically once all other zones
have processed them. Can you verify that all zones in the multisite
configuration are reachable and syncing? Does 'radosgw-admin sync
status' on any zone show that it's stuck behind on metadata sync? That
would prevent these logs from being trimmed and result in these large
omap warnings.
On 7/25/19 1:59 PM, Brett Chancellor wrote:
I'm having an issue similar to
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033611.html ;.
I don't see where any solution was proposed.
$ ceph health detail
HEALTH_WARN 1 large omap objects
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool 'us-prd-1.rgw.log'
Search the cluster log for 'Large omap object found' for more details.
$ grep "Large omap object" /var/log/ceph/ceph.log
2019-07-25 14:58:21.758321 osd.3 (osd.3) 15 : cluster [WRN] Large omap
object found. Object:
51:61eb35fe:::meta.log.e557cf47-46df-4b45-988e-9a94c5004a2e.19:head
Key count: 3382154 Size (bytes): 611384043
$ rados -p us-prd-1.rgw.log listomapkeys
meta.log.e557cf47-46df-4b45-988e-9a94c5004a2e.19 |wc -l
3382154
$ rados -p us-prd-1.rgw.log listomapvals
meta.log.e557cf47-46df-4b45-988e-9a94c5004a2e.19
This returns entries from almost every bucket, across multiple
tenants. Several of the entries are from buckets that no longer exist
on the system.
$ ceph df |egrep 'OBJECTS|.rgw.log'
POOL ID STORED OBJECTS USED %USED MAX
AVAIL
us-prd-1.rgw.log 51 758 MiB 228 758 MiB
0 102 TiB
Thanks,
-Brett
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
