I was able to remove the meta objects, but the cluster is still in WARN state
HEALTH_WARN 1 large omap objects
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool 'us-prd-1.rgw.log'
Search the cluster log for 'Large omap object found' for more details.
How do I go about clearing it out? I don't see any other references to large omap in any of the logs. I've tried restarted the mgr's, the monitors, and even the osd that reported the issue.
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool 'us-prd-1.rgw.log'
Search the cluster log for 'Large omap object found' for more details.
How do I go about clearing it out? I don't see any other references to large omap in any of the logs. I've tried restarted the mgr's, the monitors, and even the osd that reported the issue.
-Brett
On Thu, Jul 25, 2019 at 2:55 PM Brett Chancellor <bchancellor@xxxxxxxxxxxxxx> wrote:
14.2.1Thanks, I'll try that.On Thu, Jul 25, 2019 at 2:54 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote:What ceph version is this cluster running? Luminous or later should not
be writing any new meta.log entries when it detects a single-zone
configuration.
I'd recommend editing your zonegroup configuration (via 'radosgw-admin
zonegroup get' and 'put') to set both log_meta and log_data to false,
then commit the change with 'radosgw-admin period update --commit'.
You can then delete any meta.log.* and data_log.* objects from your log
pool using the rados tool.
On 7/25/19 2:30 PM, Brett Chancellor wrote:
> Casey,
> These clusters were setup with the intention of one day doing multi
> site replication. That has never happened. The cluster has a single
> realm, which contains a single zonegroup, and that zonegroup contains
> a single zone.
>
> -Brett
>
> On Thu, Jul 25, 2019 at 2:16 PM Casey Bodley <cbodley@xxxxxxxxxx
> <mailto:cbodley@xxxxxxxxxx>> wrote:
>
> Hi Brett,
>
> These meta.log objects store the replication logs for metadata
> sync in
> multisite. Log entries are trimmed automatically once all other zones
> have processed them. Can you verify that all zones in the multisite
> configuration are reachable and syncing? Does 'radosgw-admin sync
> status' on any zone show that it's stuck behind on metadata sync?
> That
> would prevent these logs from being trimmed and result in these large
> omap warnings.
>
> On 7/25/19 1:59 PM, Brett Chancellor wrote:
> > I'm having an issue similar to
> >
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033611.html .
>
> > I don't see where any solution was proposed.
> >
> > $ ceph health detail
> > HEALTH_WARN 1 large omap objects
> > LARGE_OMAP_OBJECTS 1 large omap objects
> > 1 large objects found in pool 'us-prd-1.rgw.log'
> > Search the cluster log for 'Large omap object found' for
> more details.
> >
> > $ grep "Large omap object" /var/log/ceph/ceph.log
> > 2019-07-25 14:58:21.758321 osd.3 (osd.3) 15 : cluster [WRN]
> Large omap
> > object found. Object:
> > 51:61eb35fe:::meta.log.e557cf47-46df-4b45-988e-9a94c5004a2e.19:head
> > Key count: 3382154 Size (bytes): 611384043
> >
> > $ rados -p us-prd-1.rgw.log listomapkeys
> > meta.log.e557cf47-46df-4b45-988e-9a94c5004a2e.19 |wc -l
> > 3382154
> >
> > $ rados -p us-prd-1.rgw.log listomapvals
> > meta.log.e557cf47-46df-4b45-988e-9a94c5004a2e.19
> > This returns entries from almost every bucket, across multiple
> > tenants. Several of the entries are from buckets that no longer
> exist
> > on the system.
> >
> > $ ceph df |egrep 'OBJECTS|.rgw.log'
> > POOL ID STORED OBJECTS USED %USED MAX
> > AVAIL
> > us-prd-1.rgw.log 51 758 MiB 228 758 MiB
> > 0 102 TiB
> >
> > Thanks,
> >
> > -Brett
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
