Looks like the usage log (radosgw-admin usage show), how often do you trim it? -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Oct 14, 2019 at 11:55 PM Troy Ablan <tablan@xxxxxxxxx> wrote: > > Hi folks, > > Mimic cluster here, RGW pool with only default zone. I have a > persistent error here > > LARGE_OMAP_OBJECTS 1 large omap objects > > 1 large objects found in pool 'default.rgw.log' > > Search the cluster log for 'Large omap object found' for more > details. > > I think I've narrowed it down to this namespace: > > -[~:#]- rados -p default.rgw.log --namespace usage ls > > > > > > usage.17 > usage.19 > usage.24 > > -[~:#]- rados -p default.rgw.log --namespace usage listomapkeys usage.17 > | wc -l > 21284 > > -[~:#]- rados -p default.rgw.log --namespace usage listomapkeys usage.19 > | wc -l > > > > > 1355968 > > -[~:#]- rados -p default.rgw.log --namespace usage listomapkeys usage.24 > | wc -l > > > > > 0 > > > Now, this last one take a long time to return -- minutes, even with a 0 > response, and listomapvals indeed returns a very large amount of data. > I'm doing `wc -c` on listomapvals but this hasn't returned at the time > of writing this message. Is there anything I can do about this? > > Whenever PGs on the default.rgw.log are recovering or backfilling, my > RGW cluster appears to block writes for almost two hours, and I think it > points to this object, or at least this pool. > > I've been having trouble finding any documentation about how this log > pool is used by RGW. I have a feeling updating this object happens on > every write to the cluster. How would I remove this bottleneck? Can I? > > Thanks > > -Troy > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |