|
Hi all,
We have a large omap object warning on one of our Ceph clusters.
The only reports I've seen regarding the "large omap objects" warning from other users were related to RGW bucket sharding, however we do not have RGW configured on this cluster.
The large omap object ~10GB resides in a CephFS metadata pool.
It's perhaps worth mentioning that we had to perform disaster recovery steps [1] on this cluster last year after a network issue, so we're not sure whether this large omap object is a result of those previous recovery processes or whether it's completely
unrelated.
Ceph version: 12.2.8
osd_objectstore: Bluestore
RHEL 7.5
Kernel: 4.4.135-1.el7.elrepo.x86_64
We have set: "mds_bal_fragment_size_max": "500000" (Default 100000)
$ ceph health detail
HEALTH_WARN 1 large omap objects
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool 'hpcfs_metadata'
Search the cluster log for 'Large omap object found' for more details.
# Find pg with large omap object
$ for i in `ceph pg ls-by-pool hpcfs_metadata | tail -n +2 | awk '{print $1}'`; do echo -n "$i: "; ceph pg $i query |grep num_large_omap_objects | head -1 | awk '{print $2}'; done | grep ": 1"
20.103: 1
# OSD log entry showing relevant object
osd.143 osd.143 172.26.74.23:6826/3428317 1380 : cluster [WRN] Large omap object found. Object: 20:c0ce80d4:::4.00000000:head Key count: 24698995 Size (bytes): 11410935690
# Confirm default warning thresholds for large omap object
$ ceph daemon osd.143 config show | grep osd_deep_scrub_large_omap
"osd_deep_scrub_large_omap_object_key_threshold": "2000000",
"osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824",
# Dump keys/values of problematic object, creates 46.65GB file
$ rados -p hpcfs_metadata listomapvals '4.00000000' > /tmp/hpcfs_metadata_object_omap_vals_4.00000000_20190304
$ ll /tmp/hpcfs_metadata_object_omap_vals_4.00000000_20190304
-rw-r--r-- 1 root root 50089561860 Mar 4 18:16 /tmp/hpcfs_metadata_object_omap_vals_4.00000000_20190304
# Confirm key count matches OSD log entry warning
$ rados -p hpcfs_metadata listomapkeys '4.00000000' | wc -l
24698995
# The omap keys/vals for that object appear to have been unchanged/static for at least a couple of months:
$ sha1sum /tmp/hpcfs_metadata_object_omap_vals_4.00000000_20190304
fd00ceb68607b477626178b2d81fefb926460107 /tmp/hpcfs_metadata_object_omap_vals_4.00000000_20190304
$ sha1sum /tmp/hpcfs_metadata_object_omap_vals_4_00000000_20190108
fd00ceb68607b477626178b2d81fefb926460107 /tmp/hpcfs_metadata_object_omap_vals_4_00000000_20190108
I haven't gone through all 24698995 keys yet, but while most appear to relate to objects in the hpcfs_data CephFS data pool, there are a significant number of keys (rough guess 25%) that don't appear to have corresponding objects in the hpcfs_data pool.
Any assistance or pointers to troubleshoot further would be very much appreciated.
Thanks,
Dylan
|
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
