Hello All,
I have recently had 32 large map objects appear in my default.rgw.log pool. Running luminous 12.2.8.
Not sure what to think about these. I’ve done a lot of reading about how when these normally occur it is related to a bucket needing resharding, but it doesn’t look like my default.rgw.log pool has anything in it, let alone buckets. Here’s some info on the system:
[root@elm-rgw01 ~]# ceph versions
{
"mon": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 5
},
"mgr": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 1
},
"osd": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 192
},
"mds": {},
"rgw": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 1
},
"overall": {
"ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0) luminous (stable)": 199
}
}
[root@elm-rgw01 ~]# ceph osd pool ls
.rgw.root
default.rgw.control
default.rgw.meta
default.rgw.log
default.rgw.buckets.index
default.rgw.buckets.non-ec
default.rgw.buckets.data
[root@elm-rgw01 ~]# ceph health detail
HEALTH_WARN 32 large omap objects
LARGE_OMAP_OBJECTS 32 large omap objects
32 large objects found in pool 'default.rgw.log'
Search the cluster log for 'Large omap object found' for more details.—
Looking closer at these object they are all of size 0. Also that pool shows a capacity usage of 0:
(just a sampling of the 236 objects at size 0)
[root@elm-mon01 ceph]# for i in `rados ls -p default.rgw.log`; do echo ${i}; rados stat -p default.rgw.log ${i};done
obj_delete_at_hint.0000000078
default.rgw.log/obj_delete_at_hint.0000000078 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000070
default.rgw.log/obj_delete_at_hint.0000000070 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000104
default.rgw.log/obj_delete_at_hint.0000000104 mtime 2019-03-07 11:39:20.000000, size 0
obj_delete_at_hint.0000000026
default.rgw.log/obj_delete_at_hint.0000000026 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000028
default.rgw.log/obj_delete_at_hint.0000000028 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000040
default.rgw.log/obj_delete_at_hint.0000000040 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000015
default.rgw.log/obj_delete_at_hint.0000000015 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000069
default.rgw.log/obj_delete_at_hint.0000000069 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000095
default.rgw.log/obj_delete_at_hint.0000000095 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000003
default.rgw.log/obj_delete_at_hint.0000000003 mtime 2019-03-07 11:39:19.000000, size 0
obj_delete_at_hint.0000000047
default.rgw.log/obj_delete_at_hint.0000000047 mtime 2019-03-07 11:39:19.000000, size 0
[root@elm-mon01 ceph]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR
.rgw.root 1.09KiB 4 0 12 0 0 0 14853 9.67MiB 0 0B
default.rgw.buckets.data 444TiB 166829939 0 1000979634 0 0 0 362357590 859TiB 909188749 703TiB
default.rgw.buckets.index 0B 358 0 1074 0 0 0 729694496 1.04TiB 522654976 0B
default.rgw.buckets.non-ec 0B 182 0 546 0 0 0 194204616 148GiB 97962607 0B
default.rgw.control 0B 8 0 24 0 0 0 0 0B 0 0B
default.rgw.log 0B 236 0 708 0 0 0 33268863 3.01TiB 18415356 0B
default.rgw.meta 16.2KiB 67 0 201 0 0 0 466663427 371GiB 27647 146KiB
total_objects 166830794
total_used 668TiB
total_avail 729TiB
total_space 1.36PiB
Does anyone know the importance of these objects or if they can safely be deleted?
Thank you,
Sam Liston (sam.liston@xxxxxxxx)
========================================
Center for High Performance Computing
155 S. 1452 E. Rm 405
Salt Lake City, Utah 84112 (801)232-6932
========================================
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
