Hi,
I’m suffering with our large omap object in the cluster since my colleague updated from jewel to luminous 12.2.8. This warn is still here more than a year ago.
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool 'default.rgw.log'
Search the cluster log for 'Large omap object found' for more details.
Based on the articles that I’ve read it is due to the usage logs. On the weekend I’ve set the rgw_enable_usage_log = false and restarted all the 3 rgw services.
I trim the logs as some people on the internet suggested, but it is super slow. I trim the logs month by month like this:
radosgw-admin usage trim --end-date=2019-09-01 --yes-i-really-mean-it
This command took yesterday more than a day to finish :/
I hope at least it is not increasing anymore since I turned it to false.
The luminous default thresholds are:
Size threshold: 1gb
Key threshold: 2000000
Mine is: 7gb and 41246592
2020-08-16 09:21:11.549128 osd.45 osd.45 10.118.191.247:6824/2262114 2 : cluster [WRN] Large omap object found. Object: 4:b172cd59:usage::usage.26:head Key count: 41246592 Size (bytes): 7736069974
Based on this command here is my large omap: for i in `ceph pg ls-by-pool default.rgw.log | tail -n +2 | awk '{print $1}'`; do echo -n "$i: "; ceph pg $i query |grep num_large_omap_objects | head -1 | awk '{print $2}'; done | grep ": 1"
4.d: 1
I’ve initiated a deep scrub after it and it is still running after 12 hours and make stuck pgs:
health: HEALTH_ERR
3 stuck requests are blocked > 4096 sec. Implicated osds 45
It stuck for couple of secs, and after go back to normal, but this is continously.
At the moment no impact yet, but I think it will have soon.
What should I do? What is the best and easiest way to fix my issue?
We have serious impact with the moving of this pg when an osd crash because I guess it is big and not healthy.
Here are the omap list:
for i in `rados -p default.rgw.log ls`; do echo -n “$i:“; rados -p default.rgw.log listomapkeys $i |wc -l; done > /tmp/omapkeys
cat /tmp/omapkeys
“obj_delete_at_hint.0000000078:“0
“meta.history:“0
“obj_delete_at_hint.0000000070:“0
“obj_delete_at_hint.0000000104:“0
“obj_delete_at_hint.0000000026:“0
“obj_delete_at_hint.0000000028:“0
“obj_delete_at_hint.0000000040:“0
“obj_delete_at_hint.0000000015:“0
“obj_delete_at_hint.0000000069:“0
“obj_delete_at_hint.0000000095:“0
“obj_delete_at_hint.0000000003:“0
“obj_delete_at_hint.0000000047:“0
“obj_delete_at_hint.0000000035:“0
“obj_delete_at_hint.0000000037:“0
“obj_delete_at_hint.0000000024:“0
“obj_delete_at_hint.0000000000:“0
“obj_delete_at_hint.0000000031:“0
“obj_delete_at_hint.0000000076:“0
“obj_delete_at_hint.0000000113:“0
“obj_delete_at_hint.0000000005:“0
“obj_delete_at_hint.0000000011:“0
“obj_delete_at_hint.0000000058:“0
“obj_delete_at_hint.0000000052:“0
“obj_delete_at_hint.0000000088:“0
“obj_delete_at_hint.0000000080:“0
“obj_delete_at_hint.0000000090:“0
“obj_delete_at_hint.0000000110:“0
“obj_delete_at_hint.0000000096:“0
“obj_delete_at_hint.0000000087:“0
“obj_delete_at_hint.0000000008:“0
“obj_delete_at_hint.0000000006:“0
“obj_delete_at_hint.0000000029:“0
“obj_delete_at_hint.0000000089:“0
“obj_delete_at_hint.0000000022:“0
“bilog.trim:“0
“obj_delete_at_hint.0000000016:“0
“obj_delete_at_hint.0000000041:“0
“obj_delete_at_hint.0000000018:“0
“obj_delete_at_hint.0000000092:“0
“obj_delete_at_hint.0000000014:“0
“obj_delete_at_hint.0000000112:“0
“obj_delete_at_hint.0000000007:“0
“obj_delete_at_hint.0000000021:“0
“obj_delete_at_hint.0000000064:“0
“obj_delete_at_hint.0000000071:“0
“obj_delete_at_hint.0000000074:“0
“obj_delete_at_hint.0000000081:“0
“obj_delete_at_hint.0000000009:“0
“obj_delete_at_hint.0000000121:“0
“obj_delete_at_hint.0000000125:“0
“obj_delete_at_hint.0000000082:“0
“obj_delete_at_hint.0000000105:“0
“obj_delete_at_hint.0000000059:“0
“obj_delete_at_hint.0000000077:“0
“obj_delete_at_hint.0000000032:“0
“obj_delete_at_hint.0000000053:“0
“obj_delete_at_hint.0000000091:“0
“obj_delete_at_hint.0000000065:“0
“obj_delete_at_hint.0000000083:“0
“obj_delete_at_hint.0000000010:“0
“obj_delete_at_hint.0000000045:“0
“obj_delete_at_hint.0000000002:“0
“obj_delete_at_hint.0000000116:“0
“obj_delete_at_hint.0000000034:“0
“obj_delete_at_hint.0000000101:“0
“obj_delete_at_hint.0000000079:“0
“obj_delete_at_hint.0000000049:“0
“obj_delete_at_hint.0000000117:“0
“obj_delete_at_hint.0000000044:“0
“obj_delete_at_hint.0000000066:“0
“obj_delete_at_hint.0000000068:“0
“obj_delete_at_hint.0000000085:“0
“obj_delete_at_hint.0000000073:“0
“obj_delete_at_hint.0000000038:“0
“obj_delete_at_hint.0000000118:“0
“obj_delete_at_hint.0000000036:“0
“obj_delete_at_hint.0000000103:“0
“obj_delete_at_hint.0000000119:“0
“obj_delete_at_hint.0000000098:“0
“obj_delete_at_hint.0000000027:“0
“obj_delete_at_hint.0000000019:“0
“obj_delete_at_hint.0000000039:“0
“obj_delete_at_hint.0000000100:“0
“obj_delete_at_hint.0000000093:“0
“obj_delete_at_hint.0000000004:“0
“obj_delete_at_hint.0000000063:“0
“obj_delete_at_hint.0000000122:“0
“obj_delete_at_hint.0000000057:“0
“obj_delete_at_hint.0000000054:“0
“obj_delete_at_hint.0000000114:“0
“data_log.0:“0
“obj_delete_at_hint.0000000012:“0
“obj_delete_at_hint.0000000084:“0
“obj_delete_at_hint.0000000043:“0
“obj_delete_at_hint.0000000111:“0
“obj_delete_at_hint.0000000048:“0
“obj_delete_at_hint.0000000020:“0
“obj_delete_at_hint.0000000099:“0
“obj_delete_at_hint.0000000056:“0
“obj_delete_at_hint.0000000072:“0
“obj_delete_at_hint.0000000062:“0
“obj_delete_at_hint.0000000115:“0
“obj_delete_at_hint.0000000033:“0
“obj_delete_at_hint.0000000025:“0
“obj_delete_at_hint.0000000123:“0
“obj_delete_at_hint.0000000108:“0
“obj_delete_at_hint.0000000094:“0
“obj_delete_at_hint.0000000060:“0
“obj_delete_at_hint.0000000109:“0
“obj_delete_at_hint.0000000013:“0
“obj_delete_at_hint.0000000051:“0
“obj_delete_at_hint.0000000106:“0
“obj_delete_at_hint.0000000017:“0
“obj_delete_at_hint.0000000023:“0
“obj_delete_at_hint.0000000107:“0
“obj_delete_at_hint.0000000046:“0
“obj_delete_at_hint.0000000067:“0
“obj_delete_at_hint.0000000086:“0
“obj_delete_at_hint.0000000050:“0
“obj_delete_at_hint.0000000102:“0
“obj_delete_at_hint.0000000124:“0
“obj_delete_at_hint.0000000001:“0
“obj_delete_at_hint.0000000126:“0
“obj_delete_at_hint.0000000055:“0
“obj_delete_at_hint.0000000061:“0
“obj_delete_at_hint.0000000097:“0
“obj_delete_at_hint.0000000042:“0
“obj_delete_at_hint.0000000075:“0
“obj_delete_at_hint.0000000120:“0
“obj_delete_at_hint.0000000030:“0
Thank you
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
