Some more information: HGK is the master, ASH and SGP is the secondary, let me show 1 shard in all DCs (FYI the bucket has been deleted which is relates to this bucket index). HKG and ASH give back empty command output for this: rados -p hkg or ash.rgw.buckets.index listomapvals .dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.5411362.1.23219 However in SGP it's full with values like this: ycsappfeed-images/area_id/572303/city_id/23053/country_id/35/datadate/2021-01-28/language_id/1/white_label_id/1/report_id/8/report#STD.jpg value (333 bytes) : 00000000 08 03 47 01 00 00 8a 00 00 00 79 63 73 61 70 70 |..G.......ycsapp| 00000010 66 65 65 64 2d 69 6d 61 67 65 73 2f 61 72 65 61 |feed-images/area| 00000020 5f 69 64 2f 35 37 32 33 30 33 2f 63 69 74 79 5f |_id/572303/city_| 00000030 69 64 2f 32 33 30 35 33 2f 63 6f 75 6e 74 72 79 |id/23053/country| 00000040 5f 69 64 2f 33 35 2f 64 61 74 61 64 61 74 65 2f |_id/35/datadate/| 00000050 32 30 32 31 2d 30 31 2d 32 38 2f 6c 61 6e 67 75 |2021-01-28/langu| 00000060 61 67 65 5f 69 64 2f 31 2f 77 68 69 74 65 5f 6c |age_id/1/white_l| 00000070 61 62 65 6c 5f 69 64 2f 31 2f 72 65 70 6f 72 74 |abel_id/1/report| 00000080 5f 69 64 2f 38 2f 72 65 70 6f 72 74 23 53 54 44 |_id/8/report#STD| 00000090 2e 6a 70 67 2e 7f 32 00 00 00 00 00 01 07 03 61 |.jpg..2........a| 000000a0 00 00 00 01 1c 1d 00 00 00 00 00 00 33 4c 1a 60 |............3L.`| 000000b0 79 81 72 0b 20 00 00 00 65 31 35 66 65 37 32 32 |y.r. ...e15fe722| 000000c0 64 62 31 65 39 30 65 36 37 32 63 61 65 35 65 64 |db1e90e672cae5ed| 000000d0 39 39 39 66 61 65 65 36 03 00 00 00 70 69 78 03 |999faee6....pix.| 000000e0 00 00 00 70 69 78 09 00 00 00 69 6d 61 67 65 2f |...pix....image/| 000000f0 6a 70 67 1c 1d 00 00 00 00 00 00 00 00 00 00 00 |jpg.............| 00000100 00 00 00 00 00 00 00 00 00 00 00 00 01 01 06 00 |................| 00000110 00 00 18 84 2e 7f 32 00 82 8e 02 20 00 00 00 5f |......2.... ..._| 00000120 6b 6a 47 75 70 4d 30 66 76 4e 38 37 65 39 6c 34 |kjGupM0fvN87e9l4| 00000130 42 68 70 4f 48 69 63 68 57 31 43 42 50 63 58 00 |BhpOHichW1CBPcX.| 00000140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| 0000014d The bucket with this id is not available anywhere: 9213182a-14ba-48ad-bde9-289a1c0c0de8.5411362.1 Why it is not cleaned up? How can I clean up the index pool with any unnecessary entries? Thank you Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@xxxxxxxxx --------------------------------------------------- -----Original Message----- From: Szabo, Istvan (Agoda) <Istvan.Szabo@xxxxxxxxx> Sent: Tuesday, June 8, 2021 3:56 PM To: ceph-users <ceph-users@xxxxxxx> Subject: Index pool hasn't been cleaned up and caused large omap, safe to delete the index file? Hi, In my multisite setup 1 big bucket has been deleted and seems like hasn't been cleaned up on one of the secondary site. Is it safe to delete the 11 shard objects from the index pool which holding the omaps of that bucket files? Also a quick question, is it a problem if we use like this? Create a bucket which means create in all dc Don't create any bucket sync the user upload different files in different dcs. When a bucket deletion happens would this usage behavior cause issue that different files are in the buckets? If yes, how to prevent this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@xxxxxxxxx<mailto:istvan.szabo@xxxxxxxxx> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |