Hi Folks, I noted in another context, I have work-in-progress code to address this issue. I should be able to update and push a bugfix for ceph/main fairly soon. Matt On Thu, Oct 13, 2022 at 11:45 AM Haas, Josh <jhaas@xxxxxxxxxx> wrote: > Hi Liang, > > My guess would be this bug: > > https://tracker.ceph.com/issues/44660 > https://www.spinics.net/lists/ceph-users/msg30151.html > > > It's actually existed for at least 6 years: > https://tracker.ceph.com/issues/16767 > > > Which occurs any time you reupload the same *part* in a single Multipart > Upload multiple times. For example, if my Multipart upload consists of 3 > parts, if I upload part #2 twice, then the first upload of part #2 becomes > orphaned. > > > If this was indeed the cause, you should have multiple "_multipart_" rados > objects for the same part in "rados ls". For example, here's all the rados > objects associated with a bugged bucket before I deleted it: > > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.4vkWzU4C5XLd2R6unFgbQ6aZM26vPuq8.1 > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.2~4zogSe4Ep0xvSC8j6aX71x_96cOgvQN.1 > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__shadow_file.txt.4vkWzU4C5XLd2R6unFgbQ6aZM26vPuq8.1_1 > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__shadow_file.txt.2~4zogSe4Ep0xvSC8j6aX71x_96cOgvQN.1_1 > > > If we look at just these two: > > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.4vkWzU4C5XLd2R6unFgbQ6aZM26vPuq8.1 > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.2~4zogSe4Ep0xvSC8j6aX71x_96cOgvQN.1 > > > They are in the format: > > $BUCKETID__multipart_$S3KEY.$PARTUID.$PARTNUM > > Because everything matches ($BUCKETID, $S3KEY, $PARTNUM) except for > $PARTUID, this S3 object has been affected by the bug. If you find > instances of rados keys that match on everything except $PARTUID, then this > bug is probably the cause. > > > Josh > > ________________________________ > From: 郑亮 <zhengliang0901@xxxxxxxxx> > Sent: Wednesday, October 12, 2022 1:34:31 AM > To: ceph-users@xxxxxxx > Subject: why rgw generates large quantities orphan objects? > > Hi all, > Description of problem: [RGW] Buckets/objects deletion is causing large > quantities orphan raods objects > > The cluster was running a cosbench workload, then remove the partial data > by deleting objects from the cosbench client, then we have deleted all the > buckets with the help of `s3cmd rb --recursive --force` command that > removed all the buckets, but that did not help in the space reclaimation. > > ``` > [root@node01 /]# rgw-orphan-list > > Available pools: > > device_health_metrics > > .rgw.root > > os-test.rgw.buckets.non-ec > > os-test.rgw.log > > os-test.rgw.control > > os-test.rgw.buckets.index > > os-test.rgw.meta > > os-test.rgw.buckets.data > > deeproute-replica-hdd-pool > > deeproute-replica-ssd-pool > > cephfs-metadata > > cephfs-replicated-pool > > .nfs > > Which pool do you want to search for orphans (for multiple, use > space-separated list)? os-test.rgw.buckets.data > > Pool is "os-test.rgw.buckets.data". > > Note: output files produced will be tagged with the current timestamp -- > 20221008062356. > running 'rados ls' at Sat Oct 8 06:24:05 UTC 2022 > > running 'rados ls' on pool os-test.rgw.buckets.data. > > > > running 'radosgw-admin bucket radoslist' at Sat Oct 8 06:43:21 UTC 2022 > computing delta at Sat Oct 8 06:47:17 UTC 2022 > > 39662551 potential orphans found out of a possible 39844453 (99%). > The results can be found in './orphan-list-20221008062356.out'. > > Intermediate files are './rados-20221008062356.intermediate' and > './radosgw-admin-20221008062356.intermediate'. > *** > > *** WARNING: This is EXPERIMENTAL code and the results should be used > *** only with CAUTION! > *** > Done at Sat Oct 8 06:48:07 UTC 2022. > > [root@node01 /]# radosgw-admin gc list > [] > > [root@node01 /]# cat orphan-list-20221008062356.out | wc -l > 39662551 > > [root@node01 /]# rados df > POOL_NAME USED OBJECTS CLONES COPIES > MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS > WR USED COMPR UNDER COMPR > .nfs 4.3 MiB 4 0 12 > 0 0 0 77398 76 MiB 146 79 > KiB 0 B 0 B > .rgw.root 180 KiB 16 0 48 > 0 0 0 28749 28 MiB 0 > 0 B 0 B 0 B > cephfs-metadata 932 MiB 14772 0 44316 > 0 0 0 1569690 3.8 GiB 1258651 3.4 > GiB 0 B 0 B > cephfs-replicated-pool 738 GiB 300962 0 902886 > 0 0 0 794612 470 GiB 770689 245 > GiB 0 B 0 B > deeproute-replica-hdd-pool 1016 GiB 104276 0 312828 > 0 0 0 18176216 298 GiB 441783780 6.7 > TiB 0 B 0 B > deeproute-replica-ssd-pool 30 GiB 3691 0 11073 > 0 0 0 2466079 2.1 GiB 8416232 221 > GiB 0 B 0 B > device_health_metrics 50 MiB 108 0 324 > 0 0 0 1836 1.8 MiB 1944 18 > MiB 0 B 0 B > os-test.rgw.buckets.data 5.6 TiB 39844453 0 239066718 > 0 0 0 552896177 3.0 TiB 999441015 60 > TiB 0 B 0 B > os-test.rgw.buckets.index 1.8 GiB 33 0 99 > 0 0 0 153600295 154 GiB 110916573 62 > GiB 0 B 0 B > os-test.rgw.buckets.non-ec 2.1 MiB 45 0 135 > 0 0 0 574240 349 MiB 153725 139 > MiB 0 B 0 B > os-test.rgw.control 0 B 8 0 24 > 0 0 0 0 0 B 0 > 0 B 0 B 0 B > os-test.rgw.log 3.7 MiB 346 0 1038 > 0 0 0 83877803 80 GiB 6306730 7.6 > GiB 0 B 0 B > os-test.rgw.meta 220 KiB 23 0 69 > 0 0 0 640854 506 MiB 108229 53 > MiB 0 B 0 B > > total_objects 40268737 > total_used 7.8 TiB > total_avail 1.1 PiB > total_space 1.1 PiB > ``` > ceph verison: > ``` > [root@node01 /]# ceph versions > { > "mon": { > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > pacific (stable)": 3 > }, > "mgr": { > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > pacific (stable)": 2 > }, > "osd": { > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > pacific (stable)": 108 > }, > "mds": { > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > pacific (stable)": 2 > }, > "rgw": { > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > pacific (stable)": 9 > }, > "overall": { > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > pacific (stable)": 124 > } > } > ``` > > Thanks, > Best regards > Liang Zheng > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |