Hi Ulrich, I can file a tracker on your behalf if you want. Do let me know. On Fri, Oct 14, 2022 at 1:35 PM Ulrich Klein <Ulrich.Klein@xxxxxxxxxxxxxx> wrote: > Hi, > > I’m wondering if this problem will ever get fixed? > > This multipart-orphan-problem has made it now multiple times to the list, > the tickets are up to 6 years old … and nothing changes. > It screws up per-user space accounting and uses up space for nothing. > > I’d open another ticket with easy steps to reproduce, but can’t get an > account on tracker.ceph.com (“pending administrator approval”) > > Ciao, Uli > > > On 13. 10 2022, at 17:44, Haas, Josh <jhaas@xxxxxxxxxx> wrote: > > > > Hi Liang, > > > > My guess would be this bug: > > > > https://tracker.ceph.com/issues/44660 > > https://www.spinics.net/lists/ceph-users/msg30151.html > > > > > > It's actually existed for at least 6 years: > > https://tracker.ceph.com/issues/16767 > > > > > > Which occurs any time you reupload the same *part* in a single Multipart > Upload multiple times. For example, if my Multipart upload consists of 3 > parts, if I upload part #2 twice, then the first upload of part #2 becomes > orphaned. > > > > > > If this was indeed the cause, you should have multiple "_multipart_" > rados objects for the same part in "rados ls". For example, here's all the > rados objects associated with a bugged bucket before I deleted it: > > > > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.4vkWzU4C5XLd2R6unFgbQ6aZM26vPuq8.1 > > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.2~4zogSe4Ep0xvSC8j6aX71x_96cOgvQN.1 > > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__shadow_file.txt.4vkWzU4C5XLd2R6unFgbQ6aZM26vPuq8.1_1 > > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__shadow_file.txt.2~4zogSe4Ep0xvSC8j6aX71x_96cOgvQN.1_1 > > > > > > If we look at just these two: > > > > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.4vkWzU4C5XLd2R6unFgbQ6aZM26vPuq8.1 > > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.2~4zogSe4Ep0xvSC8j6aX71x_96cOgvQN.1 > > > > > > They are in the format: > > > > $BUCKETID__multipart_$S3KEY.$PARTUID.$PARTNUM > > > > Because everything matches ($BUCKETID, $S3KEY, $PARTNUM) except for > $PARTUID, this S3 object has been affected by the bug. If you find > instances of rados keys that match on everything except $PARTUID, then this > bug is probably the cause. > > > > > > Josh > > > > ________________________________ > > From: 郑亮 <zhengliang0901@xxxxxxxxx> > > Sent: Wednesday, October 12, 2022 1:34:31 AM > > To: ceph-users@xxxxxxx > > Subject: why rgw generates large quantities orphan objects? > > > > Hi all, > > Description of problem: [RGW] Buckets/objects deletion is causing large > > quantities orphan raods objects > > > > The cluster was running a cosbench workload, then remove the partial data > > by deleting objects from the cosbench client, then we have deleted all > the > > buckets with the help of `s3cmd rb --recursive --force` command that > > removed all the buckets, but that did not help in the space reclaimation. > > > > ``` > > [root@node01 /]# rgw-orphan-list > > > > Available pools: > > > > device_health_metrics > > > > .rgw.root > > > > os-test.rgw.buckets.non-ec > > > > os-test.rgw.log > > > > os-test.rgw.control > > > > os-test.rgw.buckets.index > > > > os-test.rgw.meta > > > > os-test.rgw.buckets.data > > > > deeproute-replica-hdd-pool > > > > deeproute-replica-ssd-pool > > > > cephfs-metadata > > > > cephfs-replicated-pool > > > > .nfs > > > > Which pool do you want to search for orphans (for multiple, use > > space-separated list)? os-test.rgw.buckets.data > > > > Pool is "os-test.rgw.buckets.data". > > > > Note: output files produced will be tagged with the current timestamp -- > > 20221008062356. > > running 'rados ls' at Sat Oct 8 06:24:05 UTC 2022 > > > > running 'rados ls' on pool os-test.rgw.buckets.data. > > > > > > > > running 'radosgw-admin bucket radoslist' at Sat Oct 8 06:43:21 UTC 2022 > > computing delta at Sat Oct 8 06:47:17 UTC 2022 > > > > 39662551 potential orphans found out of a possible 39844453 (99%). > > The results can be found in './orphan-list-20221008062356.out'. > > > > Intermediate files are './rados-20221008062356.intermediate' and > > './radosgw-admin-20221008062356.intermediate'. > > *** > > > > *** WARNING: This is EXPERIMENTAL code and the results should be used > > *** only with CAUTION! > > *** > > Done at Sat Oct 8 06:48:07 UTC 2022. > > > > [root@node01 /]# radosgw-admin gc list > > [] > > > > [root@node01 /]# cat orphan-list-20221008062356.out | wc -l > > 39662551 > > > > [root@node01 /]# rados df > > POOL_NAME USED OBJECTS CLONES COPIES > > MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS > > WR USED COMPR UNDER COMPR > > .nfs 4.3 MiB 4 0 12 > > 0 0 0 77398 76 MiB 146 79 > > KiB 0 B 0 B > > .rgw.root 180 KiB 16 0 48 > > 0 0 0 28749 28 MiB 0 > > 0 B 0 B 0 B > > cephfs-metadata 932 MiB 14772 0 44316 > > 0 0 0 1569690 3.8 GiB 1258651 3.4 > > GiB 0 B 0 B > > cephfs-replicated-pool 738 GiB 300962 0 902886 > > 0 0 0 794612 470 GiB 770689 245 > > GiB 0 B 0 B > > deeproute-replica-hdd-pool 1016 GiB 104276 0 312828 > > 0 0 0 18176216 298 GiB 441783780 6.7 > > TiB 0 B 0 B > > deeproute-replica-ssd-pool 30 GiB 3691 0 11073 > > 0 0 0 2466079 2.1 GiB 8416232 221 > > GiB 0 B 0 B > > device_health_metrics 50 MiB 108 0 324 > > 0 0 0 1836 1.8 MiB 1944 18 > > MiB 0 B 0 B > > os-test.rgw.buckets.data 5.6 TiB 39844453 0 239066718 > > 0 0 0 552896177 3.0 TiB 999441015 60 > > TiB 0 B 0 B > > os-test.rgw.buckets.index 1.8 GiB 33 0 99 > > 0 0 0 153600295 154 GiB 110916573 62 > > GiB 0 B 0 B > > os-test.rgw.buckets.non-ec 2.1 MiB 45 0 135 > > 0 0 0 574240 349 MiB 153725 139 > > MiB 0 B 0 B > > os-test.rgw.control 0 B 8 0 24 > > 0 0 0 0 0 B 0 > > 0 B 0 B 0 B > > os-test.rgw.log 3.7 MiB 346 0 1038 > > 0 0 0 83877803 80 GiB 6306730 7.6 > > GiB 0 B 0 B > > os-test.rgw.meta 220 KiB 23 0 69 > > 0 0 0 640854 506 MiB 108229 53 > > MiB 0 B 0 B > > > > total_objects 40268737 > > total_used 7.8 TiB > > total_avail 1.1 PiB > > total_space 1.1 PiB > > ``` > > ceph verison: > > ``` > > [root@node01 /]# ceph versions > > { > > "mon": { > > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > > pacific (stable)": 3 > > }, > > "mgr": { > > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > > pacific (stable)": 2 > > }, > > "osd": { > > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > > pacific (stable)": 108 > > }, > > "mds": { > > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > > pacific (stable)": 2 > > }, > > "rgw": { > > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > > pacific (stable)": 9 > > }, > > "overall": { > > "ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > > pacific (stable)": 124 > > } > > } > > ``` > > > > Thanks, > > Best regards > > Liang Zheng > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- *Dhairya Parmar* He/Him/His Associate Software Engineer, CephFS Red Hat Inc. <https://www.redhat.com/> dparmar@xxxxxxxxxx <https://www.redhat.com/> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |