Hi Abhishek, Our team has developed a radosgw-admin command to reclaim the leak space caused by re-upload the same part. I have reviewed the code yestoday and the procedure is scheduled as below: 1. list the objects with ns=multipart from the bucket index with a specified shard-id, since meta and multipart part index belongs to the same shard-id. 2. if meta index exists, we ignore the object. 3. if meta index does not exists, we think the multipart index is orphan parts and any objects associated are also leaked, then we put these leaked objects to gc. It's a method to reclaim the leaked space.But in my opinion, I do not think it is good enough. And we have to trigger the reclaim through shell or humen. We have to check which object is leaked, if bucket index shard-num is too large, it is a heavy and long work. How do you think the method? I hope to modify the code well and contribute to the ceph community. I have opened a tracker for the problem. http://tracker.ceph.com/issues/20284 Best Regards, Penglaixy -- Software Engineer, ChinaNetCenter Co., ShenZhen, Guangdong Province, China 2017-06-14 0:10 GMT+08:00 Abhishek Varshney <abhishek.varshney@xxxxxxxxxxxx>: > Hi Penglaiyxy, > > I was able to reproduce the scenario described by you in current > master branch. It turns out that, in case of re-uploading a part, rgw > assigns a different prefix to it. This was introduced by the commit > [1] as a fix for racy part uploads [2]. This leads to orphaned parts > being uploaded, which have an entry in bucket index, but are not > really part of any uploaded object. Such orphaned parts should have > ideally been cleaned-up on multipart complete or abort operation. This > looks like a slightly different issue, as the clean-up of such parts > should not wait for a bucket deletion operation to happen. > > Can you open a separate tracker issue for this, if one not present already. > > [1] https://github.com/ceph/ceph/pull/1781/commits/bd8e026f88b812cc70caf6232c247844df5d99bf > [2] http://tracker.ceph.com/issues/8269 > > Thanks > Abhishek > > On Tue, Jun 13, 2017 at 12:28 PM, 于相洋 <penglaiyxy@xxxxxxxxx> wrote: >> Hi Abhishek, >> >> Your PR is practical for anyone who can not delete an non-empty bucket. >> >> But I think in one scenario that you still can not delete bucket >> because of the orphan object leaked by multipart upload. >> >> Upload a multipart object: >> >> 1. Upload part 1 with prefix 2~-DUZmxVbiv9dBycBdci2iMhiKEEUv-5 >> >> 2. The connection between rgw client and server is cutoff, I don't >> know if I have upload part one completely ,so I re-upload part 1 then >> the part 1 is attached with a new prefix like >> eOntuNHW8UdvnpbLl9UAdYGuWrL9HPH(which is tackled by rgw server ) >> >> 3. The connection between rgw client and server is cutoff again, then >> I re-upload part 1 with new prefix d9H1FWnoOwmr3IAejtYQJ2hyIjsUA7U >> >> 4. Upload part 2 and left parts >> >> 5. At the end, complete the upload. >> >> The object with names below is leaked and I can not see them through >> s3 client. (My upload size is 15MB and stripe size is 4M) >> >> center-master.4439.1__multipart_bigfile.2~-DUZmxVbiv9dBycBdci2iMhiKEEUv-5.1 >> center-master.4439.1__shadow_bigfile.2~-DUZmxVbiv9dBycBdci2iMhiKEEUv-5.1_1 >> center-master.4439.1__shadow_bigfile.2~-DUZmxVbiv9dBycBdci2iMhiKEEUv-5.1_2 >> center-master.4439.1__shadow_bigfile.2~-DUZmxVbiv9dBycBdci2iMhiKEEUv-5.1_3 >> >> center-master.4439.1__multipart_bigfile.eOntuNHW8UdvnpbLl9UAdYGuWrL9HPH.1 >> center-master.4439.1__shadow_bigfile.eOntuNHW8UdvnpbLl9UAdYGuWrL9HPH.1_1 >> center-master.4439.1__shadow_bigfile.eOntuNHW8UdvnpbLl9UAdYGuWrL9HPH.1_2 >> center-master.4439.1__shadow_bigfile.eOntuNHW8UdvnpbLl9UAdYGuWrL9HPH.1_3 >> >> The above upload process is really occurring in our production >> application and we find a lot space leaked. >> >> How can I reclaim the space leaked? >> >> Any Idea is appreciated. >> >> >> ________________________________ >> penglaiyxy >> >> From: Abhishek Varshney >> Date: 2017-06-13 00:17 >> To: Ceph Development >> Subject: rgw: leak with incomplete multiparts (was: Request for review) >> Reviving an old thread by a colleague on rgw leaking rados objects, >> the PR submitted earlier [1] had failed teuthology rgw run, due to >> radosgw-admin failing to remove a user with --purge-data flag. I tried >> to root cause the issue, and it turned out that incomplete multiparts >> need to be aborted when doing bucket rm with --purge-data. Here is the >> new PR (https://github.com/ceph/ceph/pull/15630) which handles >> incomplete multiparts with behaviour as given below: >> >> * radosgw-admin user/bucket rm with incomplete multiparts would return >> bucket not empty error. >> * radosgw-admin user/bucket rm --purge-data with incomplete multiparts >> would abort the pending multiparts and then delete the bucket. >> * S3 delete bucket API with incomplete multiparts would return bucket >> not empty error. The expectation here is on the user to either >> complete or cancel all pending multipart uploads before deleting the >> bucket. >> >> Requesting review on this PR. >> >> PS : The check for an empty bucket index here [2] in the previous PR >> [1] has been removed, as we found instances of inconsistent bucket >> index with stale entries, without corresponding objects present in >> data pool. This would have prevented the deletion of an empty bucket >> with such inconsistent indexes. I am not sure on how to reproduce such >> a scenario though. >> >> [1] https://github.com/ceph/ceph/pull/10920 >> [2] https://github.com/ceph/ceph/pull/10920/files#diff-c30965955342b98393b73be699f4e355R7349 >> >> Thanks >> Abhishek >> >> On Mon, Sep 12, 2016 at 4:23 AM, Praveen Kumar G T (Cloud Platform) >> <praveen.gt@xxxxxxxxxxxx> wrote: >>> Definitions >>> >>> Orphaned Objects: Orphaned objects are created when a Multipart upload has uploaded some or all of the parts but without executing multipart cancel or multipart complete. An incomplete Multipart upload is neither created nor destroyed, it is in orphan state >>> Leaked Objects : Objects that are not deleted on the ceph cluster but assumed to be deleted by s3 client. These objects cannot be accessed by s3 clients but still occupy space in the ceph cluster >>> >>> Problem >>> >>> A s3 bucket cannot be deleted when there are objects in it. The bucket deletion command will fail with error BucketNotEmpty. The objects in the buckets can be listed using any of the s3 clients. In case if we have orphaned objects present in the bucket, they will not be listed via the normal listing operations of the s3 clients. If the bucket is deleted when there are orphaned objects in the bucket, they will end up being leaked. This ends of using space in the ceph cluster even though the objects are not accessed by any of the s3 clients. This space is not accounted under the radosgw user account as well >>> >>> Tracker link >>> >>> http://tracker.ceph.com/issues/17164 >>> >>> Fix >>> >>> The fix will avoid deletion of buckets even if there are Orphaned objects in the bucket. So now bucket deletion command will return BucketNotEmpty when there are orphaned objects as well. >>> >>> Pull request >>> >>> https://github.com/ceph/ceph/pull/10920 >>> >>> Can somebody Please review the fix. We have already verified the fix locally. >>> >>> Regards, >>> Praveen. >>> >>> >>> -- >> >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
