We're having the same issues. I have a 1200TB pool at 90% utilization however disk utilization is only 40%
|
| ||
If you are not the intended recipient of this transmission you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. |
From: "Brian Felton" <bjfelton@xxxxxxxxx>
To: "ceph-users" <ceph-users@xxxxxxxx>
Sent: Wednesday, July 27, 2016 9:24:30 AM
Subject: [ceph-users] Cleaning Up Failed Multipart Uploads
To: "ceph-users" <ceph-users@xxxxxxxx>
Sent: Wednesday, July 27, 2016 9:24:30 AM
Subject: [ceph-users] Cleaning Up Failed Multipart Uploads
Greetings,
Background: If an object storage client re-uploads parts to a multipart object, RadosGW does not clean up all of the parts properly when the multipart upload is aborted or completed. You can read all of the gory details (including reproduction steps) in this bug report: http://tracker.ceph.com/issues/16767.My setup: Hammer 0.94.6 cluster only used for S3-compatible object storage. RGW stripe size is 4MiB.
My problem: I have buckets that are reporting TB more utilization (and, in one case, 200k more objects) than they should report. I am trying to remove the detritus from the multipart uploads, but removing the leftover parts directly from the .rgw.buckets pool is having no effect on bucket utilization (i.e. neither the object count nor the space used are declining).
I've looked at the output of 'gc list --include-all', and the removed parts are never showing up for garbage collection. Garbage collection is otherwise functioning normally and will successfully remove data for any object properly removed via the S3 interface.
I've also gone so far as to write a script to list the contents of bucket shards in the .rgw.buckets.index pool, check for the existence of the entry in .rgw.buckets, and remove entries that cannot be found, but that is also failing to decrement the size/object count counters.
What am I missing here? Where, aside from .rgw.buckets and .rgw.buckets.index is RGW looking to determine object count and space used for a bucket?
Many thanks to any and all who can assist.
Brian Felton
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com