Also recently updated to 94.2. I am also seeing a large difference
between my 'ceph df' and 'size_kb_actual' in the bucket stats. I would
assume the difference is objects awaiting gc, but 'gc list' prints
very little.
ceph df:
NAME ID USED %USED MAX AVAIL OBJECTS
...
.rgw.buckets 12 37370G 21.18 42122G 25934075
radosgw-admin bucket stats | grep size_kb_actual | sed 's# \|,##g' |
cut -d':' -f2 | awk '{s+=$1} END {print s}' | awk '{$1=$1/(1024^2);
print $1,"GB";}'
5700.74 GB <---- would expect ~37370G ?
radosgw-admin gc list --include-all has 10,000 objects, but i think
that is from an earlier today rebalance, the 37 Gig vs 5.7 GB
difference has been present for quite a while.
Any suggestions?
-Ben
On Mon, Jun 29, 2015 at 2:32 AM, Gleb Borisov <borisov.gleb@xxxxxxxxx> wrote:
> Hi
>
> We've just upgraded our storage from 0.94 to 0.94.2 and realized that we
> have a lot of garbage and corrupted objects in our buckets.
>
> First of all, we found several corrupted objects (missing data in the middle
> of object) uploaded via S3 multipart upload with enabled retry policy. It
> seems that we faced with [1]. Is there a proper way to find such objects?
> How we can deal with them?
> Of course we can check all gzipped objects (download and try to uncompress
> them), but we also have a lot of plain text data objects which we can't
> check in this way.
>
> Also we were also impacted by [2]. Now garbage collection is running
> normally, but there are 70TiB of garbage in our storage. Do we have any way
> to find and remove all such objects? (we check bucket stats and it shows
> 20TiB of data and `ceph df` reports 90TiB used).
>
> Thanks.
>
> [1] http://tracker.ceph.com/issues/11604
> [2] http://tracker.ceph.com/issues/10295
>
> --
> Best regards,
> Gleb M Borisov
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
