Re: radosgw-admin bucket check --fix returns a lot of errors (unable to find head object data)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,
I was able to clean up the objects by hand. I leave my breadcrumbs here in
case someone finds it useful.

1. Get all rados objects via `radosgw-admin bucket radoslist --bucket
$BUCKET` and filter the ones the you need to remove
2. Remove the rados objects via `rados -p $RGWDATAPOOL rm $RADOSOBJECT
3. Check which files is in which omapkey is in which index ($BUCKETID is in
radosgw-admin bucket stats)
# for index in `rados -p $RGWINDEXPOOL ls | grep $BUCKETID`; do
>   for omap in `rados -p eu-central-1.rgw.buckets.index listomapkeys
${index} | grep $FILESYOUREMOVED`; do
>     echo "Index: ${index} - OMAP ${omap}"
>   done
> done

4. Remove the omapkeys after you checked the list is what actually should
be removed from the index
# for index in `rados -p $RGWINDEXPOOL ls | grep $BUCKETID`; do
>   for omap in `rados -p eu-central-1.rgw.buckets.index listomapkeys
${index} | grep $FILESYOUREMOVED`; do
>     rados -p eu-central-1.rgw.buckets.index rmomapkey ${index} ${omap}"
>   done
> done

5. run `radosgw-admin bucket check --fix --bucket $BUCKET` to have the
correct statistics

The message unable to find head object data pool for is still there, but
now I don't care. (I also have this for a healthy bucket, where I test
stuff like this prior, which gets recreated periodically)


Am Mi., 23. Nov. 2022 um 12:22 Uhr schrieb Boris Behrens <bb@xxxxxxxxx>:

> Hi,
> we have a customer that got some _multipart_ files in his bucket, but the
> bucket got no unfinished multipart objects.
> So I tried to remove them via
>
> $ radosgw-admin object rm --bucket BUCKET
> --object=_multipart_OBJECT.qjqyT8bXiWW5jdbxpVqHxXnLWOG3koUi.1
> ERROR: object remove returned: (2) No such file or directory
>
> Doing this with --debug_ms=1 I see this line:
>  osd_op_reply(108
> ff7a8b0c-07e6-463a-861b-78f0adeba8ad.2297644274.57___multipart_OBJECT.qjqyT8bXiWW5jdbxpVqHxXnLWOG3koUi.1
> [getxattrs,stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8
> ==== 345+0+0 (crc 0 0 0) 0x7f35ac01c040 con 0x55a8b470fba0
>
> I then tried to remove the leading _ from the object name, but this also
> did not work.
> Then I proceeded to remove the rados object and just do a
> $ bucket check --fix --bucket BUCKET
> ...
> 2022-11-23T11:21:14.214+0000 7fd50a8fd980  0 int
> RGWRados::check_disk_state(librados::v14_2_0::IoCtx, const RGWBucketInfo&,
> rgw_bucket_dir_entry&, rgw_bucket_dir_entry&, ceph::bufferlist&,
> optional_yield) WARNING: unable to find head object data pool for
> "BUCKET:SOME_AVAILABLE_OBJECT", not updating version pool/epoch
> ...
>
> And the calculated bucket size, is still the same and the index will not
> get updated.
>
> What to do now?
>
> --
> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> groüen Saal.
>


-- 
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux