Hmm, It's useful to rebuild the index by rewriting a object.
But at first, I need know the all keys of objects. If I want to know all keys, I need list_objects ...
Maybe I can make an union set of instances, then copy all of them into themselves.
Anyway, I want to find out more about why it happens and how to avoid it.
Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx>于2018年10月19日周五 上午2:25写道：
On Wed, Oct 17, 2018 at 1:14 AM Yang Yang <inksink95@xxxxxxxxx> wrote:
>
> Hi,
> A few weeks ago I found radosgw index has been inconsistent with reality. Some object I can not list, but I can get them by key. Please see the details below:
>
> BACKGROUND：
> Ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
> Index pool is on ssd.
> There is a very big bucket with more than 10 million object and 500TB data.
> Ceph health is OK.
> I use s3 api on radosgw.
>
> DESCRIBE:
> When use s3 list_object() to list, some uploaded object can not be listed and some uploaded object have an old lastModified time.
> But at the same time, we can get this object by an exact key. And if I put a new object into this bucket, it can be listed.
> It seems that some indexes during a period of time have been lost.
>
> I try to run "radosgw-admin bucket check --bucket <bucketname> --fix --check-objects" and I get nothing at all.
>
> SOME ELSE：
> I found that one bucket will have many indexes, and we can use "radosgw-admin metadata list bucket.instance | grep "{bucket name}" to show them. But I can not found a doc to describe this feature. And we can use "radosgw-admin bucket stats --bucket {bucket_name}" to get id as the active instance id.
> I use "rados listomapkeys" at active(or latest) index to get all object in a index, it is really lost. But when I use "rados listomapkeys" at another index which is not active as mentioned above, I found the lost object index.
>
> Resharding is within my consideration. Listomapkeys means do this action on all shards(more than 300).
> In my understanding, a big bucket has one latest index and many old indexes. Every index has many shards. So listomapkeys on a index means listomapkeys on many shards.
>
> QUESTION:
> Why my index lost?
> How to recover？
I don't really know what happened, haven't seen this exact issue
before. You can try copying objects into themselves. That should
recreate their bucket index entry.
> Why radosgw has many index instances, how do radosgw use them and how to change active index?
Could be related to an existing bug. You can unlink the bucket and
then link a specific bucket instance version (to the user), however,
I'm not sure I recommend going this path if it isn't necessary.
Regards,
Yehuda
>
>
> Thanks,
>
> Inksink
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
