On Wed, Oct 17, 2018 at 1:14 AM Yang Yang <inksink95@xxxxxxxxx> wrote:
>
> Hi,
> A few weeks ago I found radosgw index has been inconsistent with reality. Some object I can not list, but I can get them by key. Please see the details below:
>
> BACKGROUND：
> Ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
> Index pool is on ssd.
> There is a very big bucket with more than 10 million object and 500TB data.
> Ceph health is OK.
> I use s3 api on radosgw.
>
> DESCRIBE:
> When use s3 list_object() to list, some uploaded object can not be listed and some uploaded object have an old lastModified time.
> But at the same time, we can get this object by an exact key. And if I put a new object into this bucket, it can be listed.
> It seems that some indexes during a period of time have been lost.
>
> I try to run "radosgw-admin bucket check --bucket <bucketname> --fix --check-objects" and I get nothing at all.
>
> SOME ELSE：
> I found that one bucket will have many indexes, and we can use "radosgw-admin metadata list bucket.instance | grep "{bucket name}" to show them. But I can not found a doc to describe this feature. And we can use "radosgw-admin bucket stats --bucket {bucket_name}" to get id as the active instance id.
> I use "rados listomapkeys" at active(or latest) index to get all object in a index, it is really lost. But when I use "rados listomapkeys" at another index which is not active as mentioned above, I found the lost object index.
>
> Resharding is within my consideration. Listomapkeys means do this action on all shards(more than 300).
> In my understanding, a big bucket has one latest index and many old indexes. Every index has many shards. So listomapkeys on a index means listomapkeys on many shards.
>
> QUESTION:
> Why my index lost?
> How to recover？
I don't really know what happened, haven't seen this exact issue
before. You can try copying objects into themselves. That should
recreate their bucket index entry.
> Why radosgw has many index instances, how do radosgw use them and how to change active index?
Could be related to an existing bug. You can unlink the bucket and
then link a specific bucket instance version (to the user), however,
I'm not sure I recommend going this path if it isn't necessary.
Regards,
Yehuda
>
>
> Thanks,
>
> Inksink
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
