We'll need to have more info about the current state. Was just the
omap header destroyed, or does it still exist? What does the header
contain now? Are you able to actually access objects in that bucket,
but just fail to list them?
On Mon, Apr 8, 2013 at 8:34 AM, Erdem Agaoglu <erdem.agaoglu@xxxxxxxxx> wrote:
> Hi again,
>
> I managed to change the file with some other bucket's index. --check-objects
> --fix worked but my hopes have failed as it didn't actually read through the
> files or fixed anything. Any suggestions?
>
>
> On Thu, Apr 4, 2013 at 5:56 PM, Erdem Agaoglu <erdem.agaoglu@xxxxxxxxx>
> wrote:
>>
>> Hi all,
>>
>> After a major failure, and getting our cluster health back OK (with some
>> help from inktank folks, thanks), we found out that we have managed to
>> corrupt one of our bucket indices. As far as i can track it, we are missing
>> the omapheader on that specific index, so we're unable to use radosgw-admin
>> tools to fix it.
>>
>> While a healthy (smaller) bucket answers
>> # radosgw-admin bucket check -b imgdoviz
>> { "existing_header": { "usage": { "rgw.main": { "size_kb": 4140,
>> "size_kb_actual": 4484,
>> "num_objects": 157}}},
>> "calculated_header": { "usage": { "rgw.main": { "size_kb": 4140,
>> "size_kb_actual": 4484,
>> "num_objects": 157}}}}
>>
>> The faulty one fails with
>> # radosgw-admin bucket check -b imgiz
>> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) err=(22)
>> Invalid argument
>> failed to check index err=(22) Invalid argument
>>
>> When i push further
>> # radosgw-admin bucket check -b imgiz --check-objects --fix
>> failed to list objects in bucket=imgiz(@.rgw.buckets[4470.1]) err=(22)
>> Invalid argument
>> Checking objects, decreasing bucket 2-phase commit timeout.
>> ** Note that timeout will reset only when operation completes successfully
>> **
>> ERROR: failed operation r=-22
>> ERROR: failed operation r=-22
>> ..
>> last line keeps repeating without any progress.
>>
>> I checked the file omapheaders and while the healty bucket has:
>> # rados -p .rgw.buckets getomapheader .dir.6912.3
>> header (49 bytes) :
>> 0000 : 03 02 2b 00 00 00 01 00 00 00 01 02 02 18 00 00 : ..+.............
>> 0010 : 00 a8 af 40 00 00 00 00 00 00 10 46 00 00 00 00 : ...@.......F....
>> 0020 : 00 9d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 : ................
>> 0030 : 00 : .
>>
>> the faulty one is missing it
>> # rados -p .rgw.buckets getomapheader .dir.4470.1
>> header (0 bytes) :
>>
>>
>> I'm currently in the process of understanding how to create a readable
>> header. My hopes are even while its wrong, radosgw-admin will be able to
>> read through objects and fix the necessary parts. But i'm not sure how to
>> set the new-header. It seems there is a setomapheader counterpart for
>> getomapheader but it only accepts values from commandline so i don't know
>> how to push a rgw-readable binary with it.
>>
>> Is this somewhat possible?
>>
>> Thanks in advance.
>>
>> --
>> erdem agaoglu
>
>
>
>
> --
> erdem agaoglu
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
