Fixing a 12.2.5 reshard

[Sean Purdy <s.purdy@xxxxxxxxxxxxxxxx>]

Hi,


We were on 12.2.5 when a bucket with versioning and 100k objects got stuck when autoreshard kicked in.  We could download but not upload files.  But upgrading to 12.2.7 then running bucket check now shows twice as many objects, according to bucket limit check.  How do I fix this?


Sequence:

12.2.5 autoshard happened, "radosgw-admin reshard list" showed a reshard happening but no action.
12.2.7 upgrade went fine, didn't fix anything straightaway. "radosgw-admin reshard list" same.  Still no file uploads.  bucket limit check showed 100k files in the bucket as expected, and no shards.
Ran "radosgw-admin bucket check --fix"

Now "reshard list" shows no reshards in progress, but bucket limit check shows 200k files in two shards, 100k per shard.  It should be half this.


The output of "bucket check --fix" has 
existing_header: "num_objects": 203344 for "rgw.main"
calculated_header: "num_objects": 101621

Shouldn't it install the calculated_header?



Before:

$ sudo radosgw-admin reshard list
[
  {
    
    "tenant": "",
    "bucket_name": "static",
    "bucket_id": "a5501bce-1360-43e3-af08-8f3d1e102a79.3475308.1",
    "new_instance_id": "static:a5501bce-1360-43e3-af08-8f3d1e102a79.3620665.1",
    "old_num_shards": 1,
    "new_num_shards": 2
  }
]

$ sudo radosgw-admin bucket limit check
{
    "user_id": "static",
    "buckets": [
        {
            "bucket": "static",
            "tenant": "",
            "num_objects": 101621,
            "num_shards": 0,
            "objects_per_shard": 101621,
            "fill_status": "OK"
        }
    ]
}

Output from bucket check --fix

{
    "existing_header": {
        "usage": {
            "rgw.none": {
                "size": 0,
                "size_actual": 0,
                "size_utilized": 0,
                "size_kb": 0,
                "size_kb_actual": 0,
                "size_kb_utilized": 0,
                "num_objects": 101621
            },
            "rgw.main": {
                "size": 37615290807,
                "size_actual": 38017675264,
                "size_utilized": 0,
                "size_kb": 36733683,
                "size_kb_actual": 37126636,
                "size_kb_utilized": 0,
                "num_objects": 203344
            }
        }
    },
    "calculated_header": {
        "usage": {
            "rgw.none": {
                "size": 0,
                "size_actual": 0,
                "size_utilized": 0,
                "size_kb": 0,
                "size_kb_actual": 0,
                "size_kb_utilized": 0,
                "num_objects": 101621
            },
            "rgw.main": {
                "size": 18796589005,
                "size_actual": 18997686272,
                "size_utilized": 18796589005,
                "size_kb": 18356044,
                "size_kb_actual": 18552428,
                "size_kb_utilized": 18356044,
                "num_objects": 101621
            }
        }
    }
}

After:

{
    "user_id": "static",
    "buckets": [
        {
            "bucket": "static",
            "tenant": "",
            "num_objects": 203242,
            "num_shards": 2,
            "objects_per_shard": 101621,
            "fill_status": "OK"
        }
    ]
}


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com