Sorry to send another long followup, but actually... I'm not sure how to
change the placement_rule for a bucket - or at least what I tried does
not seem to work. Using a different (more disposable) bucket, my attempt
went like this::
first created a new placement rule "old-placement" in both the zonegroup
and zone commit new period - this looks ok.
{
"key": "old-placement",
"val": {
"index_pool": ".rgw.buckets",
"data_pool": ".rgw.buckets",
"data_extra_pool": "",
"index_type": 0,
"compression": ""
}
The current metadata for the test bucket looks like this:
{
"key": "bucket.instance:boto-demo-100:default.2170793.10",
"ver": {
"tag": "_BFMtrwjsyFbhU-65IielO3q",
"ver": 652
},
"mtime": "2018-02-14 13:46:37.993218Z",
"data": {
"bucket_info": {
"bucket": {
"name": "boto-demo-100",
"marker": "default.2170793.10",
"bucket_id": "default.2170793.10",
"tenant": "",
"explicit_placement": {
"data_pool": ".rgw.buckets",
"data_extra_pool": "",
"index_pool": ".rgw.buckets"
}
},
"creation_time": "0.000000",
"owner": "xxxxxx",
"flags": 2,
"zonegroup": "default",
"placement_rule": "",
"has_instance_obj": "true",
"quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1024,
"max_size_kb": 0,
"max_objects": -1
},
"num_shards": 32,
"bi_shard_hash_type": 0,
"requester_pays": "false",
"has_website": "false",
"swift_versioning": "false",
"swift_ver_location": "",
"index_type": 0,
"mdsearch_config": [],
"reshard_status": 0,
"new_bucket_instance_id": ""
},
"attrs": [
{
"key": "user.rgw.acl",
"val": "AgKFAAAAAwIXAAAABgAAAGJseW5jaAkAAABCZW4gTHluY2gDA2IAAAABAQAAAAYAAABibHluY2gPAAAAAQAAAAYAAABibHluY2gEAzcAAAACAgQAAAAAAAAABgAAAGJseW5jaAAAAAAAAAAAAgIEAAAADwAAAAkAAABCZW4gTHluY2gAAAAAAAAAAA=="
},
{
"key": "user.rgw.idtag",
"val": ""
}
]
}
}
I edit "placement_rule": to change "" -> "old-placement" and write it
back using:
radosgw-admin metadata put bucket.instance:boto-demo-100:default.2170793.10 < boto-demo-100.json
Now when I run "radosgw-admin bucket list --bucket=boto-demo-100" I am
getting an empty list, though I'm pretty sure the bucket contains some
objects.
When I read the bucket instance metadata back again, it still reads
"placement_rule": "" so I wonder if the bucket_info change is really
taking effect.
A quick debug session seems to show it still querying the wrong pool
(100) for the index, so it seems that my attempt to update the
bucket_info is either failing or incorrect!
Graham
On 02/19/2018 04:00 PM, Graham Allan wrote:
Thanks Robin,
Of the two issues, this seems to me like it must be #22928.
Since the majority of index entries for this bucket are in the
.rgw.buckets pool, but newer entries have been created in
.rgw.buckets.index, it's clearly failing to use the explicit placement
pool - and with the index data split across two pools I don't see how
resharding could correct this.
I can get the object names from the "new" (incorrect) indexes with
something like:
for i in `rados -p .rgw.buckets.index ls - | grep
"default.2049236.2"`; do rados -p .rgw.buckets.index listomapkeys
$i|grep "^[a-zA-Z0-9]"; done
Fortunately there are only ~20, in this bucket (my grep is just a stupid
way to skip what I assume are the multipart parts, which have a
non-ascii first char).
... and these files are downloadable, at least using s3cmd (minio client
fails, it seems to try and check the index first).
Once I have these newer files downloaded, then to restore access to the
older index I like the suggestion in the issue tracker to create a new
placement target in the zone, and modify the bucket's placement rule to
match. It seems like it might be safer than copying objects from one
index pool to the other (though the latter certainly sounds faster and
easier!)
From a quick check, I suspect I probably have 40 or so other buckets
with this problem... will need to check them more closely.
Actually it looks like a lot of the affected buckets were created around
10/2016 - I suspect the placement policies were incorrect for a short
time due to confusion over the hammer->jewel upgrade (the
realm/period/zonegroup/zone conversion didn't really go smoothly!)
On 02/16/2018 11:39 PM, Robin H. Johnson wrote:
On Fri, Feb 16, 2018 at 07:06:21PM -0600, Graham Allan wrote:
[snip great debugging]
This seems similar to two open issues, could be either of them depending
on how old that bucket is.
http://tracker.ceph.com/issues/22756
http://tracker.ceph.com/issues/22928
- I have a mitigation posted to 22756.
- There's a PR posted for 22928, but it'll probably only be in v12.2.4.
--
Graham Allan
Minnesota Supercomputing Institute - gta@xxxxxxx
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
