Hello Orit,
Please find attached the output from the radosgw commands and the
relevant section from ceph.conf (radosgw)
bbp-gva-master is running 10.2.5
bbp-gva-secondary is running 10.2.7
Kind regards,
Ben Morrice
______________________________________________________________________
Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland
On 21/04/17 07:55, Orit Wasserman wrote:
Hi Ben,
On Thu, Apr 20, 2017 at 6:08 PM, Ben Morrice <ben.morrice@xxxxxxx> wrote:
Hi all,
I have tried upgrading one of our RGW servers from 10.2.5 to 10.2.7 (RHEL7)
and authentication is in a very bad state. This installation is part of a
multigw configuration, and I have just updated one host in the secondary
zone (all other hosts/zones are running 10.2.5).
On the 10.2.7 server I cannot authenticate as a user (normally backed by
OpenStack Keystone), but even worse I can also not authenticate with an
admin user.
Please see [1] for the results of performing a list bucket operation with
python boto (script works against rgw 10.2.5)
Also, if I try to authenticate from the 'master' rgw zone with a
"radosgw-admin sync status --rgw-zone=bbp-gva-master" I get:
"ERROR: failed to fetch datalog info"
"failed to retrieve sync info: (13) Permission denied"
The above errors correlates to the errors in the log on the server running
10.2.7 (debug level 20) at [2]
I'm not sure what I have done wrong or can try next?
By the way, downgrading the packages from 10.2.7 to 10.2.5 returns
authentication functionality
Can you provide the following info:
radosgw-admin period get
radsogw-admin zonegroup get
radsogw-admin zone get
Can you provide your ceph.conf?
Thanks,
Orit
[1]
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0"
encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><RequestId>tx000000000000000000004-0058f8c86a-3fa2959-bbp-gva-secondary</RequestId><HostId>3fa2959-bbp-gva-secondary-bbp-gva</HostId></Error>
[2]
/bbpsrvc15.cscs.ch/admin/log
2017-04-20 16:43:04.916253 7ff87c6c0700 15 calculated
digest=Ofg/f/NI0L4eEG1MsGk4PsVscTM=
2017-04-20 16:43:04.916255 7ff87c6c0700 15
auth_sign=qZ3qsy7AuNCOoPMhr8yNoy5qMKU=
2017-04-20 16:43:04.916255 7ff87c6c0700 15 compare=34
2017-04-20 16:43:04.916266 7ff87c6c0700 10 failed to authorize request
2017-04-20 16:43:04.916268 7ff87c6c0700 20 handler->ERRORHANDLER:
err_no=-2027 new_err_no=-2027
2017-04-20 16:43:04.916329 7ff87c6c0700 2 req 354:0.052585:s3:GET
/admin/log:get_obj:op status=0
2017-04-20 16:43:04.916339 7ff87c6c0700 2 req 354:0.052595:s3:GET
/admin/log:get_obj:http status=403
2017-04-20 16:43:04.916343 7ff87c6c0700 1 ====== req done
req=0x7ff87c6ba710 op status=0 http_status=403 ======
2017-04-20 16:43:04.916350 7ff87c6c0700 20 process_request() returned -2027
2017-04-20 16:43:04.916390 7ff87c6c0700 1 civetweb: 0x7ff990015610:
10.80.6.26 - - [20/Apr/2017:16:43:04 +0200] "GET /admin/log HTTP/1.1" 403 0
- -
2017-04-20 16:43:04.917212 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff9703a5440:18RGWMetaSyncShardCR: operate()
2017-04-20 16:43:04.917223 7ff9777e6700 20 rgw meta sync:
incremental_sync:1544: shard_id=20
mdlog_marker=1_1492686039.901886_5551978.1
sync_marker.marker=1_1492686039.901886_5551978.1 period_marker=
2017-04-20 16:43:04.917227 7ff9777e6700 20 rgw meta sync:
incremental_sync:1551: shard_id=20 syncing mdlog for shard_id=20
2017-04-20 16:43:04.917236 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.917238 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: init request
2017-04-20 16:43:04.917240 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.917241 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: reading shard status
2017-04-20 16:43:04.917303 7ff9777e6700 20 run: stack=0x7ff97000d420 is io
blocked
2017-04-20 16:43:04.918285 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.918295 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: reading shard status complete
2017-04-20 16:43:04.918307 7ff9777e6700 20 rgw meta sync: shard_id=20
marker=1_1492686039.901886_5551978.1 last_update=2017-04-20
13:00:39.0.901886s
2017-04-20 16:43:04.918316 7ff9777e6700 20
cr:s=0x7ff97000d420:op=0x7ff970066b80:24RGWCloneMetaLogCoroutine: operate()
2017-04-20 16:43:04.918317 7ff9777e6700 20 rgw meta sync: operate:
shard_id=20: sending rest request
2017-04-20 16:43:04.918381 7ff9777e6700 20 RGWEnv::set(): HTTP_DATE: Thu Apr
20 14:43:04 2017
2017-04-20 16:43:04.918390 7ff9777e6700 20 > HTTP_DATE -> Thu Apr 20
14:43:04 2017
2017-04-20 16:43:04.918404 7ff9777e6700 10 get_canon_resource():
dest=/admin/log
2017-04-20 16:43:04.918406 7ff9777e6700 10 generated canonical header: GET
--
Kind regards,
Ben Morrice
______________________________________________________________________
Ben Morrice | e: ben.morrice@xxxxxxx | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[client.radosgw.gateway]
host = bbpsrvc15
keyring = /etc/ceph/ceph.client.radosgw.gateway.keyring
rgw socket path = ""
log file = /var/log/ceph/radosgw.log
rgw_frontends = "civetweb port=80"
# rgw cache
rgw cache enabled = false
rgw thread pool size = 1024
rgw region = bbp-gva
rgw zone = bbp-gva-secondary
rgw dns name = s3.bbp.epfl.ch
# Keystone integration
rgw keystone url = https://bbpopenstack.epfl.ch:35357
rgw keystone admin user = admin
rgw keystone admin password = nottherealpassword
rgw keystone admin project = admin
rgw keystone admin domain = default
rgw keystone api version = 3
rgw keystone accepted roles = _member_, admin
rgw s3 auth use keystone = true
[client.radosgw.gateway]
host = bbpus16
keyring = /etc/ceph/ceph.client.radosgw.gateway.keyring
rgw socket path = ""
log file = /var/log/ceph/radosgw.log
rgw_frontends = "civetweb port=80"
# rgw cache
rgw cache enabled = false
rgw thread pool size = 1024
rgw region = bbp-gva
rgw zone = bbp-gva-master
rgw dns name = s3.bbp.epfl.ch
# Keystone integration
rgw keystone url = https://bbpopenstack.epfl.ch:35357
rgw keystone admin user = admin
rgw keystone admin password = nottherealpassword
rgw keystone admin tenant = admin
rgw keystone admin domain = default
rgw keystone api version = 3
rgw keystone accepted roles = _member_, admin
rgw s3 auth use keystone = true
{
"id": "6ea09956-60a7-48df-980c-2b5bbf71b565",
"epoch": 7,
"predecessor_uuid": "80026abd-49f4-436e-844f-f8743685dac5",
"sync_status": [
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
""
],
"period_map": {
"id": "6ea09956-60a7-48df-980c-2b5bbf71b565",
"zonegroups": [
{
"id": "bbp-gva",
"name": "bbp-gva",
"api_name": "",
"is_master": "true",
"endpoints": [
"http:\/\/bbpus16.epfl.ch:80\/"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "bbp-gva-master",
"zones": [
{
"id": "bbp-gva-master",
"name": "bbp-gva-master",
"endpoints": [
"http:\/\/bbpus16.epfl.ch:80\/"
],
"log_meta": "true",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
},
{
"id": "c54c61b9-0207-4841-af1e-f1b1bd1659da",
"name": "bbp-gva-secondary",
"endpoints": [
"http:\/\/bbpsrvc15.cscs.ch:80"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "b23771d0-6005-41da-8ee0-aec03db510d7"
}
],
"short_zone_ids": [
{
"key": "3e293dab-71aa-456d-9380-ebd4aca54376",
"val": 3022448784
},
{
"key": "bbp-gva-master",
"val": 1414621010
},
{
"key": "c54c61b9-0207-4841-af1e-f1b1bd1659da",
"val": 2616433292
},
{
"key": "d0bcdc95-d09a-44a0-ae6d-4251cac7b597",
"val": 1418041467
}
]
},
"master_zonegroup": "bbp-gva",
"master_zone": "bbp-gva-master",
"period_config": {
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
},
"realm_id": "b23771d0-6005-41da-8ee0-aec03db510d7",
"realm_name": "gold",
"realm_epoch": 2
}
[root@bbpsrvc15 ~]# radosgw-admin zonegroup get
{
"id": "bbp-gva",
"name": "bbp-gva",
"api_name": "",
"is_master": "true",
"endpoints": [
"http:\/\/bbpus16.epfl.ch:80\/"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "bbp-gva-master",
"zones": [
{
"id": "bbp-gva-master",
"name": "bbp-gva-master",
"endpoints": [
"http:\/\/bbpus16.epfl.ch:80\/"
],
"log_meta": "true",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
},
{
"id": "c54c61b9-0207-4841-af1e-f1b1bd1659da",
"name": "bbp-gva-secondary",
"endpoints": [
"http:\/\/bbpsrvc15.cscs.ch:80"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "b23771d0-6005-41da-8ee0-aec03db510d7"
}
[root@bbpsrvc15 ~]# radosgw-admin zone get
{
"id": "c54c61b9-0207-4841-af1e-f1b1bd1659da",
"name": "bbp-gva-secondary",
"domain_root": ".bbp-gva-secondary.domain.rgw",
"control_pool": ".bbp-gva-secondary.rgw.control",
"gc_pool": ".bbp-gva-secondary.rgw.gc",
"log_pool": ".bbp-gva-secondary.log",
"intent_log_pool": ".bbp-gva-secondary.intent-log",
"usage_log_pool": ".bbp-gva-secondary.usage",
"user_keys_pool": ".bbp-gva-secondary.users",
"user_email_pool": ".bbp-gva-secondary.users.email",
"user_swift_pool": ".bbp-gva-secondary.users.swift",
"user_uid_pool": ".bbp-gva-secondary.users.uid",
"system_key": {
"access_key": "A0QV52GF9GT3A13ADA6C",
"secret_key": "wPMQNfA8Uf6Q6imRxP2L612A11KufPZ0man5vQsQ"
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": ".bbp-gva-secondary.rgw.buckets.index",
"data_pool": ".bbp-gva-secondary.rgw.buckets",
"data_extra_pool": ".bbp-gva-secondary.rgw.buckets.extra",
"index_type": 0
}
}
],
"metadata_heap": ".bbp-gva-secondary.rgw.meta",
"realm_id": "b23771d0-6005-41da-8ee0-aec03db510d7"
}
[root@bbpus16 ~]# radosgw-admin period get
{
"id": "6ea09956-60a7-48df-980c-2b5bbf71b565",
"epoch": 7,
"predecessor_uuid": "80026abd-49f4-436e-844f-f8743685dac5",
"sync_status": [
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
""
],
"period_map": {
"id": "6ea09956-60a7-48df-980c-2b5bbf71b565",
"zonegroups": [
{
"id": "bbp-gva",
"name": "bbp-gva",
"api_name": "",
"is_master": "true",
"endpoints": [
"http:\/\/bbpus16.epfl.ch:80\/"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "bbp-gva-master",
"zones": [
{
"id": "bbp-gva-master",
"name": "bbp-gva-master",
"endpoints": [
"http:\/\/bbpus16.epfl.ch:80\/"
],
"log_meta": "true",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
},
{
"id": "c54c61b9-0207-4841-af1e-f1b1bd1659da",
"name": "bbp-gva-secondary",
"endpoints": [
"http:\/\/bbpsrvc15.cscs.ch:80"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "b23771d0-6005-41da-8ee0-aec03db510d7"
}
],
"short_zone_ids": [
{
"key": "3e293dab-71aa-456d-9380-ebd4aca54376",
"val": 3022448784
},
{
"key": "bbp-gva-master",
"val": 1414621010
},
{
"key": "c54c61b9-0207-4841-af1e-f1b1bd1659da",
"val": 2616433292
},
{
"key": "d0bcdc95-d09a-44a0-ae6d-4251cac7b597",
"val": 1418041467
}
]
},
"master_zonegroup": "bbp-gva",
"master_zone": "bbp-gva-master",
"period_config": {
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
},
"realm_id": "b23771d0-6005-41da-8ee0-aec03db510d7",
"realm_name": "gold",
"realm_epoch": 2
}
[root@bbpus16 ~]# radosgw-admin zonegroup get
{
"id": "bbp-gva",
"name": "bbp-gva",
"api_name": "",
"is_master": "true",
"endpoints": [
"http:\/\/bbpus16.epfl.ch:80\/"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "bbp-gva-master",
"zones": [
{
"id": "bbp-gva-master",
"name": "bbp-gva-master",
"endpoints": [
"http:\/\/bbpus16.epfl.ch:80\/"
],
"log_meta": "true",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
},
{
"id": "c54c61b9-0207-4841-af1e-f1b1bd1659da",
"name": "bbp-gva-secondary",
"endpoints": [
"http:\/\/bbpsrvc15.cscs.ch:80"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "b23771d0-6005-41da-8ee0-aec03db510d7"
}
[root@bbpus16 ~]# radosgw-admin period get
{
"id": "6ea09956-60a7-48df-980c-2b5bbf71b565",
"epoch": 7,
"predecessor_uuid": "80026abd-49f4-436e-844f-f8743685dac5",
"sync_status": [
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
""
],
"period_map": {
"id": "6ea09956-60a7-48df-980c-2b5bbf71b565",
"zonegroups": [
{
"id": "bbp-gva",
"name": "bbp-gva",
"api_name": "",
"is_master": "true",
"endpoints": [
"http:\/\/bbpus16.epfl.ch:80\/"
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "bbp-gva-master",
"zones": [
{
"id": "bbp-gva-master",
"name": "bbp-gva-master",
"endpoints": [
"http:\/\/bbpus16.epfl.ch:80\/"
],
"log_meta": "true",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
},
{
"id": "c54c61b9-0207-4841-af1e-f1b1bd1659da",
"name": "bbp-gva-secondary",
"endpoints": [
"http:\/\/bbpsrvc15.cscs.ch:80"
],
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "b23771d0-6005-41da-8ee0-aec03db510d7"
}
],
"short_zone_ids": [
{
"key": "3e293dab-71aa-456d-9380-ebd4aca54376",
"val": 3022448784
},
{
"key": "bbp-gva-master",
"val": 1414621010
},
{
"key": "c54c61b9-0207-4841-af1e-f1b1bd1659da",
"val": 2616433292
},
{
"key": "d0bcdc95-d09a-44a0-ae6d-4251cac7b597",
"val": 1418041467
}
]
},
"master_zonegroup": "bbp-gva",
"master_zone": "bbp-gva-master",
"period_config": {
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
},
"realm_id": "b23771d0-6005-41da-8ee0-aec03db510d7",
"realm_name": "gold",
"realm_epoch": 2
}
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
