Hi Cephers,
I'm trying to configure RGWs with multiple zonegroups within single realm.
The intention is that some buckets to be replicated and others to stay locally.
e.g.:
realm: fj
zonegroup east: zone tokyo (not replicated)
zonegroup west: zone osaka (not replicated)
zonegroup jp: zone jp-east + jp-west (replicated)
To evaluate such configuration, I tentatively built multiple zonegroups
(east, west) on a ceph cluster. I barely succeed to configure it, but
some concerns exist.
a) User accounts are not synced among zonegroups
I'm not sure if this is a issue, but the blueprint [1] stated a master
zonegroup manages user accounts as metadata like buckets.
b) Bucket creation is rejected if master zonegroup doesn't have the account
e.g.:
1) Configure east zonegroup as master.
2) Create a user "nishi" on west zonegroup (osaka zone) using radosgw-admin.
3) Try to create a bucket on west zonegroup by user nishi.
-> ERROR: S3 error: 404 (NoSuchKey)
4) Create user nishi on east zonegroup with same key.
5) Succeed to create a bucket on west zonegroup by user nishi.
c) How to restrict to place buckets on specific zonegroups?
If user accounts would synced future as the blueprint, all the zonegroups
contain same account information. It means any user can create buckets on
any zonegroups. If we want to permit to place buckets on a replicated
zonegroup for specific users, how to configure?
If user accounts will not synced as current behavior, we can restrict
to place buckets on specific zonegroups. But I cannot find best way to
configure the master zonegroup.
d) Operations for other zonegroup are not redirected
e.g.:
1) Create bucket4 on west zonegroup by nishi.
2) Try to access bucket4 from endpoint on east zonegroup.
-> Respond "301 (Moved Permanently)",
but no redirected Location header is returned.
It seems current RGW doesn't follows S3 specification [2].
To implement this feature, probably we need to define another endpoint
on each zonegroup for client accessible URL. RGW may placed behind proxy,
thus the URL may be different from endpoint URLs for replication.
Any thoughts?
[1] http://tracker.ceph.com/projects/ceph/wiki/Rgw_new_multisite_configuration
[2] http://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.html
------ FYI ------
[environments]
Ceph cluster: RHCS 2.0
RGW: RHEL 7.2 + RGW v10.2.5
zonegroup east: master
zone tokyo
endpoint http://node5:80
system user: sync-user
user azuma (+ nishi)
zonegroup west: (not master)
zone osaka
endpoint http://node5:8081
system user: sync-user (created with same key as zone tokyo)
user nishi
[detail of "b)"]
$ s3cmd -c s3nishi.cfg ls
$ s3cmd -c s3nishi.cfg mb s3://bucket3
ERROR: S3 error: 404 (NoSuchKey)
---- rgw.osaka log:
2017-02-10 11:54:13.290653 7feac3f7f700 1 ====== starting new request req=0x7feac3f79710 =====
2017-02-10 11:54:13.290709 7feac3f7f700 2 req 50:0.000057::PUT /bucket3/::initializing for trans_id = tx000000000000000000032-00589d2b55-14a2-osaka
2017-02-10 11:54:13.290720 7feac3f7f700 10 rgw api priority: s3=5 s3website=4
2017-02-10 11:54:13.290722 7feac3f7f700 10 host=node5
2017-02-10 11:54:13.290733 7feac3f7f700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2017-02-10 11:54:13.290750 7feac3f7f700 10 meta>> HTTP_X_AMZ_DATE
2017-02-10 11:54:13.290753 7feac3f7f700 10 x>> x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-10 11:54:13.290755 7feac3f7f700 10 x>> x-amz-date:20170210T025413Z
2017-02-10 11:54:13.290774 7feac3f7f700 10 handler=25RGWHandler_REST_Bucket_S3
2017-02-10 11:54:13.290775 7feac3f7f700 2 req 50:0.000124:s3:PUT /bucket3/::getting op 1
2017-02-10 11:54:13.290781 7feac3f7f700 10 op=27RGWCreateBucket_ObjStore_S3
2017-02-10 11:54:13.290782 7feac3f7f700 2 req 50:0.000130:s3:PUT /bucket3/:create_bucket:authorizing
2017-02-10 11:54:13.290798 7feac3f7f700 10 v4 signature format = 989404f270efd800843cb19183c53dc457cf96b9ea2393ba5d554a42ffc22f76
2017-02-10 11:54:13.290804 7feac3f7f700 10 v4 credential format = ZY6EJUVB38SCOWBELERQ/20170210/west/s3/aws4_request
2017-02-10 11:54:13.290806 7feac3f7f700 10 access key id = ZY6EJUVB38SCOWBELERQ
2017-02-10 11:54:13.290814 7feac3f7f700 10 credential scope = 20170210/west/s3/aws4_request
2017-02-10 11:54:13.290834 7feac3f7f700 10 canonical headers format = host:node5:8081
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
x-amz-date:20170210T025413Z
2017-02-10 11:54:13.290836 7feac3f7f700 10 delaying v4 auth
2017-02-10 11:54:13.290839 7feac3f7f700 2 req 50:0.000187:s3:PUT /bucket3/:create_bucket:normalizing buckets and tenants
2017-02-10 11:54:13.290841 7feac3f7f700 10 s->object=<NULL> s->bucket=bucket3
2017-02-10 11:54:13.290843 7feac3f7f700 2 req 50:0.000191:s3:PUT /bucket3/:create_bucket:init permissions
2017-02-10 11:54:13.290844 7feac3f7f700 2 req 50:0.000192:s3:PUT /bucket3/:create_bucket:recalculating target
2017-02-10 11:54:13.290845 7feac3f7f700 2 req 50:0.000193:s3:PUT /bucket3/:create_bucket:reading permissions
2017-02-10 11:54:13.290846 7feac3f7f700 2 req 50:0.000195:s3:PUT /bucket3/:create_bucket:init op
2017-02-10 11:54:13.290847 7feac3f7f700 2 req 50:0.000196:s3:PUT /bucket3/:create_bucket:verifying op mask
2017-02-10 11:54:13.290849 7feac3f7f700 2 req 50:0.000197:s3:PUT /bucket3/:create_bucket:verifying op permissions
2017-02-10 11:54:13.292027 7feac3f7f700 2 req 50:0.001374:s3:PUT /bucket3/:create_bucket:verifying op params
2017-02-10 11:54:13.292035 7feac3f7f700 2 req 50:0.001383:s3:PUT /bucket3/:create_bucket:pre-executing
2017-02-10 11:54:13.292037 7feac3f7f700 2 req 50:0.001385:s3:PUT /bucket3/:create_bucket:executing
2017-02-10 11:54:13.292072 7feac3f7f700 10 payload request hash = d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-10 11:54:13.292083 7feac3f7f700 10 canonical request = PUT
/bucket3/
host:node5:8081
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
x-amz-date:20170210T025413Z
host;x-amz-content-sha256;x-amz-date
d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-10 11:54:13.292084 7feac3f7f700 10 canonical request hash = 8faa5ec57f69dd7b54baa72c157b6d63f8c7db309a34a1e2a10ad6f2f585cd02
2017-02-10 11:54:13.292087 7feac3f7f700 10 string to sign = AWS4-HMAC-SHA256
20170210T025413Z
20170210/west/s3/aws4_request
8faa5ec57f69dd7b54baa72c157b6d63f8c7db309a34a1e2a10ad6f2f585cd02
2017-02-10 11:54:13.292118 7feac3f7f700 10 date_k = 454f3ad73c095e73d2482809d7a6ec8af3c4e900bc83e0a9663ea5fc336cad95
2017-02-10 11:54:13.292131 7feac3f7f700 10 region_k = e0caaddbb30ebc25840b6aaac3979d1881a14b8e9a0dfea43d8a006c8e0e504d
2017-02-10 11:54:13.292144 7feac3f7f700 10 service_k = 59d6c9158e9e3c6a1aa97ee15859d2ef9ad9c64209b63f093109844f0c7f6c04
2017-02-10 11:54:13.292171 7feac3f7f700 10 signing_k = 4dcbccd9c3da779d32758a645644c66a56f64d642eaeb39eec8e0b2facba7805
2017-02-10 11:54:13.292197 7feac3f7f700 10 signature_k = 989404f270efd800843cb19183c53dc457cf96b9ea2393ba5d554a42ffc22f76
2017-02-10 11:54:13.292198 7feac3f7f700 10 new signature = 989404f270efd800843cb19183c53dc457cf96b9ea2393ba5d554a42ffc22f76
2017-02-10 11:54:13.292199 7feac3f7f700 10 ----------------------------- Verifying signatures
2017-02-10 11:54:13.292199 7feac3f7f700 10 Signature = 989404f270efd800843cb19183c53dc457cf96b9ea2393ba5d554a42ffc22f76
2017-02-10 11:54:13.292200 7feac3f7f700 10 New Signature = 989404f270efd800843cb19183c53dc457cf96b9ea2393ba5d554a42ffc22f76
2017-02-10 11:54:13.292200 7feac3f7f700 10 -----------------------------
2017-02-10 11:54:13.292202 7feac3f7f700 10 v4 auth ok
2017-02-10 11:54:13.292238 7feac3f7f700 10 create bucket location constraint: west
2017-02-10 11:54:13.292256 7feac3f7f700 10 cache get: name=osaka.rgw.data.root+bucket3 : type miss (requested=22, cached=0)
2017-02-10 11:54:13.293369 7feac3f7f700 10 cache put: name=osaka.rgw.data.root+bucket3 info.flags=0
2017-02-10 11:54:13.293374 7feac3f7f700 10 moving osaka.rgw.data.root+bucket3 to cache LRU end
2017-02-10 11:54:13.293380 7feac3f7f700 0 sending create_bucket request to master zonegroup
2017-02-10 11:54:13.293401 7feac3f7f700 10 get_canon_resource(): dest=/bucket3/
2017-02-10 11:54:13.293403 7feac3f7f700 10 generated canonical header: PUT
Fri Feb 10 02:54:13 2017
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
/bucket3/
2017-02-10 11:54:13.299113 7feac3f7f700 10 receive_http_header
2017-02-10 11:54:13.299117 7feac3f7f700 10 received header:HTTP/1.1 404 Not Found
2017-02-10 11:54:13.299119 7feac3f7f700 10 receive_http_header
2017-02-10 11:54:13.299120 7feac3f7f700 10 received header:x-amz-request-id: tx000000000000000000005-00589d2b55-1416-tokyo
2017-02-10 11:54:13.299130 7feac3f7f700 10 receive_http_header
2017-02-10 11:54:13.299131 7feac3f7f700 10 received header:Content-Length: 175
2017-02-10 11:54:13.299133 7feac3f7f700 10 receive_http_header
2017-02-10 11:54:13.299133 7feac3f7f700 10 received header:Accept-Ranges: bytes
2017-02-10 11:54:13.299148 7feac3f7f700 10 receive_http_header
2017-02-10 11:54:13.299149 7feac3f7f700 10 received header:Content-Type: application/xml
2017-02-10 11:54:13.299150 7feac3f7f700 10 receive_http_header
2017-02-10 11:54:13.299150 7feac3f7f700 10 received header:Date: Fri, 10 Feb 2017 02:54:13 GMT
2017-02-10 11:54:13.299152 7feac3f7f700 10 receive_http_header
2017-02-10 11:54:13.299152 7feac3f7f700 10 received header:
2017-02-10 11:54:13.299248 7feac3f7f700 2 req 50:0.008596:s3:PUT /bucket3/:create_bucket:completing
2017-02-10 11:54:13.299319 7feac3f7f700 2 req 50:0.008667:s3:PUT /bucket3/:create_bucket:op status=-2
2017-02-10 11:54:13.299321 7feac3f7f700 2 req 50:0.008670:s3:PUT /bucket3/:create_bucket:http status=404
2017-02-10 11:54:13.299324 7feac3f7f700 1 ====== req done req=0x7feac3f79710 op status=-2 http_status=404 ======
2017-02-10 11:54:13.299349 7feac3f7f700 1 civetweb: 0x7feb2c02d340: 192.168.20.15 - - [10/Feb/2017:11:54:13 +0900] "PUT /bucket3/ HTTP/1.1" 404 0 - -
---- rgw.tokyo log:
2017-02-10 11:54:13.297852 7f56076c6700 1 ====== starting new request req=0x7f56076c0710 =====
2017-02-10 11:54:13.297887 7f56076c6700 2 req 5:0.000035::PUT /bucket3/::initializing for trans_id = tx000000000000000000005-00589d2b55-1416-tokyo
2017-02-10 11:54:13.297895 7f56076c6700 10 rgw api priority: s3=5 s3website=4
2017-02-10 11:54:13.297897 7f56076c6700 10 host=node5
2017-02-10 11:54:13.297906 7f56076c6700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2017-02-10 11:54:13.297912 7f56076c6700 10 x>> x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
2017-02-10 11:54:13.297929 7f56076c6700 10 handler=25RGWHandler_REST_Bucket_S3
2017-02-10 11:54:13.297937 7f56076c6700 2 req 5:0.000086:s3:PUT /bucket3/::getting op 1
2017-02-10 11:54:13.297946 7f56076c6700 10 op=27RGWCreateBucket_ObjStore_S3
2017-02-10 11:54:13.297947 7f56076c6700 2 req 5:0.000096:s3:PUT /bucket3/:create_bucket:authorizing
2017-02-10 11:54:13.297969 7f56076c6700 10 get_canon_resource(): dest=/bucket3/
2017-02-10 11:54:13.297976 7f56076c6700 10 auth_hdr:
PUT
Fri Feb 10 02:54:13 2017
x-amz-content-sha256:d8f96fbdf666b991d183a7f5cc7fcf6eaa10934786f67575bda3f734a772464a
/bucket3/
2017-02-10 11:54:13.298023 7f56076c6700 10 cache get: name=default.rgw.users.uid+nishi : type miss (requested=6, cached=0)
2017-02-10 11:54:13.298975 7f56076c6700 10 cache put: name=default.rgw.users.uid+nishi info.flags=0
2017-02-10 11:54:13.298986 7f56076c6700 10 moving default.rgw.users.uid+nishi to cache LRU end
2017-02-10 11:54:13.298991 7f56076c6700 0 User lookup failed!
2017-02-10 11:54:13.298993 7f56076c6700 10 failed to authorize request
2017-02-10 11:54:13.299077 7f56076c6700 2 req 5:0.001225:s3:PUT /bucket3/:create_bucket:op status=0
2017-02-10 11:54:13.299086 7f56076c6700 2 req 5:0.001235:s3:PUT /bucket3/:create_bucket:http status=404
2017-02-10 11:54:13.299089 7f56076c6700 1 ====== req done req=0x7f56076c0710 op status=0 http_status=404 ======
2017-02-10 11:54:13.299426 7f56076c6700 1 civetweb: 0x7f56200048c0: 192.168.20.15 - - [10/Feb/2017:11:54:13 +0900] "PUT /bucket3/ HTTP/1.1" 404 0 - -
--
KIMURA Osamu / 木村 修
Engineering Department, Storage Development Division,
Data Center Platform Business Unit, FUJITSU LIMITED
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
