Hello Everyone,
I have a Ceph test setup with 3 mons, 3 RGWs, 5 OSD nodes and 22 OSDs. RadosGW instances run on the monitor nodes and they are behind a load balancer. I run RGW instances in the full debug mode (20/20 for rgw and 20/20 for civet web).
I can easily access RGW via S3 API with any user including the admin. When I try to use Admin Ops API with the admin user, I get the errno=-1 and 403 https errors with the following details.
2018-02-07 14:22:51.308143 7ff3f4909700 20 RGWEnv::set(): HTTP_ACCEPT: text/plain, text/plain, application/json, application/*+json, */*, */*
2018-02-07 14:22:51.308190 7ff3f4909700 20 RGWEnv::set(): HTTP_USER_AGENT: Java/1.8.0_144
2018-02-07 14:22:51.308194 7ff3f4909700 20 RGWEnv::set(): HTTP_HOST: uyum.in
2018-02-07 14:22:51.308201 7ff3f4909700 20 RGWEnv::set(): HTTP_CONNECTION: keep-alive
2018-02-07 14:22:51.308205 7ff3f4909700 20 RGWEnv::set(): REQUEST_METHOD: GET
2018-02-07 14:22:51.308207 7ff3f4909700 20 RGWEnv::set(): REQUEST_URI: /admin/user/
2018-02-07 14:22:51.308210 7ff3f4909700 20 RGWEnv::set(): SCRIPT_URI: /admin/user/
2018-02-07 14:22:51.308215 7ff3f4909700 20 RGWEnv::set(): SERVER_PORT: 0
2018-02-07 14:22:51.308217 7ff3f4909700 20 RGWEnv::set(): SERVER_PORT_SECURE: 443
2018-02-07 14:22:51.308219 7ff3f4909700 20 HTTP_ACCEPT=text/plain, text/plain, application/json, application/*+json, */*, */*
2018-02-07 14:22:51.308222 7ff3f4909700 20 HTTP_CONNECTION=keep-alive
2018-02-07 14:22:51.308223 7ff3f4909700 20 HTTP_HOST=uyum.in
2018-02-07 14:22:51.308224 7ff3f4909700 20 HTTP_USER_AGENT=Java/1.8.0_144
2018-02-07 14:22:51.308227 7ff3f4909700 20 REQUEST_METHOD=GET
2018-02-07 14:22:51.308228 7ff3f4909700 20 REQUEST_URI=/admin/user/
2018-02-07 14:22:51.308229 7ff3f4909700 20 SCRIPT_URI=/admin/user/
2018-02-07 14:22:51.308230 7ff3f4909700 20 SERVER_PORT=0
2018-02-07 14:22:51.308231 7ff3f4909700 20 SERVER_PORT_SECURE=443
2018-02-07 14:22:51.308234 7ff3f4909700 1 ====== starting new request req=0x7ff3f49033f0 =====
2018-02-07 14:22:51.308323 7ff3f4909700 2 req 1:0.000084::GET /admin/user/::initializing for trans_id = tx000000000000000000001-005a7ae18b-130b-default
2018-02-07 14:22:51.308341 7ff3f4909700 10 rgw api priority: s3=5 s3website=4
2018-02-07 14:22:51.308346 7ff3f4909700 10 host=uyum.in
2018-02-07 14:22:51.308360 7ff3f4909700 20 subdomain= domain=uyum.in in_hosted_domain=1 in_hosted_domain_s3website=0
2018-02-07 14:22:51.308364 7ff3f4909700 20 final domain/bucket subdomain= domain=uyum.in in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=uyum.in s->info.request_uri=/admin/user/
2018-02-07 14:22:51.308462 7ff3f4909700 10 handler=15RGWHandler_User
2018-02-07 14:22:51.308471 7ff3f4909700 2 req 1:0.000237::GET /admin/user/::getting op 0
2018-02-07 14:22:51.308641 7ff3f4909700 10 op=15RGWOp_User_Info
2018-02-07 14:22:51.308649 7ff3f4909700 2 req 1:0.000415::GET /admin/user/:get_user_info:authorizing
2018-02-07 14:22:51.308658 7ff3f4909700 2 req 1:0.000424::GET /admin/user/:get_user_info:normalizing buckets and tenants
2018-02-07 14:22:51.308661 7ff3f4909700 2 req 1:0.000427::GET /admin/user/:get_user_info:init permissions
2018-02-07 14:22:51.308682 7ff3f4909700 2 req 1:0.000436::GET /admin/user/:get_user_info:recalculating target
2018-02-07 14:22:51.308688 7ff3f4909700 2 req 1:0.000453::GET /admin/user/:get_user_info:reading permissions
2018-02-07 14:22:51.308691 7ff3f4909700 2 req 1:0.000456::GET /admin/user/:get_user_info:init op
2018-02-07 14:22:51.308694 7ff3f4909700 2 req 1:0.000460::GET /admin/user/:get_user_info:verifying op mask
2018-02-07 14:22:51.308697 7ff3f4909700 20 required_mask= 0 user.op_mask=7
2018-02-07 14:22:51.308700 7ff3f4909700 2 req 1:0.000466::GET /admin/user/:get_user_info:verifying op permissions
2018-02-07 14:22:51.308709 7ff3f4909700 20 op->ERRORHANDLER: err_no=-1 new_err_no=-1
2018-02-07 14:22:51.309065 7ff3f4909700 2 req 1:0.000831::GET /admin/user/:get_user_info:op status=0
2018-02-07 14:22:51.309084 7ff3f4909700 2 req 1:0.000850::GET /admin/user/:get_user_info:http status=403
2018-02-07 14:22:51.309097 7ff3f4909700 1 ====== req done req=0x7ff3f49033f0 op status=0 http_status=403 ======
2018-02-07 14:22:51.309108 7ff3f4909700 20 process_request() returned -1
2018-02-07 14:22:51.309205 7ff3f4909700 1 civetweb: 0x555dc0220000: 192.168.164.23 - - [07/Feb/2018:14:22:51 +0300] "GET /admin/user/ HTTP/1.1" 1 0 - Java/1.8.0_144
The request has the following parameters, keys are hidden:
I have a Ceph test setup with 3 mons, 3 RGWs, 5 OSD nodes and 22 OSDs. RadosGW instances run on the monitor nodes and they are behind a load balancer. I run RGW instances in the full debug mode (20/20 for rgw and 20/20 for civet web).
I can easily access RGW via S3 API with any user including the admin. When I try to use Admin Ops API with the admin user, I get the errno=-1 and 403 https errors with the following details.
2018-02-07 14:22:51.308143 7ff3f4909700 20 RGWEnv::set(): HTTP_ACCEPT: text/plain, text/plain, application/json, application/*+json, */*, */*
2018-02-07 14:22:51.308190 7ff3f4909700 20 RGWEnv::set(): HTTP_USER_AGENT: Java/1.8.0_144
2018-02-07 14:22:51.308194 7ff3f4909700 20 RGWEnv::set(): HTTP_HOST: uyum.in
2018-02-07 14:22:51.308201 7ff3f4909700 20 RGWEnv::set(): HTTP_CONNECTION: keep-alive
2018-02-07 14:22:51.308205 7ff3f4909700 20 RGWEnv::set(): REQUEST_METHOD: GET
2018-02-07 14:22:51.308207 7ff3f4909700 20 RGWEnv::set(): REQUEST_URI: /admin/user/
2018-02-07 14:22:51.308210 7ff3f4909700 20 RGWEnv::set(): SCRIPT_URI: /admin/user/
2018-02-07 14:22:51.308215 7ff3f4909700 20 RGWEnv::set(): SERVER_PORT: 0
2018-02-07 14:22:51.308217 7ff3f4909700 20 RGWEnv::set(): SERVER_PORT_SECURE: 443
2018-02-07 14:22:51.308219 7ff3f4909700 20 HTTP_ACCEPT=text/plain, text/plain, application/json, application/*+json, */*, */*
2018-02-07 14:22:51.308222 7ff3f4909700 20 HTTP_CONNECTION=keep-alive
2018-02-07 14:22:51.308223 7ff3f4909700 20 HTTP_HOST=uyum.in
2018-02-07 14:22:51.308224 7ff3f4909700 20 HTTP_USER_AGENT=Java/1.8.0_144
2018-02-07 14:22:51.308227 7ff3f4909700 20 REQUEST_METHOD=GET
2018-02-07 14:22:51.308228 7ff3f4909700 20 REQUEST_URI=/admin/user/
2018-02-07 14:22:51.308229 7ff3f4909700 20 SCRIPT_URI=/admin/user/
2018-02-07 14:22:51.308230 7ff3f4909700 20 SERVER_PORT=0
2018-02-07 14:22:51.308231 7ff3f4909700 20 SERVER_PORT_SECURE=443
2018-02-07 14:22:51.308234 7ff3f4909700 1 ====== starting new request req=0x7ff3f49033f0 =====
2018-02-07 14:22:51.308323 7ff3f4909700 2 req 1:0.000084::GET /admin/user/::initializing for trans_id = tx000000000000000000001-005a7ae18b-130b-default
2018-02-07 14:22:51.308341 7ff3f4909700 10 rgw api priority: s3=5 s3website=4
2018-02-07 14:22:51.308346 7ff3f4909700 10 host=uyum.in
2018-02-07 14:22:51.308360 7ff3f4909700 20 subdomain= domain=uyum.in in_hosted_domain=1 in_hosted_domain_s3website=0
2018-02-07 14:22:51.308364 7ff3f4909700 20 final domain/bucket subdomain= domain=uyum.in in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=uyum.in s->info.request_uri=/admin/user/
2018-02-07 14:22:51.308462 7ff3f4909700 10 handler=15RGWHandler_User
2018-02-07 14:22:51.308471 7ff3f4909700 2 req 1:0.000237::GET /admin/user/::getting op 0
2018-02-07 14:22:51.308641 7ff3f4909700 10 op=15RGWOp_User_Info
2018-02-07 14:22:51.308649 7ff3f4909700 2 req 1:0.000415::GET /admin/user/:get_user_info:authorizing
2018-02-07 14:22:51.308658 7ff3f4909700 2 req 1:0.000424::GET /admin/user/:get_user_info:normalizing buckets and tenants
2018-02-07 14:22:51.308661 7ff3f4909700 2 req 1:0.000427::GET /admin/user/:get_user_info:init permissions
2018-02-07 14:22:51.308682 7ff3f4909700 2 req 1:0.000436::GET /admin/user/:get_user_info:recalculating target
2018-02-07 14:22:51.308688 7ff3f4909700 2 req 1:0.000453::GET /admin/user/:get_user_info:reading permissions
2018-02-07 14:22:51.308691 7ff3f4909700 2 req 1:0.000456::GET /admin/user/:get_user_info:init op
2018-02-07 14:22:51.308694 7ff3f4909700 2 req 1:0.000460::GET /admin/user/:get_user_info:verifying op mask
2018-02-07 14:22:51.308697 7ff3f4909700 20 required_mask= 0 user.op_mask=7
2018-02-07 14:22:51.308700 7ff3f4909700 2 req 1:0.000466::GET /admin/user/:get_user_info:verifying op permissions
2018-02-07 14:22:51.308709 7ff3f4909700 20 op->ERRORHANDLER: err_no=-1 new_err_no=-1
2018-02-07 14:22:51.309065 7ff3f4909700 2 req 1:0.000831::GET /admin/user/:get_user_info:op status=0
2018-02-07 14:22:51.309084 7ff3f4909700 2 req 1:0.000850::GET /admin/user/:get_user_info:http status=403
2018-02-07 14:22:51.309097 7ff3f4909700 1 ====== req done req=0x7ff3f49033f0 op status=0 http_status=403 ======
2018-02-07 14:22:51.309108 7ff3f4909700 20 process_request() returned -1
2018-02-07 14:22:51.309205 7ff3f4909700 1 civetweb: 0x555dc0220000: 192.168.164.23 - - [07/Feb/2018:14:22:51 +0300] "GET /admin/user/ HTTP/1.1" 1 0 - Java/1.8.0_144
The request has the following parameters, keys are hidden:
String endpointUrl = "https://uyum.io/admin/user”;
String accessKey = “***”;
String secretKey = “***”;
String urlPath = "/";
uriParams.put("format", "json");
uriParams.put("uid", “user1”)
My admin user has all the required caps (see the output of command rados-admin user info —uid “admin-api-user”, keys are hidden).
{
"user_id": "admin-api-user",
"display_name": "Admin API User",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "admin-api-user",
"access_key": “***",
"secret_key": “***"
}
],
"swift_keys": [],
"caps": [
{
"type": "buckets",
"perm": "*"
},
{
"type": "metadata",
"perm": "*"
},
{
"type": "usage",
"perm": "*"
},
{
"type": "users",
"perm": "*"
},
{
"type": "zone",
"perm": "*"
}
],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": true,
"check_on_raw": false,
"max_size": 268435456000,
"max_size_kb": 262144000,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw”
}
I googled the error without any success. Does anybody have any idea about the problem? Am i missing something?
Best regards,
Huseyin
{
"user_id": "admin-api-user",
"display_name": "Admin API User",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "admin-api-user",
"access_key": “***",
"secret_key": “***"
}
],
"swift_keys": [],
"caps": [
{
"type": "buckets",
"perm": "*"
},
{
"type": "metadata",
"perm": "*"
},
{
"type": "usage",
"perm": "*"
},
{
"type": "users",
"perm": "*"
},
{
"type": "zone",
"perm": "*"
}
],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": true,
"check_on_raw": false,
"max_size": 268435456000,
"max_size_kb": 262144000,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw”
}
I googled the error without any success. Does anybody have any idea about the problem? Am i missing something?
Best regards,
Huseyin
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
