Nick,
Thanks a lot for you input so far.
I re-ran the fix script from scratch and it turned out I made some mistakes in the process. I managed to run it correctly and now I am able to create buckets but I still can't upload anything. I looked for any issues in our configuration by searching at zonegroups, zones etc. but I haven't found anything missing there or in the logs. I'm attaching a log for failed file upload to an existing bucket and output of RGW configuration.
1. Creating bucket
2016-07-22 09:40:17.579446 7f40547f8700 20 RGWEnv::set(): HTTP_HOST: 10.1.68.29:8080
2016-07-22 09:40:17.579461 7f40547f8700 20 RGWEnv::set(): HTTP_ACCEPT_ENCODING: identity
2016-07-22 09:40:17.579462 7f40547f8700 20 RGWEnv::set(): CONTENT_LENGTH: 0
2016-07-22 09:40:17.579463 7f40547f8700 20 RGWEnv::set(): HTTP_X_AMZ_CONTENT_SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-07-22 09:40:17.579466 7f40547f8700 20 RGWEnv::set(): HTTP_X_AMZ_STORAGE_CLASS: STANDARD
2016-07-22 09:40:17.579479 7f40547f8700 20 RGWEnv::set(): HTTP_X_AMZ_META_S3CMD_ATTRS: uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1469007939/atime:1469007939/md5:d8160ddb9f4681ec985e03429f842b88/ctime:1469023832
2016-07-22 09:40:17.579481 7f40547f8700 20 RGWEnv::set(): HTTP_X_AMZ_DATE: 20160722T094017Z
2016-07-22 09:40:17.579482 7f40547f8700 20 RGWEnv::set(): CONTENT_TYPE: application/octet-stream
2016-07-22 09:40:17.579483 7f40547f8700 20 RGWEnv::set(): HTTP_AUTHORIZATION: AWS4-HMAC-SHA256 Credential=7VM2JP5QFARP8UMUW2KH/20160722/US/s3/aws4_request,SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=dac5cb849ed2057d925a43f702b9e1f135618fd04d95beb943954df1d7c0df1c
2016-07-22 09:40:17.579485 7f40547f8700 20 RGWEnv::set(): REQUEST_METHOD: POST
2016-07-22 09:40:17.579486 7f40547f8700 20 RGWEnv::set(): REQUEST_URI: /test-bucket-0/s3-test-file-1
2016-07-22 09:40:17.579486 7f40547f8700 20 RGWEnv::set(): QUERY_STRING: uploads
2016-07-22 09:40:17.579488 7f40547f8700 20 RGWEnv::set(): REMOTE_USER:
2016-07-22 09:40:17.579489 7f40547f8700 20 RGWEnv::set(): SCRIPT_URI: /test-bucket-0/s3-test-file-1
2016-07-22 09:40:17.579492 7f40547f8700 20 RGWEnv::set(): SERVER_PORT: 8080
2016-07-22 09:40:17.579493 7f40547f8700 20 CONTENT_LENGTH=0
2016-07-22 09:40:17.579494 7f40547f8700 20 CONTENT_TYPE=application/octet-stream
2016-07-22 09:40:17.579494 7f40547f8700 20 HTTP_ACCEPT_ENCODING=identity
2016-07-22 09:40:17.579498 7f40547f8700 20 HTTP_AUTHORIZATION=AWS4-HMAC-SHA256 Credential=7VM2JP5QFARP8UMUW2KH/20160722/US/s3/aws4_request,SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class,Signature=dac5cb849ed2057d925a43f702b9e1f135618fd04d95beb943954df1d7c0df1c
2016-07-22 09:40:17.579499 7f40547f8700 20 HTTP_HOST=10.1.68.29:8080
2016-07-22 09:40:17.579499 7f40547f8700 20 HTTP_X_AMZ_CONTENT_SHA256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-07-22 09:40:17.579500 7f40547f8700 20 HTTP_X_AMZ_DATE=20160722T094017Z
2016-07-22 09:40:17.579500 7f40547f8700 20 HTTP_X_AMZ_META_S3CMD_ATTRS=uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1469007939/atime:1469007939/md5:d8160ddb9f4681ec985e03429f842b88/ctime:1469023832
2016-07-22 09:40:17.579501 7f40547f8700 20 HTTP_X_AMZ_STORAGE_CLASS=STANDARD
2016-07-22 09:40:17.579501 7f40547f8700 20 QUERY_STRING=uploads
2016-07-22 09:40:17.579502 7f40547f8700 20 REMOTE_USER=
2016-07-22 09:40:17.579502 7f40547f8700 20 REQUEST_METHOD=POST
2016-07-22 09:40:17.579502 7f40547f8700 20 REQUEST_URI=/test-bucket-0/s3-test-file-1
2016-07-22 09:40:17.579503 7f40547f8700 20 SCRIPT_URI=/test-bucket-0/s3-test-file-1
2016-07-22 09:40:17.579503 7f40547f8700 20 SERVER_PORT=8080
2016-07-22 09:40:17.579505 7f40547f8700 1 ====== starting new request req=0x7f40547f2710 =====
2016-07-22 09:40:17.579527 7f40547f8700 2 req 5:0.000022::POST /test-bucket-0/s3-test-file-1::initializing for trans_id = tx000000000000000000005-005791ea01-8c23-default
2016-07-22 09:40:17.579530 7f40547f8700 10 host=10.1.68.29
2016-07-22 09:40:17.579533 7f40547f8700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2016-07-22 09:40:17.579542 7f40547f8700 10 meta>> HTTP_X_AMZ_CONTENT_SHA256
2016-07-22 09:40:17.579547 7f40547f8700 10 meta>> HTTP_X_AMZ_DATE
2016-07-22 09:40:17.579549 7f40547f8700 10 meta>> HTTP_X_AMZ_META_S3CMD_ATTRS
2016-07-22 09:40:17.579550 7f40547f8700 10 meta>> HTTP_X_AMZ_STORAGE_CLASS
2016-07-22 09:40:17.579552 7f40547f8700 10 x>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-07-22 09:40:17.579553 7f40547f8700 10 x>> x-amz-date:20160722T094017Z
2016-07-22 09:40:17.579553 7f40547f8700 10 x>> x-amz-meta-s3cmd-attrs:uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1469007939/atime:1469007939/md5:d8160ddb9f4681ec985e03429f842b88/ctime:1469023832
2016-07-22 09:40:17.579554 7f40547f8700 10 x>> x-amz-storage-class:STANDARD
2016-07-22 09:40:17.579571 7f40547f8700 20 get_handler handler=22RGWHandler_REST_Obj_S3
2016-07-22 09:40:17.579582 7f40547f8700 10 handler=22RGWHandler_REST_Obj_S3
2016-07-22 09:40:17.579583 7f40547f8700 2 req 5:0.000079:s3:POST /test-bucket-0/s3-test-file-1::getting op 4
2016-07-22 09:40:17.579586 7f40547f8700 10 op=28RGWInitMultipart_ObjStore_S3
2016-07-22 09:40:17.579587 7f40547f8700 2 req 5:0.000082:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:authorizing
2016-07-22 09:40:17.579602 7f40547f8700 10 v4 signedheaders format = content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
2016-07-22 09:40:17.579604 7f40547f8700 10 v4 signature format = dac5cb849ed2057d925a43f702b9e1f135618fd04d95beb943954df1d7c0df1c
2016-07-22 09:40:17.579609 7f40547f8700 10 v4 credential format = 7VM2JP5QFARP8UMUW2KH/20160722/US/s3/aws4_request
2016-07-22 09:40:17.579610 7f40547f8700 10 access key id = 7VM2JP5QFARP8UMUW2KH
2016-07-22 09:40:17.579611 7f40547f8700 10 credential scope = 20160722/US/s3/aws4_request
2016-07-22 09:40:17.579621 7f40547f8700 20 get_system_obj_state: rctx=0x7f40547f1070 obj=.users:7VM2JP5QFARP8UMUW2KH state=0x7f419400a5c8 s->prefetch_data=0
2016-07-22 09:40:17.579628 7f40547f8700 10 cache get: name=.users+7VM2JP5QFARP8UMUW2KH : type miss (requested=6, cached=3)
2016-07-22 09:40:17.581192 7f40547f8700 10 cache put: name=.users+7VM2JP5QFARP8UMUW2KH info.flags=6
2016-07-22 09:40:17.581210 7f40547f8700 10 moving .users+7VM2JP5QFARP8UMUW2KH to cache LRU end
2016-07-22 09:40:17.581218 7f40547f8700 20 get_system_obj_state: s->obj_tag was set empty
2016-07-22 09:40:17.581223 7f40547f8700 10 cache get: name=.users+7VM2JP5QFARP8UMUW2KH : hit (requested=1, cached=7)
2016-07-22 09:40:17.581246 7f40547f8700 20 get_system_obj_state: rctx=0x7f40547f0d30 obj=.users.uid:mnarusze state=0x7f419400fbd8 s->prefetch_data=0
2016-07-22 09:40:17.581253 7f40547f8700 10 cache get: name=.users.uid+mnarusze : hit (requested=22, cached=23)
2016-07-22 09:40:17.581256 7f40547f8700 20 get_system_obj_state: s->obj_tag was set empty
2016-07-22 09:40:17.581257 7f40547f8700 10 cache get: name=.users.uid+mnarusze : hit (requested=17, cached=23)
2016-07-22 09:40:17.581283 7f40547f8700 10 chain_cache_entry: cache_locator=.users.uid+mnarusze
2016-07-22 09:40:17.581329 7f40547f8700 10 canonical headers format = content-type:application/octet-stream
host:10.1.68.29:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160722T094017Z
x-amz-meta-s3cmd-attrs:uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1469007939/atime:1469007939/md5:d8160ddb9f4681ec985e03429f842b88/ctime:1469023832
x-amz-storage-class:STANDARD
2016-07-22 09:40:17.581377 7f40547f8700 10 payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-07-22 09:40:17.581392 7f40547f8700 10 canonical request = POST
/test-bucket-0/s3-test-file-1
uploads=
content-type:application/octet-stream
host:10.1.68.29:8080
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20160722T094017Z
x-amz-meta-s3cmd-attrs:uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1469007939/atime:1469007939/md5:d8160ddb9f4681ec985e03429f842b88/ctime:1469023832
x-amz-storage-class:STANDARD
content-type;host;x-amz-content-sha256;x-amz-date;x-amz-meta-s3cmd-attrs;x-amz-storage-class
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-07-22 09:40:17.581395 7f40547f8700 10 canonical request hash = 4564d15ed3560f4275d58e38a167a8a5d92ab705bd825a5a9210024d0df65cf1
2016-07-22 09:40:17.581398 7f40547f8700 10 string to sign = AWS4-HMAC-SHA256
20160722T094017Z
20160722/US/s3/aws4_request
4564d15ed3560f4275d58e38a167a8a5d92ab705bd825a5a9210024d0df65cf1
2016-07-22 09:40:17.581431 7f40547f8700 10 date_k = a17edb8f7e5dc6878bd8f9e71f89cf39b1a03781b39b4d6b2744c7adbb179310
2016-07-22 09:40:17.581447 7f40547f8700 10 region_k = 4734eb4a1723c1b849e055935169719211c9bfd4c50fd68f6ad9f4e28d7d3965
2016-07-22 09:40:17.581462 7f40547f8700 10 service_k = 53c0af6c67a6682c576a23ff64b9f3fc41310e02d9b322649d4d6818c785ce51
2016-07-22 09:40:17.581476 7f40547f8700 10 signing_k = 9b33b4ef7041591e3ad4ecfa445abbc51c9e667316f1c5b7ba06b97ee7243a31
2016-07-22 09:40:17.581491 7f40547f8700 10 signature_k = dac5cb849ed2057d925a43f702b9e1f135618fd04d95beb943954df1d7c0df1c
2016-07-22 09:40:17.581494 7f40547f8700 10 new signature = dac5cb849ed2057d925a43f702b9e1f135618fd04d95beb943954df1d7c0df1c
2016-07-22 09:40:17.581494 7f40547f8700 10 ----------------------------- Verifying signatures
2016-07-22 09:40:17.581495 7f40547f8700 10 Signature = dac5cb849ed2057d925a43f702b9e1f135618fd04d95beb943954df1d7c0df1c
2016-07-22 09:40:17.581496 7f40547f8700 10 New Signature = dac5cb849ed2057d925a43f702b9e1f135618fd04d95beb943954df1d7c0df1c
2016-07-22 09:40:17.581497 7f40547f8700 10 -----------------------------
2016-07-22 09:40:17.581498 7f40547f8700 10 v4 auth ok
2016-07-22 09:40:17.581502 7f40547f8700 2 req 5:0.001997:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:normalizing buckets and tenants
2016-07-22 09:40:17.581506 7f40547f8700 10 s->object=s3-test-file-1 s->bucket=test-bucket-0
2016-07-22 09:40:17.581509 7f40547f8700 2 req 5:0.002004:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:init permissions
2016-07-22 09:40:17.581519 7f40547f8700 20 get_system_obj_state: rctx=0x7f40547f1e50 obj=.rgw:test-bucket-0 state=0x7f41940113d8 s->prefetch_data=0
2016-07-22 09:40:17.581523 7f40547f8700 10 cache get: name=.rgw+test-bucket-0 : type miss (requested=22, cached=19)
2016-07-22 09:40:17.583211 7f40547f8700 10 cache put: name=.rgw+test-bucket-0 info.flags=22
2016-07-22 09:40:17.583234 7f40547f8700 10 moving .rgw+test-bucket-0 to cache LRU end
2016-07-22 09:40:17.583251 7f40547f8700 20 get_system_obj_state: s->obj_tag was set empty
2016-07-22 09:40:17.583253 7f40547f8700 20 Read xattr: user.rgw.idtag
2016-07-22 09:40:17.583253 7f40547f8700 20 Read xattr: user.rgw.manifest
2016-07-22 09:40:17.583257 7f40547f8700 10 cache get: name=.rgw+test-bucket-0 : hit (requested=17, cached=23)
2016-07-22 09:40:17.583298 7f40547f8700 20 rgw_get_bucket_info: bucket instance: test-bucket-0(@{i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[default.25311.1])
2016-07-22 09:40:17.583303 7f40547f8700 20 reading from .rgw:.bucket.meta.test-bucket-0:default.25311.1
2016-07-22 09:40:17.583309 7f40547f8700 20 get_system_obj_state: rctx=0x7f40547f1e50 obj=.rgw:.bucket.meta.test-bucket-0:default.25311.1 state=0x7f4194011d38 s->prefetch_data=0
2016-07-22 09:40:17.583313 7f40547f8700 10 cache get: name=.rgw+.bucket.meta.test-bucket-0:default.25311.1 : hit (requested=22, cached=23)
2016-07-22 09:40:17.583317 7f40547f8700 20 get_system_obj_state: s->obj_tag was set empty
2016-07-22 09:40:17.583318 7f40547f8700 20 Read xattr: user.rgw.acl
2016-07-22 09:40:17.583318 7f40547f8700 20 Read xattr: user.rgw.idtag
2016-07-22 09:40:17.583319 7f40547f8700 20 Read xattr: user.rgw.manifest
2016-07-22 09:40:17.583320 7f40547f8700 10 cache get: name=.rgw+.bucket.meta.test-bucket-0:default.25311.1 : hit (requested=17, cached=23)
2016-07-22 09:40:17.583332 7f40547f8700 10 chain_cache_entry: cache_locator=.rgw+test-bucket-0
2016-07-22 09:40:17.583333 7f40547f8700 10 chain_cache_entry: cache_locator=.rgw+.bucket.meta.test-bucket-0:default.25311.1
2016-07-22 09:40:17.583347 7f40547f8700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Owner><ID>mnarusze</ID><DisplayName>Maciej Naruszewicz</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:type="CanonicalUser"><ID>mnarusze</ID><DisplayName>Maciej Naruszewicz</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2016-07-22 09:40:17.583376 7f40547f8700 2 req 5:0.003871:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:recalculating target
2016-07-22 09:40:17.583380 7f40547f8700 2 req 5:0.003875:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:reading permissions
2016-07-22 09:40:17.583394 7f40547f8700 2 req 5:0.003889:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:init op
2016-07-22 09:40:17.583396 7f40547f8700 2 req 5:0.003891:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:verifying op mask
2016-07-22 09:40:17.583397 7f40547f8700 20 required_mask= 2 user.op_mask=7
2016-07-22 09:40:17.583398 7f40547f8700 2 req 5:0.003893:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:verifying op permissions
2016-07-22 09:40:17.583400 7f40547f8700 5 Searching permissions for uid=mnarusze mask=50
2016-07-22 09:40:17.583401 7f40547f8700 5 Found permission: 15
2016-07-22 09:40:17.583403 7f40547f8700 5 Searching permissions for group=1 mask=50
2016-07-22 09:40:17.583403 7f40547f8700 5 Permissions for group not found
2016-07-22 09:40:17.583404 7f40547f8700 5 Searching permissions for group=2 mask=50
2016-07-22 09:40:17.583407 7f40547f8700 5 Permissions for group not found
2016-07-22 09:40:17.583407 7f40547f8700 5 Getting permissions id=mnarusze owner=mnarusze perm=2
2016-07-22 09:40:17.583408 7f40547f8700 10 uid=mnarusze requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2
2016-07-22 09:40:17.583409 7f40547f8700 2 req 5:0.003905:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:verifying op params
2016-07-22 09:40:17.583410 7f40547f8700 2 req 5:0.003906:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:pre-executing
2016-07-22 09:40:17.583411 7f40547f8700 2 req 5:0.003907:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:executing
2016-07-22 09:40:17.583427 7f40547f8700 10 x>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-07-22 09:40:17.583432 7f40547f8700 10 x>> x-amz-date:20160722T094017Z
2016-07-22 09:40:17.583435 7f40547f8700 10 x>> x-amz-meta-s3cmd-attrs:uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1469007939/atime:1469007939/md5:d8160ddb9f4681ec985e03429f842b88/ctime:1469023832
2016-07-22 09:40:17.583439 7f40547f8700 10 x>> x-amz-storage-class:STANDARD
2016-07-22 09:40:17.583479 7f40547f8700 20 get_obj_state: rctx=0x7f40547f1e50 obj=test-bucket-0:_multipart_s3-test-file-1.2~ykp9njk9acgZMSXAgnwzyQUeNk7T1Y_.meta state=0x7f419401e2d8 s->prefetch_data=0
2016-07-22 09:40:17.584806 7f40547f8700 20 get_obj_state: rctx=0x7f40547f1e50 obj=test-bucket-0:_multipart_s3-test-file-1.2~ykp9njk9acgZMSXAgnwzyQUeNk7T1Y_.meta state=0x7f419401e2d8 s->prefetch_data=0
2016-07-22 09:40:17.584814 7f40547f8700 20 prepare_atomic_modification: state is not atomic. state=0x7f419401e2d8
2016-07-22 09:40:17.584891 7f40547f8700 20 reading from .rgw:.bucket.meta.test-bucket-0:default.25311.1
2016-07-22 09:40:17.584900 7f40547f8700 20 get_system_obj_state: rctx=0x7f40547f0cc0 obj=.rgw:.bucket.meta.test-bucket-0:default.25311.1 state=0x7f419402fdd8 s->prefetch_data=0
2016-07-22 09:40:17.584905 7f40547f8700 10 cache get: name=.rgw+.bucket.meta.test-bucket-0:default.25311.1 : hit (requested=22, cached=23)
2016-07-22 09:40:17.584909 7f40547f8700 20 get_system_obj_state: s->obj_tag was set empty
2016-07-22 09:40:17.584910 7f40547f8700 10 cache get: name=.rgw+.bucket.meta.test-bucket-0:default.25311.1 : hit (requested=17, cached=23)
2016-07-22 09:40:17.584922 7f40547f8700 20 bucket index object: .dir.default.25311.1
2016-07-22 09:40:17.589535 7f40547f8700 2 req 5:0.010030:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:completing
2016-07-22 09:40:17.589544 7f40547f8700 0 WARNING: set_req_state_err err_no=95 resorting to 500
2016-07-22 09:40:17.589660 7f40547f8700 2 req 5:0.010155:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:op status=-95
2016-07-22 09:40:17.589664 7f40547f8700 2 req 5:0.010159:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:http status=500
2016-07-22 09:40:17.589667 7f40547f8700 1 ====== req done req=0x7f40547f2710 op status=-95 http_status=500 ======
2016-07-22 09:40:17.589681 7f40547f8700 20 process_request() returned -95
2. RGW config
# radosgw-admin zone get
{
"id": "default",
"name": "default",
"domain_root": ".rgw",
"control_pool": ".rgw.control",
"gc_pool": ".rgw.gc",
"log_pool": ".log",
"intent_log_pool": ".intent-log",
"usage_log_pool": ".usage",
"user_keys_pool": ".users",
"user_email_pool": ".users.email",
"user_swift_pool": ".users.swift",
"user_uid_pool": ".users.uid",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": ".rgw.buckets.index",
"data_pool": ".rgw.buckets",
"data_extra_pool": ".rgw.buckets.extra",
"index_type": 0
}
}
],
"metadata_heap": "default.rgw.meta",
"realm_id": "37b9817e-8715-4e32-a394-ccb85653c691"
}
# radosgw zonegroup get
{
"id": "default",
"name": "default",
"api_name": "",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "default",
"zones": [
{
"id": "default",
"name": "default",
"endpoints": [],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "37b9817e-8715-4e32-a394-ccb85653c691"
}
# radosgw-admin period get
{
"id": "a720a7e9-cc9d-4c87-9c84-df06731da530",
"epoch": 1,
"predecessor_uuid": "697be6a7-072e-4888-8438-524c7ac2ed9f",
"sync_status": [
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
"",
""
],
"period_map": {
"id": "a720a7e9-cc9d-4c87-9c84-df06731da530",
"zonegroups": [
{
"id": "default",
"name": "default",
"api_name": "",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "default",
"zones": [
{
"id": "default",
"name": "default",
"endpoints": [],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": "37b9817e-8715-4e32-a394-ccb85653c691"
}
],
"short_zone_ids": [
{
"key": "default",
"val": 2610307010
}
]
},
"master_zonegroup": "default",
"master_zone": "default",
"period_config": {
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
},
"realm_id": "37b9817e-8715-4e32-a394-ccb85653c691",
"realm_name": "default_realm",
"realm_epoch": 2
}
-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Naruszewicz, Maciej
Sent: Thursday, July 21, 2016 3:12 PM
To: nick@xxxxxxx
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: Unknown error (95->500) when creating buckets or putting files to RGW after upgrade from Infernalis to Jewel
Hi Nick,
Thanks for your suggestion, I've tried the script on an isolated testing cluster. Unfortunately, the script did not help us a lot, it only made creating buckets possible.
The logs I provided earlier actually make some sense because they were collected using RGW in Jewel and Ceph in Infernalis, so it only makes sense that some of the operations requested by RGW are not supported. However, if both Ceph and RGW are upgraded to Jewel I still get the following errors when creating a bucket and trying to upload a file:
1) Trying to create a bucket:
2016-07-21 12:10:39.389397 7f67d57fa700 0 sending create_bucket request to master zonegroup
2016-07-21 12:10:39.389399 7f67d57fa700 0 ERROR: endpoints not configured for upstream zone
2016-07-21 12:10:39.389403 7f67d57fa700 2 req 2:0.003300:s3:PUT /test-bucket-2/:create_bucket:completing
2016-07-21 12:10:39.389406 7f67d57fa700 0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-07-21 12:10:39.389486 7f67d57fa700 2 req 2:0.003383:s3:PUT /test-bucket-2/:create_bucket:op status=-5
2016-07-21 12:10:39.389491 7f67d57fa700 2 req 2:0.003388:s3:PUT /test-bucket-2/:create_bucket:http status=500
I looked at the zonegroup (simplest setup with one zone and one zonegroup which was probably created during upgrade) and indeed, it does not contain any endpoints:
# radosgw-admin zonegroup get --zonegroup-id {
"id": "default",
"name": "default",
"api_name": "",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "",
"zones": [
{
"id": "default",
"name": "default",
"endpoints": [],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": ""
}
In one cluster, we have one RGW instance, in the second we have three. I wonder whether setting up the zonegroup is needed at all...? I'll try to modify the zonegroup settings and see if it might help with anything.
2) Trying to upload a file:
2016-07-21 12:40:55.851011 7f67737fe700 2 req 5:0.003166:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:verifying op params
2016-07-21 12:40:55.851012 7f67737fe700 2 req 5:0.003167:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:pre-executing
2016-07-21 12:40:55.851014 7f67737fe700 2 req 5:0.003168:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:executing
2016-07-21 12:40:55.851031 7f67737fe700 10 x>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-07-21 12:40:55.851037 7f67737fe700 10 x>> x-amz-date:20160721T124055Z
2016-07-21 12:40:55.851041 7f67737fe700 10 x>> x-amz-meta-s3cmd-attrs:uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1469007939/atime:1469007939/md5:d8160ddb9f4681ec985e03429f842b88/ctime:1469023832
2016-07-21 12:40:55.851048 7f67737fe700 10 x>> x-amz-storage-class:STANDARD
2016-07-21 12:40:55.851122 7f67737fe700 20 get_obj_state: rctx=0x7f67737f7e50 obj=test-bucket-0:_multipart_s3-test-file-1.2~orci2-8OGWvX6FkSCsreSitUc-DEQ7Z.meta state=0x7f6888023358 s->prefetch_data=0
2016-07-21 12:40:55.852738 7f67737fe700 20 get_obj_state: rctx=0x7f67737f7e50 obj=test-bucket-0:_multipart_s3-test-file-1.2~orci2-8OGWvX6FkSCsreSitUc-DEQ7Z.meta state=0x7f6888023358 s->prefetch_data=0
2016-07-21 12:40:55.852746 7f67737fe700 20 prepare_atomic_modification: state is not atomic. state=0x7f6888023358
2016-07-21 12:40:55.852841 7f67737fe700 20 reading from .rgw:.bucket.meta.test-bucket-0:default.25873.1
2016-07-21 12:40:55.852860 7f67737fe700 20 get_system_obj_state: rctx=0x7f67737f6cc0 obj=.rgw:.bucket.meta.test-bucket-0:default.25873.1 state=0x7f6888034e48 s->prefetch_data=0
2016-07-21 12:40:55.852863 7f67737fe700 10 cache get: name=.rgw+.bucket.meta.test-bucket-0:default.25873.1 : hit (requested=22, cached=23)
2016-07-21 12:40:55.852884 7f67737fe700 20 get_system_obj_state: s->obj_tag was set empty
2016-07-21 12:40:55.852886 7f67737fe700 10 cache get: name=.rgw+.bucket.meta.test-bucket-0:default.25873.1 : hit (requested=17, cached=23)
2016-07-21 12:40:55.852908 7f67737fe700 20 bucket index object: .dir.default.25873.1
2016-07-21 12:40:55.857254 7f67737fe700 2 req 5:0.009408:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:completing
2016-07-21 12:40:55.857262 7f67737fe700 0 WARNING: set_req_state_err err_no=95 resorting to 500
2016-07-21 12:40:55.857413 7f67737fe700 2 req 5:0.009567:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:op status=-95
I cannot see any error here and I'm not even sure where to look.
Thanks,
MN
-----Original Message-----
From: nick@xxxxxxx [mailto:nick@xxxxxxx]
Sent: Tuesday, July 19, 2016 8:48 AM
To: ceph-users@xxxxxxxxxxxxxx
Cc: Naruszewicz, Maciej <maciej.naruszewicz@xxxxxxxxx>
Subject: Re: Unknown error (95->500) when creating buckets or putting files to RGW after upgrade from Infernalis to Jewel
Hi Maciej,
we also had problems when upgrading our infernalis RGW cluster to jewel. In the end I managed to upgrade with the help of a script (from Yehuda). Search for the thread " radosgw hammer -> jewel upgrade (default zone & region config)" on the mailing list. There you can find more information about this....altough I do not know if the issue you experience is the same like we had.
Cheers
Nick
On Monday, July 18, 2016 02:13:15 PM Naruszewicz, Maciej wrote:
> Hi,
>
> We recently upgraded our Ceph Cluster to Jewel including RGW.
> Everything seems to be in order except for RGW which doesn't let us
> create buckets or add new files.
>
> # s3cmd --version
> s3cmd version 1.6.1
>
> # s3cmd mb s3://test
> WARNING: Retrying failed request: /
> WARNING: 500 (UnknownError)
> WARNING: Waiting 3 sec...
>
> # s3cmd put test s3://nginx-proxy/test
> upload: 'test' -> 's3://nginx-proxy/test' [1 of 1]
> 7 of 7 100% in 0s 224.55 B/s done
> WARNING: Upload failed: /test (500 (UnknownError))
> WARNING: Waiting 3 sec...
>
> I am able to read and even remove files, I just can't add anything new.
>
> I enabled RGW logs to check what went wrong and got the following
> trying to upload a file:
>
> 2016-07-18 12:09:22.301512 7fdcc57fa700 1 -- 10.251.97.13:0/563287553
> -->
> 10.251.97.1:6800/4104 -- osd_op(client.199724.0:927 11.1f0a02a1
> default.194977.1_test [getxattrs,stat] snapc 0=[]
> ack+read+known_if_redirected e479) v7 -- ?+0 0x7fdd64020220 con
> 0x7fde100487c0 2016-07-18 12:09:22.303323 7fddef3f3700 1 --
> 10.251.97.13:0/563287553 <== osd.27 10.251.97.1:6800/4104 10 ====
> osd_op_reply(927 default.194977.1_test [getxattrs,stat] v0'0 uv0 ack =
> -2
> ((2) No such file or directory)) v6 ==== 230+0+0 (25 91304629 0 0)
> 0x7fda70000d00 con 0x7fde100487c0
> 2016-07-18 12:09:22.303629 7fdcc57fa700 1 -- 10.251.97.13:0/563287553
> -->
> 10.251.97.1:6818/6493 -- osd_op(client.199724.0:928 10.cecde97a
> .dir.default.194977.1 [call rgw.bucket_prepare_op] snapc 0=[]
> ondisk+write+known_if_redirected e479 ) v7 -- ?+0 0x7fdd6402af60 con
> 0x7fde10032110
> 2016-07-18 12:09:22.308437 7fddee9e9700 1 -- 10.251.97.13:0/563287553
> <==
> osd.6 10.251.97.1:6818/6493 13 ==== osd_op_reply(928
> .dir.default.194977.1 [call] v479'126 uv126 ondisk = 0) v6 ====
> 188+0+0 (1238951509 0 0)
> 0x7fda6c000cc0 con 0x 7fde10032110
> 2016-07-18 12:09:22.308528 7fdcc57fa700 1 -- 10.251.97.13:0/563287553
> -->
> 10.251.97.1:6800/4104 -- osd_op(client.199724.0:929 11.1f0a02a1
> default.194977.1_test [create 0~0 [excl],setxattr user.rgw.idtag
> (17),writefull 0~7,setxattr user.r gw.manifest (413),setxattr
> user.rgw.acl (127),setxattr user.rgw.content_type (11),setxattr
> user.rgw.etag (33),setxattr user.rgw.x-amz-content-sha256
> (65),setxattr user.rgw.x-amz-date (17),setxattr
> user.rgw.x-amz-meta-s3cmd-attrs (133),set xattr
> user.rgw.x-amz-storage-class (9),call rgw.obj_store_pg_ver,setxattr
> user.rgw.source_zone (4)] snapc 0=[] ondisk+write+known_if_redirected
> e479)
> v7 -- ?+0 0x7fdd64024ae0 con 0x7fde100487c0 2016-07-18 12:09:22.309371
> 7fddef3f3700 1 -- 10.251.97.13:0/563287553 <== osd.27
> 10.251.97.1:6800/4104 11 ==== osd_op_reply(929 default.194977.1_test
> [create 0~0 [excl],setxattr (17),writefull 0~7,setxattr (413),setxattr
> (127),setxattr ( 11),setxattr (33),setxattr (65),setxattr
> (17),setxattr (133),setxattr (9),call,setxattr (4)] v0'0 uv0 ondisk =
> -95 ((95) Operation not supported)) v6 ==== 692+0+0 (982388421 0 0)
> 0x7fda70000d00 con
> 0x7fde100487c0 2016-07-18 12:09:22.309471 7fdcc57fa700 1 --
> 10.251.97.13:0/563287553 --> 10.251.97.1:6818/6493 --
> osd_op(client.199724.0:930 10.cecde97a .dir.default.194977.1 [call
> rgw.bucket_complete_op] snapc 0=[]
> ack+ondisk+write+known_if_redirected
> e479) v7 -- ?+0 0x7fdd64024ae0 con 0x7fde10032110
> 2016-07-18 12:09:22.309504 7fdcc57fa700 2 req 3:0.047834:s3:PUT
> /nginx-proxy/test:put_obj:completing 2016-07-18 12:09:22.309509
> 7fdcc57fa700 0 WARNING: set_req_state_err err_no=95 resorting to 500
> 2016-07-18 12:09:22.309580 7fdcc57fa700 2 req 3:0.047910:s3:PUT
> /nginx-proxy/test:put_obj:op status=-95 2016-07-18 12:09:22.309585
> 7fdcc57fa700 2 req 3:0.047915:s3:PUT /nginx-proxy/test:put_obj:http
> status=500
>
> I tried to look for any information around this error but I only found
> one similar unanswered thread.
>
> The issue disappears if I use RGW Infernalis instead, the create does
> not fail and everything goes smoothly. It is also not dependent on the
> daemons version, the situation is the same in our second
> Infernalis-based cluster where only RGW was updated for tests.
>
> Could anyone recommend what is wrong here?
>
> Thanks,
> MN
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
