Hi Nick,
Thanks for your suggestion, I've tried the script on an isolated testing cluster. Unfortunately, the script did not help us a lot, it only made creating buckets possible.
The logs I provided earlier actually make some sense because they were collected using RGW in Jewel and Ceph in Infernalis, so it only makes sense that some of the operations requested by RGW are not supported. However, if both Ceph and RGW are upgraded to Jewel I still get the following errors when creating a bucket and trying to upload a file:
1) Trying to create a bucket:
2016-07-21 12:10:39.389397 7f67d57fa700 0 sending create_bucket request to master zonegroup
2016-07-21 12:10:39.389399 7f67d57fa700 0 ERROR: endpoints not configured for upstream zone
2016-07-21 12:10:39.389403 7f67d57fa700 2 req 2:0.003300:s3:PUT /test-bucket-2/:create_bucket:completing
2016-07-21 12:10:39.389406 7f67d57fa700 0 WARNING: set_req_state_err err_no=5 resorting to 500
2016-07-21 12:10:39.389486 7f67d57fa700 2 req 2:0.003383:s3:PUT /test-bucket-2/:create_bucket:op status=-5
2016-07-21 12:10:39.389491 7f67d57fa700 2 req 2:0.003388:s3:PUT /test-bucket-2/:create_bucket:http status=500
I looked at the zonegroup (simplest setup with one zone and one zonegroup which was probably created during upgrade) and indeed, it does not contain any endpoints:
# radosgw-admin zonegroup get --zonegroup-id
{
"id": "default",
"name": "default",
"api_name": "",
"is_master": "true",
"endpoints": [],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "",
"zones": [
{
"id": "default",
"name": "default",
"endpoints": [],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 0,
"read_only": "false"
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": []
}
],
"default_placement": "default-placement",
"realm_id": ""
}
In one cluster, we have one RGW instance, in the second we have three. I wonder whether setting up the zonegroup is needed at all...? I'll try to modify the zonegroup settings and see if it might help with anything.
2) Trying to upload a file:
2016-07-21 12:40:55.851011 7f67737fe700 2 req 5:0.003166:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:verifying op params
2016-07-21 12:40:55.851012 7f67737fe700 2 req 5:0.003167:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:pre-executing
2016-07-21 12:40:55.851014 7f67737fe700 2 req 5:0.003168:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:executing
2016-07-21 12:40:55.851031 7f67737fe700 10 x>> x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2016-07-21 12:40:55.851037 7f67737fe700 10 x>> x-amz-date:20160721T124055Z
2016-07-21 12:40:55.851041 7f67737fe700 10 x>> x-amz-meta-s3cmd-attrs:uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1469007939/atime:1469007939/md5:d8160ddb9f4681ec985e03429f842b88/ctime:1469023832
2016-07-21 12:40:55.851048 7f67737fe700 10 x>> x-amz-storage-class:STANDARD
2016-07-21 12:40:55.851122 7f67737fe700 20 get_obj_state: rctx=0x7f67737f7e50 obj=test-bucket-0:_multipart_s3-test-file-1.2~orci2-8OGWvX6FkSCsreSitUc-DEQ7Z.meta state=0x7f6888023358 s->prefetch_data=0
2016-07-21 12:40:55.852738 7f67737fe700 20 get_obj_state: rctx=0x7f67737f7e50 obj=test-bucket-0:_multipart_s3-test-file-1.2~orci2-8OGWvX6FkSCsreSitUc-DEQ7Z.meta state=0x7f6888023358 s->prefetch_data=0
2016-07-21 12:40:55.852746 7f67737fe700 20 prepare_atomic_modification: state is not atomic. state=0x7f6888023358
2016-07-21 12:40:55.852841 7f67737fe700 20 reading from .rgw:.bucket.meta.test-bucket-0:default.25873.1
2016-07-21 12:40:55.852860 7f67737fe700 20 get_system_obj_state: rctx=0x7f67737f6cc0 obj=.rgw:.bucket.meta.test-bucket-0:default.25873.1 state=0x7f6888034e48 s->prefetch_data=0
2016-07-21 12:40:55.852863 7f67737fe700 10 cache get: name=.rgw+.bucket.meta.test-bucket-0:default.25873.1 : hit (requested=22, cached=23)
2016-07-21 12:40:55.852884 7f67737fe700 20 get_system_obj_state: s->obj_tag was set empty
2016-07-21 12:40:55.852886 7f67737fe700 10 cache get: name=.rgw+.bucket.meta.test-bucket-0:default.25873.1 : hit (requested=17, cached=23)
2016-07-21 12:40:55.852908 7f67737fe700 20 bucket index object: .dir.default.25873.1
2016-07-21 12:40:55.857254 7f67737fe700 2 req 5:0.009408:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:completing
2016-07-21 12:40:55.857262 7f67737fe700 0 WARNING: set_req_state_err err_no=95 resorting to 500
2016-07-21 12:40:55.857413 7f67737fe700 2 req 5:0.009567:s3:POST /test-bucket-0/s3-test-file-1:init_multipart:op status=-95
I cannot see any error here and I'm not even sure where to look.
Thanks,
MN
-----Original Message-----
From: nick@xxxxxxx [mailto:nick@xxxxxxx]
Sent: Tuesday, July 19, 2016 8:48 AM
To: ceph-users@xxxxxxxxxxxxxx
Cc: Naruszewicz, Maciej <maciej.naruszewicz@xxxxxxxxx>
Subject: Re: Unknown error (95->500) when creating buckets or putting files to RGW after upgrade from Infernalis to Jewel
Hi Maciej,
we also had problems when upgrading our infernalis RGW cluster to jewel. In the end I managed to upgrade with the help of a script (from Yehuda). Search for the thread " radosgw hammer -> jewel upgrade (default zone & region config)" on the mailing list. There you can find more information about this....altough I do not know if the issue you experience is the same like we had.
Cheers
Nick
On Monday, July 18, 2016 02:13:15 PM Naruszewicz, Maciej wrote:
> Hi,
>
> We recently upgraded our Ceph Cluster to Jewel including RGW.
> Everything seems to be in order except for RGW which doesn't let us
> create buckets or add new files.
>
> # s3cmd --version
> s3cmd version 1.6.1
>
> # s3cmd mb s3://test
> WARNING: Retrying failed request: /
> WARNING: 500 (UnknownError)
> WARNING: Waiting 3 sec...
>
> # s3cmd put test s3://nginx-proxy/test
> upload: 'test' -> 's3://nginx-proxy/test' [1 of 1]
> 7 of 7 100% in 0s 224.55 B/s done
> WARNING: Upload failed: /test (500 (UnknownError))
> WARNING: Waiting 3 sec...
>
> I am able to read and even remove files, I just can't add anything new.
>
> I enabled RGW logs to check what went wrong and got the following
> trying to upload a file:
>
> 2016-07-18 12:09:22.301512 7fdcc57fa700 1 -- 10.251.97.13:0/563287553
> -->
> 10.251.97.1:6800/4104 -- osd_op(client.199724.0:927 11.1f0a02a1
> default.194977.1_test [getxattrs,stat] snapc 0=[]
> ack+read+known_if_redirected e479) v7 -- ?+0 0x7fdd64020220 con
> 0x7fde100487c0 2016-07-18 12:09:22.303323 7fddef3f3700 1 --
> 10.251.97.13:0/563287553 <== osd.27 10.251.97.1:6800/4104 10 ====
> osd_op_reply(927 default.194977.1_test [getxattrs,stat] v0'0 uv0 ack =
> -2
> ((2) No such file or directory)) v6 ==== 230+0+0 (25 91304629 0 0)
> 0x7fda70000d00 con 0x7fde100487c0
> 2016-07-18 12:09:22.303629 7fdcc57fa700 1 -- 10.251.97.13:0/563287553
> -->
> 10.251.97.1:6818/6493 -- osd_op(client.199724.0:928 10.cecde97a
> .dir.default.194977.1 [call rgw.bucket_prepare_op] snapc 0=[]
> ondisk+write+known_if_redirected e479 ) v7 -- ?+0 0x7fdd6402af60 con
> 0x7fde10032110
> 2016-07-18 12:09:22.308437 7fddee9e9700 1 -- 10.251.97.13:0/563287553
> <==
> osd.6 10.251.97.1:6818/6493 13 ==== osd_op_reply(928
> .dir.default.194977.1 [call] v479'126 uv126 ondisk = 0) v6 ====
> 188+0+0 (1238951509 0 0)
> 0x7fda6c000cc0 con 0x 7fde10032110
> 2016-07-18 12:09:22.308528 7fdcc57fa700 1 -- 10.251.97.13:0/563287553
> -->
> 10.251.97.1:6800/4104 -- osd_op(client.199724.0:929 11.1f0a02a1
> default.194977.1_test [create 0~0 [excl],setxattr user.rgw.idtag
> (17),writefull 0~7,setxattr user.r gw.manifest (413),setxattr
> user.rgw.acl (127),setxattr user.rgw.content_type (11),setxattr
> user.rgw.etag (33),setxattr user.rgw.x-amz-content-sha256
> (65),setxattr user.rgw.x-amz-date (17),setxattr
> user.rgw.x-amz-meta-s3cmd-attrs (133),set xattr
> user.rgw.x-amz-storage-class (9),call rgw.obj_store_pg_ver,setxattr
> user.rgw.source_zone (4)] snapc 0=[] ondisk+write+known_if_redirected
> e479)
> v7 -- ?+0 0x7fdd64024ae0 con 0x7fde100487c0 2016-07-18 12:09:22.309371
> 7fddef3f3700 1 -- 10.251.97.13:0/563287553 <== osd.27
> 10.251.97.1:6800/4104 11 ==== osd_op_reply(929 default.194977.1_test
> [create 0~0 [excl],setxattr (17),writefull 0~7,setxattr (413),setxattr
> (127),setxattr ( 11),setxattr (33),setxattr (65),setxattr
> (17),setxattr (133),setxattr (9),call,setxattr (4)] v0'0 uv0 ondisk =
> -95 ((95) Operation not supported)) v6 ==== 692+0+0 (982388421 0 0)
> 0x7fda70000d00 con
> 0x7fde100487c0 2016-07-18 12:09:22.309471 7fdcc57fa700 1 --
> 10.251.97.13:0/563287553 --> 10.251.97.1:6818/6493 --
> osd_op(client.199724.0:930 10.cecde97a .dir.default.194977.1 [call
> rgw.bucket_complete_op] snapc 0=[]
> ack+ondisk+write+known_if_redirected
> e479) v7 -- ?+0 0x7fdd64024ae0 con 0x7fde10032110
> 2016-07-18 12:09:22.309504 7fdcc57fa700 2 req 3:0.047834:s3:PUT
> /nginx-proxy/test:put_obj:completing 2016-07-18 12:09:22.309509
> 7fdcc57fa700 0 WARNING: set_req_state_err err_no=95 resorting to 500
> 2016-07-18 12:09:22.309580 7fdcc57fa700 2 req 3:0.047910:s3:PUT
> /nginx-proxy/test:put_obj:op status=-95 2016-07-18 12:09:22.309585
> 7fdcc57fa700 2 req 3:0.047915:s3:PUT /nginx-proxy/test:put_obj:http
> status=500
>
> I tried to look for any information around this error but I only found
> one similar unanswered thread.
>
> The issue disappears if I use RGW Infernalis instead, the create does
> not fail and everything goes smoothly. It is also not dependent on the
> daemons version, the situation is the same in our second
> Infernalis-based cluster where only RGW was updated for tests.
>
> Could anyone recommend what is wrong here?
>
> Thanks,
> MN
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
