Hi, After upgrading our OpenStack cluster to Kilo and upgrading Ceph from Giant to Hammer, RadosGW stopped working. All other services using Ceph work fine. RadosGW is configured to use Keystone for authentication. # swift list Account GET failed: http://object.api.openstack.cyso.net/swift/v1?format=json 400 Bad Request {"Code":"InvalidArgument"} Funny thing is that when I do a Curl on that URL and remove the format=json part it does work. root at controller1.eq1:~# curl -i 'http://object.api.openstack.cyso.net/swift/v1?format=json' -X GET -H "X-Auth-Token: XXXXX" HTTP/1.1 400 Bad Request Date: Tue, 26 May 2015 15:49:17 GMT Server: Apache/2.4.7 (Ubuntu) Accept-Ranges: bytes Content-Length: 26 Connection: close Content-Type: application/json; charset=utf-8 {"Code":"InvalidArgument"} root at controller1.eq1:~# curl -i 'http://object.api.openstack.cyso.net/swift/v1' -X GET -H "X-Auth-Token: XXXXXX" HTTP/1.1 200 OK Date: Tue, 26 May 2015 15:49:36 GMT Server: Apache/2.4.7 (Ubuntu) Vary: Accept-Encoding Connection: close Transfer-Encoding: chunked Content-Type: text/plain; charset=utf-8 testbucket In the RadosGW log I see: 2015-05-26 17:43:37.352569 7f0fce0ff840 0 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff), process radosgw, pid 4259 2015-05-26 17:43:37.435921 7f0f8a4f2700 0 ERROR: can't get key: ret=-2 2015-05-26 17:43:37.435932 7f0f8a4f2700 0 ERROR: sync_all_users() returned ret=-2 2015-05-26 17:43:37.436179 7f0fce0ff840 0 framework: fastcgi 2015-05-26 17:43:37.436191 7f0fce0ff840 0 framework: civetweb 2015-05-26 17:43:37.436198 7f0fce0ff840 0 framework conf key: port, val: 7480 2015-05-26 17:43:37.436208 7f0fce0ff840 0 starting handler: civetweb 2015-05-26 17:43:37.453013 7f0fce0ff840 0 starting handler: fastcgi 2015-05-26 17:43:41.347403 7f0d487a0700 1 ====== starting new request req=0x7f0d7c011590 ===== 2015-05-26 17:43:41.498851 7f0d487a0700 0 validated token: openstack:admin expires: 1432658621 2015-05-26 17:43:41.509877 7f0d487a0700 0 ERROR: could not get stats for buckets 2015-05-26 17:43:41.509943 7f0d487a0700 1 ====== req done req=0x7f0d7c011590 http_status=400 ====== 2015-05-26 17:43:44.140730 7f0d897fa700 1 ====== starting new request req=0x7f0d7c0162e0 ===== 2015-05-26 17:43:44.285039 7f0d897fa700 0 validated token: openstack:admin expires: 1432658624 2015-05-26 17:43:44.290175 7f0d897fa700 0 ERROR: could not get stats for buckets 2015-05-26 17:43:44.290222 7f0d897fa700 1 ====== req done req=0x7f0d7c0162e0 http_status=400 ====== 2015-05-26 17:43:46.972310 7f0d77fff700 1 ====== starting new request req=0x7f0d7c016270 ===== 2015-05-26 17:43:47.121784 7f0d77fff700 0 validated token: openstack:admin expires: 1432658626 2015-05-26 17:43:47.125191 7f0d77fff700 0 ERROR: could not get stats for buckets 2015-05-26 17:43:47.125241 7f0d77fff700 1 ====== req done req=0x7f0d7c016270 http_status=400 ====== Permissions haven't changed from the previous version: client.radosgw.controller1 key: XXXXXXXX caps: [mon] allow rwx caps: [osd] allow rwx My ceph.conf: [client.radosgw.controller1] rgw keystone token cache size = 100 rgw keystone revocation interval = 3600 log file = /var/log/radosgw/client.radosgw.controller1.log rgw keystone admin token = XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX rgw keystone accepted roles = s3, swift, admin, _member_ rgw s3 auth use keystone = true host = controller1 keyring = /etc/ceph/ceph.client.radosgw.controller1.keyring rgw keystone url = https://identity-admin.api.openstack.cyso.net:35357 nss db path = /var/lib/ceph/radosgw/nss rgw print continue = false rgw socket path = /var/run/ceph/ceph.radosgw.controller1.fastcgi.sock rgw usage max user shards = 1 rgw usage log tick interval = 60 rgw usage log flush threshold = 1024 rgw usage max shards = 32 rgw enable usage log = true Apache config: <VirtualHost 10.1.0.11:80> ServerName object.api.openstack.cyso.net DocumentRoot "/var/www" <Directory "/var/www"> Options +ExecCGI AllowOverride All Require all granted SetHandler fastcgi-script </Directory> ErrorLog "/var/log/apache2/object_endpoint_error.log" ServerSignature Off CustomLog "/var/log/apache2/object_endpoint_access.log" combined RewriteEngine On RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] ServerAlias *.object.api.openstack.cyso.net FastCgiExternalServer /var/www/s3gw.fcgi -socket /var/run/ceph/ceph.radosgw.controller1.fastcgi.sock AllowEncodedSlashes on </VirtualHost> Any thoughts on what could be wrong? Kind regards, Arnoud de Jonge
 |