Radosgw only gives 400s

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

I'm trying to figure out what's wrong with my radosgw setup.  I am
getting HTTP 400 responses when running some basic stat/upload tests,
which work fine when using openstack (folsom) swift.  Additionally, I
get HTTP 400 responses when trying to use the openstack (folsom)
dashboard to list or create containers.

I'm using the ceph-modified mod_fastcgi, and apache 2.2.23 with the worker mpm.

Here is a snip of the log which starts after I start apache and radosgw:

Mar  4 16:32:48 node-172-17-0-13 fcgi-pm[28606]: [notice] FastCGI:
process manager initialized (pid 28606)
Mar  4 16:32:48 node-172-17-0-13 apache2[28605]: [notice]
Apache/2.2.23 (Unix) mod_fastcgi/mod_fastcgi-SNAP-0910052141
configured -- resuming normal operations
Mar  4 16:32:48 node-172-17-0-13 apache2[28605]: [info] Server built:
Mar  2 2013 21:17:00
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.788234 7fd895016780  0 ceph version 0.56.3
(6eb7e15a4783b122e9b0c85ea9ba064145958aa5), process radosgw, pid 28746
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.788382 7fd895016780  0 ext_mime_map_init(): failed to open
file=/etc/mime.types ret=-2
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.791035 7fd87f7fe700  2 garbage collection: start
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.811325 7fd87ebf9700  2 keystone revoke thread: start
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.811377 7fd87ebf9700 20 sending request to
http://172.17.2.3:35357/v2.0/tokens/revoked
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.812816 7fd895016780 10 allocated request req=0x7fd8968df5f0
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.824017 7fd87ebf9700 10 request returned {"signed":
"-----BEGIN CMS-----\nMIIBQwYJKoZIhvcNAQcCoIIBNDCCATACAQExCTAHBgUrDgMCGjAeBgkqhkiG9w0B\nBwGgEQQPeyJyZXZva2VkIjogW119MYH/MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMx\nDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQx\nGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0B\nAQEFAASBgC7gYTOTf9SQEZKrk82UJVUUnPHyecpEI2kHb3IZ3iTmnm/IVHMAlNs7\n9Kmz6S9z0jFDWMqHxlnLAVuqzjGJMZFChuDTe4f/lRnHqd3I/ZlckVmAUw0ECAX5\nDmsQ+uziQYqHOiUGxeV0rwOu+x+khrxlbPv9x+2GcnIuEkG11ytr\n-----END
CMS-----\n"}
Mar  4 16:32:53 node-172-17-0-13 keystone: wsgi DEBUG 172.17.2.3 - -
[04/Mar/2013 16:32:53] "GET /v2.0/tokens/revoked HTTP/1.1" 200 633
0.011981
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: add_child:  <- signed
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.824115 7fd87ebf9700 10 signed=-----BEGIN CMS-----
Mar  4 16:32:53 node-172-17-0-13 radosgw.log:
MIIBQwYJKoZIhvcNAQcCoIIBNDCCATACAQExCTAHBgUrDgMCGjAeBgkqhkiG9w0B
Mar  4 16:32:53 node-172-17-0-13 radosgw.log:
BwGgEQQPeyJyZXZva2VkIjogW119MYH/MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMx
Mar  4 16:32:53 node-172-17-0-13 radosgw.log:
DjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQx
Mar  4 16:32:53 node-172-17-0-13 radosgw.log:
GDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0B
Mar  4 16:32:53 node-172-17-0-13 radosgw.log:
AQEFAASBgC7gYTOTf9SQEZKrk82UJVUUnPHyecpEI2kHb3IZ3iTmnm/IVHMAlNs7
Mar  4 16:32:53 node-172-17-0-13 radosgw.log:
9Kmz6S9z0jFDWMqHxlnLAVuqzjGJMZFChuDTe4f/lRnHqd3I/ZlckVmAUw0ECAX5
Mar  4 16:32:53 node-172-17-0-13 radosgw.log:
DmsQ+uziQYqHOiUGxeV0rwOu+x+khrxlbPv9x+2GcnIuEkG11ytr
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: -----END CMS-----
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.824124 7fd87ebf9700 10
content=MIIBQwYJKoZIhvcNAQcCoIIBNDCCATACAQExCTAHBgUrDgMCGjAeBgkqhkiG9w0BBwGgEQQPeyJyZXZva2VkIjogW119MYH/MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgC7gYTOTf9SQEZKrk82UJVUUnPHyecpEI2kHb3IZ3iTmnm/IVHMAlNs79Kmz6S9z0jFDWMqHxlnLAVuqzjGJMZFChuDTe4f/lRnHqd3I/ZlckVmAUw0ECAX5DmsQ+uziQYqHOiUGxeV0rwOu+x+khrxlbPv9x+2GcnIuEkG11ytr
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.824132 7fd87ebf9700  0 ceph_decode_cms returned -95
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.824138 7fd87ebf9700  0 ERROR: keystone revocation processing
returned error r=-95
Mar  4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04
16:32:53.906447 7fd87f7fe700  2 garbage collection: stop
Mar  4 16:33:46 node-172-17-0-13 keystone: wsgi DEBUG 10.1.10.12 - -
[04/Mar/2013 16:33:46] "POST /v2.0/tokens HTTP/1.1" 200 2711 0.084738
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891625 7fd895016780 20 enqueued request req=0x7fd8968df5f0
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891641 7fd895016780 20 RGWWQ:
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891643 7fd895016780 20 req: 0x7fd8968df5f0
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891646 7fd895016780 10 allocated request req=0x7fd8968e4a50
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891700 7fd8237be700 20 dequeued request req=0x7fd8968df5f0
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891709 7fd8237be700 20 RGWWQ: empty
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891716 7fd8237be700  1 ====== starting new request
req=0x7fd8968df5f0 =====
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891751 7fd8237be700  2 req 1:0.000035::::initializing
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891786 7fd8237be700 10
s->object=AUTH_2c5cfd34f2af41919847c0f4e0a28e04 s->bucket=v1
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891812 7fd8237be700  2 req 1:0.000096::HEAD
/v1/AUTH_2c5cfd34f2af41919847c0f4e0a28e04::http status=400
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.891877 7fd8237be700  1 ====== req done req=0x7fd8968df5f0
http_status=400 ======
Mar  4 16:33:46 node-172-17-0-13 keystone: wsgi DEBUG 10.1.10.12 - -
[04/Mar/2013 16:33:46] "POST /v2.0/tokens HTTP/1.1" 200 2711 0.078482
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978717 7fd895016780 20 enqueued request req=0x7fd8968e4a50
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978726 7fd895016780 20 RGWWQ:
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978727 7fd895016780 20 req: 0x7fd8968e4a50
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978730 7fd895016780 10 allocated request req=0x7fd8968e4e80
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978763 7fd8387e8700 20 dequeued request req=0x7fd8968e4a50
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978769 7fd8387e8700 20 RGWWQ: empty
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978773 7fd8387e8700  1 ====== starting new request
req=0x7fd8968e4a50 =====
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978800 7fd8387e8700  2 req 2:0.000028::::initializing
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978826 7fd8387e8700 10
s->object=AUTH_2c5cfd34f2af41919847c0f4e0a28e04/container-33771340493
s->bucket=v1
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978843 7fd8387e8700  2 req 2:0.000071::PUT
/v1/AUTH_2c5cfd34f2af41919847c0f4e0a28e04/container-33771340493::http
status=400
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.978880 7fd8387e8700  1 ====== req done req=0x7fd8968e4a50
http_status=400 ======
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982832 7fd895016780 20 enqueued request req=0x7fd8968e4e80
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982839 7fd895016780 20 RGWWQ:
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982841 7fd895016780 20 req: 0x7fd8968e4e80
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982844 7fd895016780 10 allocated request req=0x7fd8968e22c0
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982881 7fd81cfb1700 20 dequeued request req=0x7fd8968e4e80
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982891 7fd81cfb1700 20 RGWWQ: empty
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982894 7fd81cfb1700  1 ====== starting new request
req=0x7fd8968e4e80 =====
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982923 7fd81cfb1700  2 req 3:0.000029::::initializing
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982941 7fd81cfb1700 10
s->object=AUTH_2c5cfd34f2af41919847c0f4e0a28e04/container-33771340493/file-43854197694
s->bucket=v1
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982960 7fd81cfb1700  2 req 3:0.000066::PUT
/v1/AUTH_2c5cfd34f2af41919847c0f4e0a28e04/container-33771340493/file-43854197694::http
status=400
Mar  4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04
16:33:46.982995 7fd81cfb1700  1 ====== req done req=0x7fd8968e4e80
http_status=400 ======

Radosgw is started via upstart using this command:
setuidgid apache radosgw -n client.radosgw.gateway -f -d

Here is the relevant addition to ceph.conf:
[client.radosgw.gateway]
	rgw socket path = /var/run/radosgw/radosgw.sock
	rgw keystone url = http://172.17.2.3:35357
        rgw keystone admin token = vEPetvbcJl5V

Here is the apache config:
# Defaults.
ServerRoot /usr/lib64/apache2
PidFile /var/run/apache/apache.pid
HostnameLookups Off
# This shouldn't need to be anything valid, if we get redirects here then there
# is an error in our config, as we shouldn't be sending redirects with a
# different hostname than was provided by the client
ServerName localhost
ServerSignature Off
ServerTokens ProductOnly
Timeout 60
KeepAlive On
KeepAliveTimeout 5
StartServers 2
ServerLimit 2
ThreadsPerChild 64
MinSpareThreads 128
MaxSpareThreads 128
# For worker MPM, set MaxClients = ServerLimit * ThreadsPerChild
MaxClients 128
MaxRequestsPerChild 10000
ErrorLog syslog:daemon
TraceEnable Off
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule rewrite_module modules/mod_rewrite.so
DocumentRoot /var/www/empty
LogLevel info
<Directory />
	Options None
	AllowOverride None
	Order deny,allow
	Deny from all
</Directory>

# Rados gateway.
Listen 8888
LoadModule fastcgi_module modules/mod_fastcgi.so
FastCgiIpcDir /var/run/fastcgi
# Note: The fcgi file here doesn't have to exist since we start the application
# externally, but the directory structure should or apache will complain about
# the document root not existing.
FastCgiExternalServer /var/www/empty/radosgw.fcgi -socket
/var/run/radosgw/radosgw.sock
<Directory /var/www/empty>
	Options +ExecCGI
	SetHandler fastcgi-script
	Order allow,deny
	Allow from all
</Directory>
<VirtualHost *:8888>
	RewriteEngine On
	#RewriteRule ^(.*) /radosgw.fcgi/$1
	RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*)
/radosgw.fcgi?page=$1&params=$2&%{QUERY_STRING}
[E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
	#RewriteRule  ^/(.*) /radosgw.fcgi?params=$1&%{QUERY_STRING}
[E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
	AllowEncodedSlashes On
</VirtualHost>

I've tried all the rewrite rules above.

We would like to support both the swift and s3 APIs on this service,
using separate ports if we have to.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux