On Mon, Mar 4, 2013 at 8:49 AM, Nick Bartos <nick@xxxxxxxxxxxxxxx> wrote: > I'm trying to figure out what's wrong with my radosgw setup. I am > getting HTTP 400 responses when running some basic stat/upload tests, > which work fine when using openstack (folsom) swift. Additionally, I > get HTTP 400 responses when trying to use the openstack (folsom) > dashboard to list or create containers. > > I'm using the ceph-modified mod_fastcgi, and apache 2.2.23 with the worker mpm. > > Here is a snip of the log which starts after I start apache and radosgw: > > Mar 4 16:32:48 node-172-17-0-13 fcgi-pm[28606]: [notice] FastCGI: > process manager initialized (pid 28606) > Mar 4 16:32:48 node-172-17-0-13 apache2[28605]: [notice] > Apache/2.2.23 (Unix) mod_fastcgi/mod_fastcgi-SNAP-0910052141 > configured -- resuming normal operations > Mar 4 16:32:48 node-172-17-0-13 apache2[28605]: [info] Server built: > Mar 2 2013 21:17:00 > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.788234 7fd895016780 0 ceph version 0.56.3 > (6eb7e15a4783b122e9b0c85ea9ba064145958aa5), process radosgw, pid 28746 > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.788382 7fd895016780 0 ext_mime_map_init(): failed to open > file=/etc/mime.types ret=-2 > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.791035 7fd87f7fe700 2 garbage collection: start > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.811325 7fd87ebf9700 2 keystone revoke thread: start > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.811377 7fd87ebf9700 20 sending request to > http://172.17.2.3:35357/v2.0/tokens/revoked > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.812816 7fd895016780 10 allocated request req=0x7fd8968df5f0 > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.824017 7fd87ebf9700 10 request returned {"signed": > "-----BEGIN CMS-----\nMIIBQwYJKoZIhvcNAQcCoIIBNDCCATACAQExCTAHBgUrDgMCGjAeBgkqhkiG9w0B\nBwGgEQQPeyJyZXZva2VkIjogW119MYH/MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMx\nDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQx\nGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0B\nAQEFAASBgC7gYTOTf9SQEZKrk82UJVUUnPHyecpEI2kHb3IZ3iTmnm/IVHMAlNs7\n9Kmz6S9z0jFDWMqHxlnLAVuqzjGJMZFChuDTe4f/lRnHqd3I/ZlckVmAUw0ECAX5\nDmsQ+uziQYqHOiUGxeV0rwOu+x+khrxlbPv9x+2GcnIuEkG11ytr\n-----END > CMS-----\n"} > Mar 4 16:32:53 node-172-17-0-13 keystone: wsgi DEBUG 172.17.2.3 - - > [04/Mar/2013 16:32:53] "GET /v2.0/tokens/revoked HTTP/1.1" 200 633 > 0.011981 > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: add_child: <- signed > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.824115 7fd87ebf9700 10 signed=-----BEGIN CMS----- > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: > MIIBQwYJKoZIhvcNAQcCoIIBNDCCATACAQExCTAHBgUrDgMCGjAeBgkqhkiG9w0B > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: > BwGgEQQPeyJyZXZva2VkIjogW119MYH/MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMx > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: > DjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQx > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: > GDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0B > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: > AQEFAASBgC7gYTOTf9SQEZKrk82UJVUUnPHyecpEI2kHb3IZ3iTmnm/IVHMAlNs7 > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: > 9Kmz6S9z0jFDWMqHxlnLAVuqzjGJMZFChuDTe4f/lRnHqd3I/ZlckVmAUw0ECAX5 > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: > DmsQ+uziQYqHOiUGxeV0rwOu+x+khrxlbPv9x+2GcnIuEkG11ytr > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: -----END CMS----- > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.824124 7fd87ebf9700 10 > content=MIIBQwYJKoZIhvcNAQcCoIIBNDCCATACAQExCTAHBgUrDgMCGjAeBgkqhkiG9w0BBwGgEQQPeyJyZXZva2VkIjogW119MYH/MIH8AgEBMFwwVzELMAkGA1UEBhMCVVMxDjAMBgNVBAgTBVVuc2V0MQ4wDAYDVQQHEwVVbnNldDEOMAwGA1UEChMFVW5zZXQxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbQIBATAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgC7gYTOTf9SQEZKrk82UJVUUnPHyecpEI2kHb3IZ3iTmnm/IVHMAlNs79Kmz6S9z0jFDWMqHxlnLAVuqzjGJMZFChuDTe4f/lRnHqd3I/ZlckVmAUw0ECAX5DmsQ+uziQYqHOiUGxeV0rwOu+x+khrxlbPv9x+2GcnIuEkG11ytr > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.824132 7fd87ebf9700 0 ceph_decode_cms returned -95 > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.824138 7fd87ebf9700 0 ERROR: keystone revocation processing > returned error r=-95 > Mar 4 16:32:53 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:32:53.906447 7fd87f7fe700 2 garbage collection: stop > Mar 4 16:33:46 node-172-17-0-13 keystone: wsgi DEBUG 10.1.10.12 - - > [04/Mar/2013 16:33:46] "POST /v2.0/tokens HTTP/1.1" 200 2711 0.084738 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891625 7fd895016780 20 enqueued request req=0x7fd8968df5f0 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891641 7fd895016780 20 RGWWQ: > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891643 7fd895016780 20 req: 0x7fd8968df5f0 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891646 7fd895016780 10 allocated request req=0x7fd8968e4a50 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891700 7fd8237be700 20 dequeued request req=0x7fd8968df5f0 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891709 7fd8237be700 20 RGWWQ: empty > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891716 7fd8237be700 1 ====== starting new request > req=0x7fd8968df5f0 ===== > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891751 7fd8237be700 2 req 1:0.000035::::initializing > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891786 7fd8237be700 10 > s->object=AUTH_2c5cfd34f2af41919847c0f4e0a28e04 s->bucket=v1 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891812 7fd8237be700 2 req 1:0.000096::HEAD > /v1/AUTH_2c5cfd34f2af41919847c0f4e0a28e04::http status=400 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.891877 7fd8237be700 1 ====== req done req=0x7fd8968df5f0 > http_status=400 ====== > Mar 4 16:33:46 node-172-17-0-13 keystone: wsgi DEBUG 10.1.10.12 - - > [04/Mar/2013 16:33:46] "POST /v2.0/tokens HTTP/1.1" 200 2711 0.078482 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978717 7fd895016780 20 enqueued request req=0x7fd8968e4a50 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978726 7fd895016780 20 RGWWQ: > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978727 7fd895016780 20 req: 0x7fd8968e4a50 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978730 7fd895016780 10 allocated request req=0x7fd8968e4e80 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978763 7fd8387e8700 20 dequeued request req=0x7fd8968e4a50 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978769 7fd8387e8700 20 RGWWQ: empty > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978773 7fd8387e8700 1 ====== starting new request > req=0x7fd8968e4a50 ===== > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978800 7fd8387e8700 2 req 2:0.000028::::initializing > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978826 7fd8387e8700 10 > s->object=AUTH_2c5cfd34f2af41919847c0f4e0a28e04/container-33771340493 > s->bucket=v1 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978843 7fd8387e8700 2 req 2:0.000071::PUT That's wrong. Missing the path, maybe rewrite rule is broken? > /v1/AUTH_2c5cfd34f2af41919847c0f4e0a28e04/container-33771340493::http > status=400 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.978880 7fd8387e8700 1 ====== req done req=0x7fd8968e4a50 > http_status=400 ====== > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982832 7fd895016780 20 enqueued request req=0x7fd8968e4e80 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982839 7fd895016780 20 RGWWQ: > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982841 7fd895016780 20 req: 0x7fd8968e4e80 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982844 7fd895016780 10 allocated request req=0x7fd8968e22c0 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982881 7fd81cfb1700 20 dequeued request req=0x7fd8968e4e80 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982891 7fd81cfb1700 20 RGWWQ: empty > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982894 7fd81cfb1700 1 ====== starting new request > req=0x7fd8968e4e80 ===== > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982923 7fd81cfb1700 2 req 3:0.000029::::initializing > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982941 7fd81cfb1700 10 > s->object=AUTH_2c5cfd34f2af41919847c0f4e0a28e04/container-33771340493/file-43854197694 > s->bucket=v1 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982960 7fd81cfb1700 2 req 3:0.000066::PUT > /v1/AUTH_2c5cfd34f2af41919847c0f4e0a28e04/container-33771340493/file-43854197694::http > status=400 > Mar 4 16:33:46 node-172-17-0-13 radosgw.log: 2013-03-04 > 16:33:46.982995 7fd81cfb1700 1 ====== req done req=0x7fd8968e4e80 > http_status=400 ====== > > Radosgw is started via upstart using this command: > setuidgid apache radosgw -n client.radosgw.gateway -f -d > > Here is the relevant addition to ceph.conf: > [client.radosgw.gateway] > rgw socket path = /var/run/radosgw/radosgw.sock > rgw keystone url = http://172.17.2.3:35357 > rgw keystone admin token = vEPetvbcJl5V > > Here is the apache config: > # Defaults. > ServerRoot /usr/lib64/apache2 > PidFile /var/run/apache/apache.pid > HostnameLookups Off > # This shouldn't need to be anything valid, if we get redirects here then there > # is an error in our config, as we shouldn't be sending redirects with a > # different hostname than was provided by the client > ServerName localhost > ServerSignature Off > ServerTokens ProductOnly > Timeout 60 > KeepAlive On > KeepAliveTimeout 5 > StartServers 2 > ServerLimit 2 > ThreadsPerChild 64 > MinSpareThreads 128 > MaxSpareThreads 128 > # For worker MPM, set MaxClients = ServerLimit * ThreadsPerChild > MaxClients 128 > MaxRequestsPerChild 10000 > ErrorLog syslog:daemon > TraceEnable Off > LoadModule authz_host_module modules/mod_authz_host.so > LoadModule rewrite_module modules/mod_rewrite.so > DocumentRoot /var/www/empty > LogLevel info > <Directory /> > Options None > AllowOverride None > Order deny,allow > Deny from all > </Directory> > > # Rados gateway. > Listen 8888 > LoadModule fastcgi_module modules/mod_fastcgi.so > FastCgiIpcDir /var/run/fastcgi > # Note: The fcgi file here doesn't have to exist since we start the application > # externally, but the directory structure should or apache will complain about > # the document root not existing. > FastCgiExternalServer /var/www/empty/radosgw.fcgi -socket > /var/run/radosgw/radosgw.sock > <Directory /var/www/empty> > Options +ExecCGI > SetHandler fastcgi-script > Order allow,deny > Allow from all > </Directory> > <VirtualHost *:8888> > RewriteEngine On > #RewriteRule ^(.*) /radosgw.fcgi/$1 > RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*) > /radosgw.fcgi?page=$1¶ms=$2&%{QUERY_STRING} > [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] > #RewriteRule ^/(.*) /radosgw.fcgi?params=$1&%{QUERY_STRING} > [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] This is the one that you should use. Actually, you can try this: RewriteRule ^/(.*) /radosgw.fcgi?%{QUERY_STRING} [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L] > AllowEncodedSlashes On > </VirtualHost> > > I've tried all the rewrite rules above. > > We would like to support both the swift and s3 APIs on this service, > using separate ports if we have to. You can have it either way. Yehuda _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |