RadosGW problems

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Kurt,

I have pointed my DNS '*.gateway.testes.local' and 'gateway.testes.local,
to the same IP (the radosgw server).

I have added rgw_dns_name has you suggested to the config (it was comment
out). I will try everything and give feedback.

By the way, when I restart ceph-radosgw service, I get this in the logs
(which previous I did not see anything):

2014-08-18 15:19:44.812039 7fbf417fa700  1 handle_sigterm
2014-08-18 15:19:44.812104 7fbf417fa700  1 handle_sigterm set alarm for 120
2014-08-18 15:19:44.812235 7fbf5c495880 -1 shutting down
2014-08-18 15:19:44.812305 7fbf40ff9700  0 ERROR: FCGX_Accept_r returned -4
2014-08-18 15:19:44.812432 7fbf417fa700  1 handle_sigterm
2014-08-18 15:19:44.857506 7fbf5c495880  1 final shutdown
2014-08-18 15:19:45.010597 7fb318b96880  0 ceph version 0.80.5
(38b73c67d375a2552d8ed67843c8a65c2c0feba6), process radosgw, pid 3242
2014-08-18 15:19:45.219582 7fb318b96880  0 framework: fastcgi
2014-08-18 15:19:45.219599 7fb318b96880  0 starting handler: fastcgi
2014-08-18 15:19:45.692248 7fb2fe6fb700  0 ERROR: can't read user header:
ret=-2
2014-08-18 15:19:45.692273 7fb2fe6fb700  0 ERROR: sync_user() failed,
user=teste ret=-2

The last 2 lines look suspicious...


*Marco Garc?s*
*#sysadmin*
Maputo - Mozambique
*[Phone]* +258 84 4105579
*[Skype]* marcogarces


On Mon, Aug 18, 2014 at 2:58 PM, Bachelder, Kurt <
Kurt.Bachelder at sierra-cedar.com> wrote:

>  Hi Marco,
>
>  Is your DNS setup to use the wildcard (*.gateway.testes.local)?
>
>  I noticed that you're using it in the server alias, but that you don't
> have an "rgw_dns_name" configured in your ceph.conf.  The rgw_dns_name
> should be set to "gateway.testes.local" if your dns is configured to use
> the wildcard naming with that subdomain.
>
>  Also see that you're using SSL... which domain have you signed?
> *.gateway.testes.local?
>
>  Since you can create a bucket, but not write to it, I'm wondering if
> there's an issue with the way your client is attempting to access the
> bucket... can you resolve <bucket>.gateway.testes.local from your client?
>
>  Kurt
>
>
> -------- Original message --------
> From: Marco Garc?s
> Date:08/18/2014 6:33 AM (GMT-05:00)
> To: Linux Chips
> Cc: "Bachelder, Kurt" , ceph-users at lists.ceph.com
> Subject: Re: [ceph-users] RadosGW problems
>
>   Hi there,
>
>  I have FastCgiWrapper Off in fastcgi.conf file; I also have SELinux in
> permissive state; 'ps aux | grep rados' shows me radosgw is running;
>
>  The problems stays the same... I can login with S3 credentials, create
> buckets, but uploads write this in the logs:
> [Mon Aug 18 12:00:28.636378 2014] [:error] [pid 11251] [client
> 10.5.1.1:49680] FastCGI: comm with server "/var/www/cgi-bin/s3gw.fcgi"
> aborted: idle timeout (3
> 0 sec)
> [Mon Aug 18 12:00:28.676825 2014] [:error] [pid 11251] [client
> 10.5.1.1:49680] FastCGI: incomplete headers (0 bytes) received from
> server "/var/www/cgi-bin/s3
> gw.fcgi"
>
>  When I try Swift credentials, I cannot login at all.. I have tested both
> Cyberduck and Swift client on the command line, and I always get this on
> the logs:
> "GET /v1.0 HTTP/1.1" 404 78 "-" "Cyberduck/4.5 (Mac OS X/10.9.3) (x86_64)"
> "GET /v1.0 HTTP/1.1" 404 78 "-" "python-swiftclient-2.2.0"
>
>  In S3 login, when I upload a file, I can see it almost at 100% complete,
> but then it fails with the above errors.
>
>  A strange thing is... the /var/log/ceph/client.radosgw.gateway.log is
> not getting updated, I don't see any new logs in there.
>
>  Thank you once again for your help, Marco Garc?s
>
>
>  *Marco Garc?s*
> *#sysadmin*
>  Maputo - Mozambique
> *[Phone]* +258 84 4105579
> *[Skype]* marcogarces
>
>
> On Mon, Aug 18, 2014 at 12:08 AM, Linux Chips <linux.chips at gmail.com>
> wrote:
>
>> On Mon 18 Aug 2014 12:45:33 AM AST, Bachelder, Kurt wrote:
>>
>>> Hi Marco ?
>>>
>>> In CentOS 6, you also had to edit /etc/httpd/conf.d/fastcgi.conf to
>>> turn OFF the fastcgi wrapper.  I haven?t tested in v7 yet, but I?d
>>> guess it?s required there too:
>>>
>>> # wrap all fastcgi script calls in suexec
>>>
>>> FastCgiWrapper Off
>>>
>>> Give that a try, if you haven?t already ? restart httpd and
>>> ceph-radosgw afterward.
>>>
>>> Kurt
>>>
>>>  *From:*ceph-users [mailto:ceph-users-bounces at lists.ceph.com] *On
>>> Behalf Of *Marco Garc?s
>>> *Sent:* Friday, August 15, 2014 12:46 PM
>>> *To:* ceph-users at lists.ceph.com
>>> *Subject:* [ceph-users] RadosGW problems
>>>
>>>
>>> Hi there,
>>>
>>> I am using CentOS 7 with Ceph version 0.80.5
>>> (38b73c67d375a2552d8ed67843c8a65c2c0feba6), 3 OSD, 3 MON, 1 RadosGW
>>> (which also serves as ceph-deploy node)
>>>
>>> I followed all the instructions in the docs, regarding setting up a
>>> basic Ceph cluster, and then followed the one to setup RadosGW.
>>>
>>> I can't seem to use the Swift interface, and the S3 interface, times
>>> out after 30 seconds.
>>>
>>> [Fri Aug 15 18:25:33.290877 2014] [:error] [pid 6197] [client
>>>  10.5.5.222:58051 <http://10.5.5.222:58051>] FastCGI: comm with server
>>>
>>> "/var/www/cgi-bin/s3gw.fcgi" aborted: idle timeout (30 sec)
>>>
>>> [Fri Aug 15 18:25:33.291781 2014] [:error] [pid 6197] [client
>>>  10.5.5.222:58051 <http://10.5.5.222:58051>] FastCGI: incomplete
>>>
>>> headers (0 bytes) received from server "/var/www/cgi-bin/s3gw.fcgi"
>>>
>>>  *My ceph.conf:*
>>>
>>>
>>> [global]
>>>
>>> fsid = 581bcd61-8760-4756-a7c8-e8275c0957ad
>>>
>>> mon_initial_members = CEPH01, CEPH02, CEPH03
>>>
>>> mon_host = 10.2.27.81,10.2.27.82,10.2.27.83
>>>
>>>  public network = 10.2.27.0/25 <http://10.2.27.0/25>
>>>
>>>
>>> auth_cluster_required = cephx
>>>
>>> auth_service_required = cephx
>>>
>>> auth_client_required = cephx
>>>
>>> filestore_xattr_use_omap = true
>>>
>>> osd pool default size = 2
>>>
>>> osd pool default pg num = 333
>>>
>>> osd pool default pgp num = 333
>>>
>>> osd journal size = 1024
>>>
>>> [client.radosgw.gateway]
>>>
>>> host = GATEWAY
>>>
>>> keyring = /etc/ceph/ceph.client.radosgw.keyring
>>>
>>> rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
>>>
>>> log file = /var/log/ceph/client.radosgw.gateway.log
>>>
>>> rgw print continue = false
>>>
>>> rgw enable ops log = true
>>>
>>>  *My apache rgw.conf:*
>>>
>>>
>>> FastCgiExternalServer /var/www/cgi-bin/s3gw.fcgi -socket
>>> /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
>>>
>>> <VirtualHost *:443>
>>>
>>>         SSLEngine on
>>>
>>>         SSLCertificateFile /etc/pki/tls/certs/ca_rgw.crt
>>>
>>>         SSLCertificateKeyFile /etc/pki/tls/private/ca_rgw.key
>>>
>>>         SetEnv SERVER_PORT_SECURE 443
>>>
>>>         ServerName gateway.testes.local
>>>
>>>         ServerAlias *.gateway.testes.local
>>>
>>>         ServerAdmin marco.garces at testes.co.mz
>>>  <mailto:marco.garces at testes.co.mz>
>>>
>>>
>>>         DocumentRoot /var/www/cgi-bin
>>>
>>>         RewriteEngine On
>>>
>>>         #RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING}
>>> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
>>>
>>>         RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*)
>>> /s3gw.fcgi?page=$1&params=$2&%{QUERY_STRING}
>>> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
>>>
>>>         <IfModule mod_fastcgi.c>
>>>
>>>                 <Directory /var/www>
>>>
>>>                         Options +ExecCGI
>>>
>>>                         AllowOverride All
>>>
>>>                         SetHandler fastcgi-script
>>>
>>>                         Order allow,deny
>>>
>>>                         Allow from all
>>>
>>>                         AuthBasicAuthoritative Off
>>>
>>>                 </Directory>
>>>
>>>         </IfModule>
>>>
>>>         AllowEncodedSlashes On
>>>
>>>         ErrorLog /var/log/httpd/error_rgw_ssl.log
>>>
>>>         CustomLog /var/log/httpd/access_rgw_ssl.log combined
>>>
>>>         ServerSignature Off
>>>
>>> </VirtualHost>
>>>
>>>  *My /var/www/cgi-bin/s3gw.fcgi *
>>>
>>>
>>> #!/bin/sh
>>>
>>> exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway
>>>
>>>  *My Rados user:*
>>>
>>>
>>> radosgw-admin user info --uid=johndoe
>>>
>>> { "user_id": "johndoe",
>>>
>>>   "display_name": "John Doe",
>>>
>>>    "email": "john at example.com <mailto:john at example.com>",
>>>
>>>
>>>   "suspended": 0,
>>>
>>>   "max_buckets": 1000,
>>>
>>>   "auid": 0,
>>>
>>>   "subusers": [
>>>
>>>         { "id": "johndoe:swift",
>>>
>>>           "permissions": "full-control"}],
>>>
>>>   "keys": [
>>>
>>>         { "user": "johndoe:swift",
>>>
>>>           "access_key": "265DJESOJGSK953EE4LE",
>>>
>>>           "secret_key": ""},
>>>
>>>         { "user": "johndoe",
>>>
>>>           "access_key": "U4AR5757MCON3AZYAB97",
>>>
>>>           "secret_key": "05rg47Oa+njo8uxTeX+urBPF0ZRPWvVq8nfrC5cN"}],
>>>
>>>   "swift_keys": [
>>>
>>>         { "user": "johndoe:swift",
>>>
>>>           "secret_key": "Lags5xwX5aiPgkG\/QqA8HygKs6AQYO46dBXS0ZGS"}],
>>>
>>>   "caps": [],
>>>
>>>   "op_mask": "read, write, delete",
>>>
>>>   "default_placement": "",
>>>
>>>   "placement_tags": [],
>>>
>>>   "bucket_quota": { "enabled": false,
>>>
>>>       "max_size_kb": -1,
>>>
>>>       "max_objects": -1},
>>>
>>>   "user_quota": { "enabled": false,
>>>
>>>       "max_size_kb": -1,
>>>
>>>       "max_objects": -1},
>>>
>>>   "temp_url_keys": []}
>>>
>>> I can reach https://gateway.testes.local, and I can login with S3, but
>>> cant login with Swift (using Cyberduck). Also, I can create buckets
>>> with S3, but if I upload a file, it times out with the error above.
>>> There is a necessity to use both the S3 and Swift API.
>>>
>>> Can you help me? Thank you in advance, regards, Marco Garc?s
>>>
>>>
>>>
>>>  _______________________________________________
>>> ceph-users mailing list
>>> ceph-users at lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>> this error appears when httpd can not talk to radosgw process. dose it
>> happen with all the requests or some of them? generally if it worked with
>> one request it would with all of them, except when it is under load. so if
>> you can create a bucket you should be able to upload files.
>> i did not try centos7, but it may be similar, as we use 6. may be it is a
>> permission issue (if it happens all the time). try turning selinux to
>> permissive and see (setenforce 0), make sure the file is readable/writable
>> by both the radosgw and httpd. also confirm that radosgw is running (ps aux
>> | grep rados) as it tends to silently die if some thing is not correct,  it
>> drove me nuts until i figured it was the log file that it can not write to.
>> if you have high number of connection you would also need to patch fcgi
>> to use poll instead of select. that was the only way we could push it to
>> serve more than 900 users.
>> we have all our radosgws running both swift and s3 with no special
>> config, it just worked.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140818/4085877f/attachment.htm>


[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux