Hi, we are currently using the patched fastcgi version (2.4.7-0910042141-6-gd4fffda) Updating to a more recent version is currently blocked by http://tracker.ceph.com/issues/6453 Is there a documentation for running radosgw with nginx? I only find some mailinglist posts with some config snippets. Sebastian On 30.11.2013, at 20:46, Andrew Woodward wrote: > Are you using the inktank patched FastCGI sever? http://gitbuilder.ceph.com > > Alternately try another script sever like ngnix as already suggested. > > On Nov 29, 2013 12:23 PM, "German Anders" <ganders@xxxxxxxxxxxx> wrote: > Thanks a lot Sebastian, i'm going to try that, also i'm having an issue while trying to test a rbd creation, i've install in the deploy server the ceph-client: > > ceph@ceph-deploy01:/etc/ceph$ sudo rbd -n client.ceph-test -k /home/ceph/ceph-cluster/ceph.client.admin.keyring create --size 10240 cephdata > 2013-11-29 15:20:25.683930 7fcd9979c780 0 librados: client.ceph-openstack authentication error (1) Operation not permitted > rbd: couldn't connect to the cluster! > > Anyone know what could be the issue here? maybe it has something to do with keys or maybe not... > > Thanks in advance, > > Best regards, > > German Anders > > > > > > > >> --- Original message --- >> Asunto: Re: radosgw daemon stalls on download of some files >> De: Sebastian <webmaster@xxxxxxxx> >> Para: ceph-users <ceph-users@xxxxxxxxxxxxxx> >> Fecha: Friday, 29/11/2013 16:18 >> >> Hi Yehuda, >> >> >>> It's interesting, the responses are received but seems that they >>> aren't being handled (hence the following pings). There are a few >>> things that you could look at. First, try to connect to the admin >>> socket and see if you get any useful information from there. This >>> could include in-flight requests, look for other requests that have >>> not completed. Also see if there's indication for requests throttling. >> >> Do you refer to the methods mentioned here? http://ceph.com/docs/dumpling/radosgw/troubleshooting/? >> Unfortunately the socket file is not present. Do i have to activate it in the config somehow? I could not find any reference to that in the docs. Is it already included in my radosgw version? >> radosgw -v >> ceph version 0.67.4 (ad85b8bfafea6232d64cb7ba76a8b6e8252fa0c7) >> >>> Another thing to look at would be at the seemingly unrelated timeout >>> messages. These should not happen and might indicate that there's >>> something that is holding you up that shouldn't. Try searching for the >>> same thread id that is specified in these messages (omit the 0x >>> prefix), and see what's the last thing that it's doing. >> >> I checked that: >> http://pastebin.com/Z23PWwjt >> i do not see anything unusual before the messages happen, but maybe you see something odd. >> >> >>> You could also try turning on also 'debug objecter = 20', see if it >>> provides more info (it's very verbose though). >>> >> >> Did that, but that is way to verbose for me ;) I uploaded it here: >> http://pastebin.com/VBPAVP6z >> There might be some requests mixed into it, but the one for cdn/52974400c6dd6ca719000004/source.avi is the one that stalled. >> >>> How much are you loading the gateway before that happens? We've seen a >>> similar issue in the past that was related to the fcgi library that is >>> dynamically linked with the radosgw process (that is, not the apache >>> mod_fastcgi module). This, however, would only happen when there's >>> heavy load and the fd numbers handled by the radosgw surpassed 1024 >>> (buggy library that was using select() instead of poll()). >> >> There are not that many requests on the Storage, maybe 10-20 req/min. The cluster serves as a source for a CDN, so once the resource is fetched it should not be fetched again soon. I checked for the open files, and there are only about 10-20 open file handles for the radosgw process. So this probably is not the issue. >> >> Sebastian >> >> >>> >>> Yehuda >>> >>> On Fri, Nov 29, 2013 at 7:28 AM, Sebastian <webmaster@xxxxxxxx> wrote: >>>> Hi, >>>> >>>> thanks for the hint. I tried this again and noticed that the time out message does seem to be unrelated. Here is the log file for a stalling request with debug turned on: >>>> http://pastebin.com/DcQuc9wP >>>> >>>> I really cannot really find a real "error" in the log. The download stalls at about 500kb at that point though. Restarting radosgw fixes it for 1 download only, the next one is broken again. But as i said this does not happen for all files. >>>> >>>> Sebastian >>>> >>>> On 27.11.2013, at 21:53, Yehuda Sadeh wrote: >>>> >>>>> On Wed, Nov 27, 2013 at 4:46 AM, Sebastian <webmaster@xxxxxxxx> wrote: >>>>>> Hi, >>>>>> >>>>>> we have a setup of 4 Servers running ceph and radosgw. We use it as an internal S3 service for our files. The Servers run Debian Squeeze with Ceph 0.67.4. >>>>>> >>>>>> The cluster has been running smoothly for quite a while, but we are currently experiencing issues with the radosgw. For some files the HTTP Download just stalls at around 500kb. >>>>>> >>>>>> The Apache error log just says: >>>>>> [error] [client ] FastCGI: comm with server "/var/www/s3gw.fcgi" aborted: idle timeout (30 sec) >>>>>> [error] [client ] Handler for fastcgi-script returned invalid result code 1 >>>>>> >>>>>> radosgw logging: >>>>>> 7f00bc66a700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7f00934bb700' had timed out after 600 >>>>>> 7f00bc66a700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7f00ab4eb700' had timed out after 600 >>>>>> >>>>>> The interesting thing is that the cluster health is fine an only some files are not working properly. Most of them just work fine. A restart of radosgw fixes the issue. The other ceph logs are also clean. >>>>>> >>>>>> Any idea why this happens? >>>>>> >>>>> >>>>> No, but you can turn on 'debug ms = 1' on your gateway ceph.conf, and >>>>> that might give some better indication. >>>>> >>>>> Yehuda >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |