This turns out to have been a configuration change to nginx that I forgot I had made. It wasn't passing all the http options through any more, so authentication was failing. On 5/13/2014 12:43 PM, Brian Rak wrote: > > On 5/13/2014 12:29 PM, Yehuda Sadeh wrote: >> On Tue, May 13, 2014 at 8:52 AM, Brian Rak <brak at gameservers.com> wrote: >>> I hit a "bug" where radosgw crashed with >>> >>> -101> 2014-05-13 15:26:07.188494 7fde82886820 0 ERROR: FCGX_Accept_r >>> returned -24 >> too many files opened. You probably need to adjust your limits. >> >> >>> .... >>> 0> 2014-05-13 15:26:07.193772 7fde82886820 -1 rgw/rgw_main.cc: In >>> function >>> 'virtual void RGWProcess::RGWWQ::_clear()' thread 7fde82886820 time >>> 2014-05-13 15:26:07.192212 >>> rgw/rgw_main.cc: 181: FAILED assert(process->m_req_queue.empty()) >>> >>> ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) >>> 1: (ThreadPool::WorkQueue<RGWRequest>::_process(RGWRequest*)+0) >>> [0x4ae9a0] >>> 2: (ThreadPool::stop(bool)+0x1f5) [0x7fde8193ee05] >>> 3: (RGWProcess::run()+0x358) [0x4ab808] >>> 4: (main()+0x866) [0x4ac796] >>> 5: (__libc_start_main()+0xfd) [0x7fde7fe02d1d] >>> 6: radosgw() [0x45d239] >>> >>> >>> It seems it ran out of fds, so I increased the limit and restarted it. >>> However, now the user I was using can no longer access anything. Every >>> action the user attempts is rejected with a 403 error. >>> >>> How do I enable logging of authentication issues? I don't see anything >>> obvious in the config reference, and running radosgw with --debug >>> doesn't >>> produce output that makes sense. >> >> I usualy set "debug rgw = 20", and "debug ms = 1". > Ah, that helped. It seems that s3cmd and ceph are calculating > different signatures for some reason. Aside from a mismatch between > public/secret keys, is there something else that could be causing this? > > > 2014-05-13 16:39:30.895180 7fb0db5fe700 20 enqueued request > req=0x7fb0f0012ce0 > 2014-05-13 16:39:30.895201 7fb0db5fe700 20 RGWWQ: > 2014-05-13 16:39:30.895202 7fb0db5fe700 20 req: 0x7fb0f0012ce0 > 2014-05-13 16:39:30.895206 7fb0db5fe700 10 allocated request > req=0x7fb0f00148d0 > 2014-05-13 16:39:30.895281 7fb0cb1e4700 20 dequeued request > req=0x7fb0f0012ce0 > 2014-05-13 16:39:30.895286 7fb0cb1e4700 20 RGWWQ: empty > 2014-05-13 16:39:30.895310 7fb0cb1e4700 20 CONTENT_LENGTH=0 > 2014-05-13 16:39:30.895311 7fb0cb1e4700 20 CONTENT_TYPE= > 2014-05-13 16:39:30.895313 7fb0cb1e4700 20 FCGI_ROLE=RESPONDER > 2014-05-13 16:39:30.895314 7fb0cb1e4700 20 HTTP_ACCEPT_ENCODING=identity > 2014-05-13 16:39:30.895315 7fb0cb1e4700 20 HTTP_AUTHORIZATION=AWS > MYACCESSKEY:FmglMmhJONIhpRHw7z9DgtvdnDI= > 2014-05-13 16:39:30.895316 7fb0cb1e4700 20 HTTP_CONTENT_LENGTH=0 > 2014-05-13 16:39:30.895317 7fb0cb1e4700 20 HTTP_HOST=MYHOST > 2014-05-13 16:39:30.895318 7fb0cb1e4700 20 HTTP_X_AMZ_DATE=Tue, 13 May > 2014 16:39:30 +0000 > 2014-05-13 16:39:30.895319 7fb0cb1e4700 20 QUERY_STRING= > 2014-05-13 16:39:30.895320 7fb0cb1e4700 20 REQUEST_METHOD=GET > 2014-05-13 16:39:30.895322 7fb0cb1e4700 1 ====== starting new request > req=0x7fb0f0012ce0 ===== > 2014-05-13 16:39:30.895335 7fb0cb1e4700 2 req 5:0.000013::GET > ::initializing > 2014-05-13 16:39:30.895340 7fb0cb1e4700 10 host=MYHOST > rgw_dns_name=MYHOST > 2014-05-13 16:39:30.895353 7fb0cb1e4700 10 meta>> HTTP_X_AMZ_DATE > 2014-05-13 16:39:30.895358 7fb0cb1e4700 10 x>> x-amz-date:Tue, 13 May > 2014 16:39:30 +0000 > 2014-05-13 16:39:30.895374 7fb0cb1e4700 10 s->object=<NULL> > s->bucket=<NULL> > 2014-05-13 16:39:30.895380 7fb0cb1e4700 2 req 5:0.000059:s3:GET > ::getting op > 2014-05-13 16:39:30.895384 7fb0cb1e4700 2 req 5:0.000062:s3:GET > :list_buckets:authorizing > 2014-05-13 16:39:30.895423 7fb0cb1e4700 20 get_obj_state: > rctx=0x7fb0f00141f0 obj=.users:MYACCESSKEY state=0x7fb0f0017d48 > s->prefetch_data=0 > 2014-05-13 16:39:30.895432 7fb0cb1e4700 10 cache get: > name=.users+MYACCESSKEY : hit > 2014-05-13 16:39:30.895439 7fb0cb1e4700 20 get_obj_state: s->obj_tag > was set empty > 2014-05-13 16:39:30.895445 7fb0cb1e4700 10 cache get: > name=.users+MYACCESSKEY : hit > 2014-05-13 16:39:30.895478 7fb0cb1e4700 20 get_obj_state: > rctx=0x7fb0f00141f0 obj=.users.uid:centosmirror state=0x7fb0f00186f8 > s->prefetch_data=0 > 2014-05-13 16:39:30.895483 7fb0cb1e4700 10 cache get: > name=.users.uid+centosmirror : hit > 2014-05-13 16:39:30.895486 7fb0cb1e4700 20 get_obj_state: s->obj_tag > was set empty > 2014-05-13 16:39:30.895490 7fb0cb1e4700 10 cache get: > name=.users.uid+centosmirror : hit > 2014-05-13 16:39:30.895579 7fb0cb1e4700 10 get_canon_resource(): dest= > 2014-05-13 16:39:30.895583 7fb0cb1e4700 10 auth_hdr: > GET > > > > x-amz-date:Tue, 13 May 2014 16:39:30 +0000 > > 2014-05-13 16:39:30.895650 7fb0cb1e4700 15 calculated > digest=ck/6o9TgR73JLPT43SxIt39KgBI= > 2014-05-13 16:39:30.895653 7fb0cb1e4700 15 > auth_sign=FmglMmhJONIhpRHw7z9DgtvdnDI= > 2014-05-13 16:39:30.895654 7fb0cb1e4700 15 compare=-1 > 2014-05-13 16:39:30.895657 7fb0cb1e4700 10 failed to authorize request > 2014-05-13 16:39:30.895681 7fb0cb1e4700 5 nothing to log for operation > 2014-05-13 16:39:30.895684 7fb0cb1e4700 2 req 5:0.000362:s3:GET > :list_buckets:http status=403 > 2014-05-13 16:39:30.895688 7fb0cb1e4700 1 ====== req done > req=0x7fb0f0012ce0 http_status=403 ====== > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |