[0.94.4] radosgw initialization timeout, failed to initialize

["James O'Neill" <hemebond@xxxxxxxxx>]

I upgraded to 0.94.4 yesterday and now radosgw will not run on any of the servers. The service itself will run, but it's not listening (using civetweb with port 80 specified). Run manually I get the follow output:

root@dbp-ceph01:~# /usr/bin/radosgw --cluster=ceph --id radosgw.gateway -d

2015-10-23 08:23:31.364333 7fb5fba58840 0 ceph version 0.94.4 (95292699291242794510b39ffde3f4df67898d3a), process radosgw, pid 17928

2015-10-23 08:23:31.380126 7fb5fba56700 0 -- :/1017928 >> 127.0.0.1:6789/0 pipe(0x2fc8aa0 sd=7 :0 s=1 pgs=0 cs=0 l=1 c=0x2fccd90).fault

2015-10-23 08:23:34.389209 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:23:34.389933 7fb5fba58840 20 get_obj_state: rctx=0x7fffab0b0490 obj=.rgw.root:default.region state=0x2fc9b00 s->prefetch_data=0

2015-10-23 08:23:34.397513 7fb5fba58840 20 get_obj_state: s->obj_tag was set empty

2015-10-23 08:23:34.397696 7fb5fba58840 20 get_obj_state: rctx=0x7fffab0b0490 obj=.rgw.root:default.region state=0x2fc9b00 s->prefetch_data=0

2015-10-23 08:23:34.397767 7fb5fba58840 20 rados->read ofs=0 len=524288

2015-10-23 08:23:34.399305 7fb5fba58840 20 rados->read r=0 bl.length=17

2015-10-23 08:23:34.399553 7fb5fba58840 20 get_obj_state: rctx=0x7fffab0b0490 obj=.rgw.root:region_info.default state=0x2fc9ad0 s->prefetch_data=0

2015-10-23 08:23:34.401343 7fb5fba58840 20 get_obj_state: s->obj_tag was set empty

2015-10-23 08:23:34.401516 7fb5fba58840 20 get_obj_state: rctx=0x7fffab0b0490 obj=.rgw.root:region_info.default state=0x2fc9ad0 s->prefetch_data=0

2015-10-23 08:23:34.401585 7fb5fba58840 20 rados->read ofs=0 len=524288

2015-10-23 08:23:34.403133 7fb5fba58840 20 rados->read r=0 bl.length=153

2015-10-23 08:23:34.403218 7fb5fba58840 20 get_obj_state: rctx=0x7fffab0b0630 obj=.rgw.root:zone_info.default state=0x2fc9ad0 s->prefetch_data=0

2015-10-23 08:23:34.405494 7fb5fba58840 20 get_obj_state: s->obj_tag was set empty

2015-10-23 08:23:34.405528 7fb5fba58840 20 get_obj_state: rctx=0x7fffab0b0630 obj=.rgw.root:zone_info.default state=0x2fc9ad0 s->prefetch_data=0

2015-10-23 08:23:34.405535 7fb5fba58840 20 rados->read ofs=0 len=524288

2015-10-23 08:23:34.407244 7fb5fba58840 20 rados->read r=0 bl.length=678

2015-10-23 08:23:34.407322 7fb5fba58840 2 zone default is master

2015-10-23 08:23:34.407490 7fb5fba58840 20 get_obj_state: rctx=0x7fffab0b0630 obj=.rgw.root:region_map state=0x2fc9ad0 s->prefetch_data=0

2015-10-23 08:23:56.389610 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:24:18.389851 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:24:40.390144 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:25:02.390385 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:25:24.390609 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:25:46.390908 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:26:08.391126 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:26:30.391357 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:26:52.391679 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:27:14.391980 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:27:36.392259 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:27:58.392475 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:28:20.392762 7fb5e5ffb700 2 RGWDataChangesLog::ChangesRenewThread: start

2015-10-23 08:28:31.364740 7fb5f0204700 -1 Initialization timeout, failed to initialize

Between each of those ChangesRenewThread entries are lines like:

2015-10-23 08:35:08.379161 7f6efaffd700 1 -- 172.17.241.161:0/1018414 --> 172.17.241.163:6800/31263 -- ping magic: 0 v1 -- ?+0 0x7f6ecc000ab0 con 0x3947710

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com