|
Recently upgraded a cluster from Nautilus to Pacific 16.2.7 and now my RGW will not start. I've cranked up the debug logging but still see no indication of what is causing the
problem.
The last error is: " Initialization timeout, failed to initialize".
I suspect it is because of something broken related to zonegroups/zones/realms but at this point there is only 1 realm (default), 1 ZG ( default) and 1 zone (default). There are a lot of "periods", but I'm not sure if it's safe to
delete them or not or even if they are related to the issue. I have about 60 GB of data spread across about 12 buckets and 10 users and need to get this running without losing data.
any suggestions?
The end of the logging looks like this:
2022-03-07T14:27:56.872-0500 7fa3fdf6eb40 2 all 8 watchers are set, enabling cache 2022-03-07T14:27:57.236-0500 7fa3fdf6eb40 20 rgw main: get_system_obj_state: rctx=0x7ffe5d5504f0 obj=default.rgw.log:meta.history state=0x55d297076bb0 s->prefetch_data=0 2022-03-07T14:27:57.236-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:27:57.236-0500 7fa3fdf6eb40 20 rgw main: get_system_obj_state: s->obj_tag was set empty 2022-03-07T14:27:57.236-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=50 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=10 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=883 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 14 rgw main: found period 38eddde8-c99d-4f37-9cb7-7ac55604e49e in local storage 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: find_oldest_period advancing to predecessor period 38eddde8-c99d-4f37-9cb7-7ac55604e49e 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=10 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=883 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 14 rgw main: found period a5745721-0d1d-42a2-9870-2c0e69d82549 in local storage 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: find_oldest_period advancing to predecessor period a5745721-0d1d-42a2-9870-2c0e69d82549 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=10 2022-03-07T14:27:57.260-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=883 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 14 rgw main: found period f7f602dc-44aa-4b48-ae83-0735cf855eca in local storage 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: find_oldest_period advancing to predecessor period f7f602dc-44aa-4b48-ae83-0735cf855eca 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=10 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=883 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 14 rgw main: found period 51e6a4c2-95a3-4fb0-aa07-854ff482bbbd in local storage 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: find_oldest_period advancing to predecessor period 51e6a4c2-95a3-4fb0-aa07-854ff482bbbd 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=10 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: rados->read ofs=0 len=0 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: rados_obj.operate() r=0 bl.length=249 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 14 rgw main: found period dc26cf62-8141-4eb3-9168-58e681eea2ce in local storage 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 20 rgw main: find_oldest_period advancing to predecessor period dc26cf62-8141-4eb3-9168-58e681eea2ce 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 10 rgw main: find_oldest_period returning first period dc26cf62-8141-4eb3-9168-58e681eea2ce 2022-03-07T14:27:57.264-0500 7fa3fdf6eb40 10 rgw main: rewriting mdlog history 2022-03-07T14:27:57.464-0500 7fa3fdf6eb40 10 rgw main: distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.log:meta.history, ofs0, ns] 2022-03-07T14:27:57.468-0500 7fa3ea7fc700 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 3610130465690709 cookie 94362967315728 notifier 536607355 bl.length()=225 2022-03-07T14:27:57.468-0500 7fa3fdf6eb40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2022-03-07T14:27:57.468-0500 7fa3fdf6eb40 20 check_secure_mon_conn(): mode 1 is insecure 2022-03-07T14:28:19.236-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:28:41.237-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:29:03.237-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:29:25.237-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:29:47.238-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:30:09.238-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:30:31.239-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:30:53.235-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start ^[[A2022-03-07T14:31:15.235-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:31:37.236-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:31:59.236-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:32:21.236-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:32:43.237-0500 7fa3c1fe3700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2022-03-07T14:32:51.357-0500 7fa3fcb7d700 -1 Initialization timeout, failed to initialize
thanks,
Will Ingersoll |
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
