Hello, On Wed, 1 Jun 2016 12:31:41 -0500 Jeffrey McDonald wrote: > Hi, > > I just performed a minor ceph upgrade on my ubuntu 14.04 cluster from > ceph version to0.94.6-1trusty to 0.94.7-1trusty. Upon restarting the > OSDs, I receive the error message: > Unfortunately (despite what common sense would suggest) there are no minor upgrades in Ceph. As in, a lot of the point releases in stable, LTS Ceph versions have caused more havoc than major version upgrades, often due to backported features. Most notably in recent history the 0.94.6 release that had massive data loss potential with cache tiers. What you're seeing looks a lot like what was discussed and solved here, it's always a good idea to google for error messages you see: http://www.spinics.net/lists/ceph-devel/msg30450.html Regards, Christian > 2016-06-01 12:17:49.219512 7f64a70ea8c0 0 monclient: wait_auth_rotating > timed out after 30 > 2016-06-01 12:17:49.219605 7f64a70ea8c0 -1 osd.177 282877 unable to > obtain rotating service keys; retrying > 2016-06-01 12:18:19.219740 7f64a70ea8c0 0 monclient: wait_auth_rotating > timed out after 30 > 2016-06-01 12:18:19.219782 7f64a70ea8c0 -1 osd.177 282877 unable to > obtain rotating service keys; retrying > 2016-06-01 12:18:49.219869 7f64a70ea8c0 0 monclient: wait_auth_rotating > timed out after 30 > 2016-06-01 12:18:49.219908 7f64a70ea8c0 -1 osd.177 282877 unable to > obtain rotating service keys; retrying > > eventually the OSD fails to start. Oddly, this affects only some of > the OSDs on this host. > On the mon host, there is a warning: > > 2016-06-01 11:22:52.171152 osd.177 10.31.0.71:6842/10245 433 : cluster > [WRN] failed to encode map e282667 with expected crc > 2016-06-01 11:22:55.399487 osd.177 10.31.0.71:6842/10245 434 : cluster > [WRN] failed to encode map e282668 with expected crc > 2016-06-01 11:22:55.572238 osd.177 10.31.0.71:6842/10245 435 : cluster > [WRN] failed to encode map e282668 with expected crc > 2016-06-01 11:22:59.571924 osd.177 10.31.0.71:6842/10245 436 : cluster > [WRN] failed to encode map e282669 with expected crc > 2016-06-01 11:22:59.665925 osd.177 10.31.0.71:6842/10245 437 : cluster > [WRN] failed to encode map e282669 with expected crc > 2016-06-01 11:22:59.671475 osd.177 10.31.0.71:6842/10245 438 : cluster > [WRN] failed to encode map e282669 with expected crc > 2016-06-01 11:22:59.780646 osd.177 10.31.0.71:6842/10245 439 : cluster > [WRN] failed to encode map e282669 with expected crc > 2016-06-01 11:23:00.778250 osd.177 10.31.0.71:6842/10245 440 : cluster > [WRN] failed to encode map e282670 with expected crc > 2016-06-01 11:23:02.075996 osd.177 10.31.0.71:6842/10245 441 : cluster > [WRN] failed to encode map e282671 with expected crc > 2016-06-01 11:23:03.110117 osd.177 10.31.0.71:6842/10245 442 : cluster > [WRN] failed to encode map e282672 with expected crc > 2016-06-01 11:23:03.255597 osd.177 10.31.0.71:6842/10245 443 : cluster > [WRN] failed to encode map e282672 with expected crc > 2016-06-01 11:23:03.258223 osd.177 10.31.0.71:6842/10245 444 : cluster > [WRN] failed to encode map e282672 with expected crc > 2016-06-01 11:23:05.287753 osd.177 10.31.0.71:6842/10245 445 : cluster > [WRN] failed to encode map e282673 with expected crc > > > How do I clear these up after the upgrade? All of the filesystems on > the OSDs are mounted and the keyrings are there...... > > Thanks, > Jeff > > -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |