Would you enable debug for osd.177 debug osd = 20 debug filestore = 20 debug ms = 1 Cheers, Shinobu On Thu, Jun 2, 2016 at 2:31 AM, Jeffrey McDonald <jmcdonal@xxxxxxx> wrote: > Hi, > > I just performed a minor ceph upgrade on my ubuntu 14.04 cluster from ceph > version to0.94.6-1trusty to 0.94.7-1trusty. Upon restarting the OSDs, I > receive the error message: > > 2016-06-01 12:17:49.219512 7f64a70ea8c0 0 monclient: wait_auth_rotating > timed out after 30 > 2016-06-01 12:17:49.219605 7f64a70ea8c0 -1 osd.177 282877 unable to obtain > rotating service keys; retrying > 2016-06-01 12:18:19.219740 7f64a70ea8c0 0 monclient: wait_auth_rotating > timed out after 30 > 2016-06-01 12:18:19.219782 7f64a70ea8c0 -1 osd.177 282877 unable to obtain > rotating service keys; retrying > 2016-06-01 12:18:49.219869 7f64a70ea8c0 0 monclient: wait_auth_rotating > timed out after 30 > 2016-06-01 12:18:49.219908 7f64a70ea8c0 -1 osd.177 282877 unable to obtain > rotating service keys; retrying > > eventually the OSD fails to start. Oddly, this affects only some of the > OSDs on this host. > On the mon host, there is a warning: > > 2016-06-01 11:22:52.171152 osd.177 10.31.0.71:6842/10245 433 : cluster [WRN] > failed to encode map e282667 with expected crc > 2016-06-01 11:22:55.399487 osd.177 10.31.0.71:6842/10245 434 : cluster [WRN] > failed to encode map e282668 with expected crc > 2016-06-01 11:22:55.572238 osd.177 10.31.0.71:6842/10245 435 : cluster [WRN] > failed to encode map e282668 with expected crc > 2016-06-01 11:22:59.571924 osd.177 10.31.0.71:6842/10245 436 : cluster [WRN] > failed to encode map e282669 with expected crc > 2016-06-01 11:22:59.665925 osd.177 10.31.0.71:6842/10245 437 : cluster [WRN] > failed to encode map e282669 with expected crc > 2016-06-01 11:22:59.671475 osd.177 10.31.0.71:6842/10245 438 : cluster [WRN] > failed to encode map e282669 with expected crc > 2016-06-01 11:22:59.780646 osd.177 10.31.0.71:6842/10245 439 : cluster [WRN] > failed to encode map e282669 with expected crc > 2016-06-01 11:23:00.778250 osd.177 10.31.0.71:6842/10245 440 : cluster [WRN] > failed to encode map e282670 with expected crc > 2016-06-01 11:23:02.075996 osd.177 10.31.0.71:6842/10245 441 : cluster [WRN] > failed to encode map e282671 with expected crc > 2016-06-01 11:23:03.110117 osd.177 10.31.0.71:6842/10245 442 : cluster [WRN] > failed to encode map e282672 with expected crc > 2016-06-01 11:23:03.255597 osd.177 10.31.0.71:6842/10245 443 : cluster [WRN] > failed to encode map e282672 with expected crc > 2016-06-01 11:23:03.258223 osd.177 10.31.0.71:6842/10245 444 : cluster [WRN] > failed to encode map e282672 with expected crc > 2016-06-01 11:23:05.287753 osd.177 10.31.0.71:6842/10245 445 : cluster [WRN] > failed to encode map e282673 with expected crc > > > How do I clear these up after the upgrade? All of the filesystems on the > OSDs are mounted and the keyrings are there...... > > Thanks, > Jeff > > > -- > > Jeffrey McDonald, PhD > Assistant Director for HPC Operations > Minnesota Supercomputing Institute > University of Minnesota Twin Cities > 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx > 117 Pleasant St SE phone: +1 612 625-6905 > Minneapolis, MN 55455 fax: +1 612 624-8861 > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Email: shinobu@xxxxxxxxx shinobu@xxxxxxxxxx _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |