Hi,
--
I just performed a minor ceph upgrade on my ubuntu 14.04 cluster from ceph version to0.94.6-1trusty to 0.94.7-1trusty. Upon restarting the OSDs, I receive the error message:
2016-06-01 12:17:49.219512 7f64a70ea8c0 0 monclient: wait_auth_rotating timed out after 30
2016-06-01 12:17:49.219605 7f64a70ea8c0 -1 osd.177 282877 unable to obtain rotating service keys; retrying
2016-06-01 12:18:19.219740 7f64a70ea8c0 0 monclient: wait_auth_rotating timed out after 30
2016-06-01 12:18:19.219782 7f64a70ea8c0 -1 osd.177 282877 unable to obtain rotating service keys; retrying
2016-06-01 12:18:49.219869 7f64a70ea8c0 0 monclient: wait_auth_rotating timed out after 30
2016-06-01 12:18:49.219908 7f64a70ea8c0 -1 osd.177 282877 unable to obtain rotating service keys; retrying
eventually the OSD fails to start. Oddly, this affects only some of the OSDs on this host.
On the mon host, there is a warning:
2016-06-01 11:22:52.171152 osd.177 10.31.0.71:6842/10245 433 : cluster [WRN] failed to encode map e282667 with expected crc
2016-06-01 11:22:55.399487 osd.177 10.31.0.71:6842/10245 434 : cluster [WRN] failed to encode map e282668 with expected crc
2016-06-01 11:22:55.572238 osd.177 10.31.0.71:6842/10245 435 : cluster [WRN] failed to encode map e282668 with expected crc
2016-06-01 11:22:59.571924 osd.177 10.31.0.71:6842/10245 436 : cluster [WRN] failed to encode map e282669 with expected crc
2016-06-01 11:22:59.665925 osd.177 10.31.0.71:6842/10245 437 : cluster [WRN] failed to encode map e282669 with expected crc
2016-06-01 11:22:59.671475 osd.177 10.31.0.71:6842/10245 438 : cluster [WRN] failed to encode map e282669 with expected crc
2016-06-01 11:22:59.780646 osd.177 10.31.0.71:6842/10245 439 : cluster [WRN] failed to encode map e282669 with expected crc
2016-06-01 11:23:00.778250 osd.177 10.31.0.71:6842/10245 440 : cluster [WRN] failed to encode map e282670 with expected crc
2016-06-01 11:23:02.075996 osd.177 10.31.0.71:6842/10245 441 : cluster [WRN] failed to encode map e282671 with expected crc
2016-06-01 11:23:03.110117 osd.177 10.31.0.71:6842/10245 442 : cluster [WRN] failed to encode map e282672 with expected crc
2016-06-01 11:23:03.255597 osd.177 10.31.0.71:6842/10245 443 : cluster [WRN] failed to encode map e282672 with expected crc
2016-06-01 11:23:03.258223 osd.177 10.31.0.71:6842/10245 444 : cluster [WRN] failed to encode map e282672 with expected crc
2016-06-01 11:23:05.287753 osd.177 10.31.0.71:6842/10245 445 : cluster [WRN] failed to encode map e282673 with expected crc
How do I clear these up after the upgrade? All of the filesystems on the OSDs are mounted and the keyrings are there......
Thanks,
Jeff
Jeffrey McDonald, PhD Assistant Director for HPC Operations Minnesota Supercomputing Institute University of Minnesota Twin Cities 599 Walter Library email: jeffrey.mcdonald@xxxxxxxxxxx 117 Pleasant St SE phone: +1 612 625-6905 Minneapolis, MN 55455 fax: +1 612 624-8861
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
