| Hello Everyone I have recently upgraded ceph version in our test environment from 0.65 to 0.76 ( latest available release ) Upgrade was successful on all monitor nodes as well as all OSDs After up gradation i am facing a wiered problems with OSD , OSDS are randomly flapping UP and DOWN , after a certain time all OSDs becomes UP and after few minutes 1 or 2 of them went down. Below are the required outputs , please advice some troubleshooting steps. # id weight type name up/down reweight -1 5.65 root default -2 0 host ceph-node1 -3 1.72 host ceph-node2 4 0.43 osd.4 up 1 5 0.43 osd.5 up 1 6 0.43 osd.6 up 1 7 0.43 osd.7 down 1 -4 1.31 host ceph-node4 8 0.88 osd.8 up 1 1 0.43 osd.1 up 1 -5 1.31 host ceph-node5 9 0.88 osd.9 up 1 2 0.43 osd.2 down 1 -6 0.88 host ceph-node6 10 0.88 osd.10 up 1 -7 0.43 host ceph-node3 0 0.43 osd.0 up 1 # id weight type name up/down reweight -1 5.65 root default -2 0 host ceph-node1 -3 1.72 host ceph-node2 4 0.43 osd.4 up 1 5 0.43 osd.5 down 1 6 0.43 osd.6 up 1 7 0.43 osd.7 down 1 -4 1.31 host ceph-node4 8 0.88 osd.8 up 1 1 0.43 osd.1 up 1 -5 1.31 host ceph-node5 9 0.88 osd.9 up 1 2 0.43 osd.2 up 1 -6 0.88 host ceph-node6 10 0.88 osd.10 up 1 -7 0.43 host ceph-node3 0 0.43 osd.0 up 1 # id weight type name up/down reweight -1 5.65 root default -2 0 host ceph-node1 -3 1.72 host ceph-node2 4 0.43 osd.4 up 1 5 0.43 osd.5 down 1 6 0.43 osd.6 up 1 7 0.43 osd.7 down 1 -4 1.31 host ceph-node4 8 0.88 osd.8 up 1 1 0.43 osd.1 up 1 -5 1.31 host ceph-node5 9 0.88 osd.9 up 1 2 0.43 osd.2 down 1 -6 0.88 host ceph-node6 10 0.88 osd.10 up 1 -7 0.43 host ceph-node3 0 0.43 osd.0 up 1 I have tried checking logs of the osd those are getting down , below are the logs OSD.5 2014-01-08 07:57:17.205815 7f5f752c3700 -1 osd.5 249671 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:54.101138 (cutoff 2014-01-08 07:56:57.205790) 2014-01-08 07:57:17.205821 7f5f752c3700 -1 osd.5 249671 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:10.171581 (cutoff 2014-01-08 07:56:57.205790) 2014-01-08 07:57:17.231841 7f5f5b13e700 0 auth: could not find secret_id=2612 2014-01-08 07:57:17.231855 7f5f5b13e700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:57:17.231907 7f5f5b13e700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.46:6801/2009929 pipe(0x1177b980 sd=133 :6805 s=0 pgs=0 cs=0 l=0 c=0x16ae0580).accept: got bad authorizer 2014-01-08 07:57:17.256551 7f5f6963d700 0 auth: could not find secret_id=2612 2014-01-08 07:57:17.256579 7f5f6963d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:57:17.256614 7f5f6963d700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.47:6805/7024358 pipe(0x11779b80 sd=124 :6805 s=0 pgs=0 cs=0 l=0 c=0x16ae6b40).accept: got bad authorizer 2014-01-08 07:57:17.438798 7f5f6963d700 0 auth: could not find secret_id=2612 2014-01-08 07:57:17.438842 7f5f6963d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:57:17.438880 7f5f6963d700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.46:6801/2009929 pipe(0x1177e180 sd=124 :6805 s=0 pgs=0 cs=0 l=0 c=0x16ae1ce0).accept: got bad authorizer 2014-01-08 07:57:18.115240 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.1 ever on either front or back, first ping sent 2014-01-08 07:51:50.143695 (cutoff 2014-01-08 07:56:58.115238) 2014-01-08 07:57:18.115246 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:53:15.968317 (cutoff 2014-01-08 07:56:58.115238) 2014-01-08 07:57:18.115250 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:54.101138 (cutoff 2014-01-08 07:56:58.115238) 2014-01-08 07:57:18.115254 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:10.171581 (cutoff 2014-01-08 07:56:58.115238) 2014-01-08 07:57:18.115293 7f5f6963d700 0 auth: could not find secret_id=2612 2014-01-08 07:57:18.115302 7f5f6963d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:57:18.115307 7f5f6963d700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.46:6801/2009929 pipe(0x1177b480 sd=65 :6805 s=0 pgs=0 cs=0 l=0 c=0x16ae0840).accept: got bad authorizer 2014-01-08 07:57:18.115358 7f5f5b13e700 0 auth: could not find secret_id=2612 2014-01-08 07:57:18.115366 7f5f5b13e700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:57:18.115370 7f5f5b13e700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.47:6805/7024358 pipe(0x1177fa80 sd=146 :6805 s=0 pgs=0 cs=0 l=0 c=0x388e300).accept: got bad authorizer 2014-01-08 07:57:18.427737 7f5f5b13e700 0 auth: could not find secret_id=2612 2014-01-08 07:57:18.427755 7f5f5b13e700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:57:18.427760 7f5f5b13e700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.46:6801/2009929 pipe(0x1177fd00 sd=65 :6805 s=0 pgs=0 cs=0 l=0 c=0x388f4e0).accept: got bad authorizer 2014-01-08 07:57:18.803829 7f5f5b13e700 0 auth: could not find secret_id=2612 2014-01-08 07:57:18.803850 7f5f5b13e700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:57:18.803902 7f5f5b13e700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.46:6801/2009929 pipe(0x11779400 sd=65 :6805 s=0 pgs=0 cs=0 l=0 c=0x388f640).accept: got bad authorizer 2014-01-08 07:57:18.804592 7f5f6963d700 0 auth: could not find secret_id=2613 2014-01-08 07:57:18.804614 7f5f6963d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2613 2014-01-08 07:57:18.804995 7f5f6963d700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.37:6813/22087 pipe(0x11778780 sd=146 :6805 s=0 pgs=0 cs=0 l=0 c=0x388f0c0).accept: got bad authorizer 2014-01-08 07:57:19.416382 7f5f5b13e700 0 auth: could not find secret_id=2613 2014-01-08 07:57:19.416434 7f5f5b13e700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2613 2014-01-08 07:57:19.418569 7f5f5b13e700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.45:6802/15187 pipe(0x1177a080 sd=65 :6805 s=0 pgs=0 cs=0 l=0 c=0x388fe80).accept: got bad authorizer 2014-01-08 07:57:21.616086 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.1 ever on either front or back, first ping sent 2014-01-08 07:51:50.143695 (cutoff 2014-01-08 07:57:01.616084) 2014-01-08 07:57:21.616095 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:53:15.968317 (cutoff 2014-01-08 07:57:01.616084) 2014-01-08 07:57:21.616099 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:54.101138 (cutoff 2014-01-08 07:57:01.616084) 2014-01-08 07:57:21.616104 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:10.171581 (cutoff 2014-01-08 07:57:01.616084) 2014-01-08 07:57:27.370230 7f5f69940700 0 cephx: verify_reply couldn't decrypt with error: error decoding block for decryption 2014-01-08 07:57:27.370263 7f5f69940700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.37:6801/20982 pipe(0x112b01680 sd=30 :44500 s=1 pgs=218 cs=6 l=0 c=0x37582520).failed verifying authorize reply 2014-01-08 07:57:28.857366 7f5f6963d700 0 auth: could not find secret_id=2613 2014-01-08 07:57:28.857429 7f5f6963d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2613 2014-01-08 07:57:28.857448 7f5f5b13e700 0 auth: could not find secret_id=2613 2014-01-08 07:57:28.857474 7f5f5b13e700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2613 2014-01-08 07:57:28.857480 7f5f5b13e700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.37:6813/22087 pipe(0x1177d780 sd=116 :6805 s=0 pgs=0 cs=0 l=0 c=0x388f7a0).accept: got bad authorizer 2014-01-08 07:57:28.857457 7f5f6963d700 0 -- 192.168.1.37:6805/21364 >> 192.168.1.37:6801/20982 pipe(0x1177a800 sd=146 :6805 s=0 pgs=0 cs=0 l=0 c=0x388eb40).accept: got bad authorizer 2014-01-08 07:57:28.858309 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.1 ever on either front or back, first ping sent 2014-01-08 07:51:50.143695 (cutoff 2014-01-08 07:57:08.858308) 2014-01-08 07:57:28.858314 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:53:15.968317 (cutoff 2014-01-08 07:57:08.858308) 2014-01-08 07:57:28.858318 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:54.101138 (cutoff 2014-01-08 07:57:08.858308) 2014-01-08 07:57:28.858323 7f5f5f505700 -1 osd.5 249672 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:10.171581 (cutoff 2014-01-08 07:57:08.858308) OSD.7 2014-01-08 07:43:03.824274 7fd4bc3fc700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:43.823437) 2014-01-08 07:43:03.824299 7fd4bc3fc700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:43.823437) 2014-01-08 07:43:03.908884 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:43.908881) 2014-01-08 07:43:03.908902 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:43.908881) 2014-01-08 07:43:03.908910 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:43.908881) 2014-01-08 07:43:04.909175 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:44.909170) 2014-01-08 07:43:04.909193 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:44.909170) 2014-01-08 07:43:04.909201 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:44.909170) 2014-01-08 07:43:05.909452 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:45.909447) 2014-01-08 07:43:05.909472 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:45.909447) 2014-01-08 07:43:05.909480 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:45.909447) 2014-01-08 07:43:06.909792 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:46.909784) 2014-01-08 07:43:06.909812 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:46.909784) 2014-01-08 07:43:06.909822 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:46.909784) 2014-01-08 07:43:07.910175 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:47.910170) 2014-01-08 07:43:07.910195 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:47.910170) 2014-01-08 07:43:07.910204 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:47.910170) 2014-01-08 07:43:08.110541 7fd4c6ce2700 0 auth: could not find secret_id=2612 2014-01-08 07:43:08.110601 7fd4c6ce2700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:43:08.111521 7fd4c6ce2700 0 -- 192.168.1.37:6809/21737 >> 192.168.1.37:6801/20982 pipe(0xfc14e900 sd=182 :6809 s=0 pgs=0 cs=0 l=0 c=0x188d9fa0).accept: got bad authorizer 2014-01-08 07:43:08.332771 7fd4c6ce2700 0 auth: could not find secret_id=2612 2014-01-08 07:43:08.332804 7fd4c6ce2700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:43:08.332813 7fd4c6ce2700 0 -- 192.168.1.37:6809/21737 >> 192.168.1.37:6813/22087 pipe(0xfc14eb80 sd=182 :6809 s=0 pgs=0 cs=0 l=0 c=0x188dc8e0).accept: got bad authorizer 2014-01-08 07:43:08.525056 7fd4bc3fc700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:48.525054) 2014-01-08 07:43:08.525076 7fd4bc3fc700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:48.525054) 2014-01-08 07:43:08.525085 7fd4bc3fc700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:48.525054) 2014-01-08 07:43:08.959269 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:48.959264) 2014-01-08 07:43:08.959286 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:48.959264) 2014-01-08 07:43:08.959294 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:48.959264) 2014-01-08 07:43:09.146204 7fd4c6ce2700 0 auth: could not find secret_id=2612 2014-01-08 07:43:09.146245 7fd4c6ce2700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:43:09.146256 7fd4c6ce2700 0 -- 192.168.1.37:6809/21737 >> 192.168.1.41:6801/2217 pipe(0xfc14a080 sd=182 :6809 s=0 pgs=0 cs=0 l=0 c=0x188da7e0).accept: got bad authorizer 2014-01-08 07:43:09.959513 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:49.959506) 2014-01-08 07:43:09.959532 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:49.959506) 2014-01-08 07:43:09.959540 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:49.959506) 2014-01-08 07:43:10.959902 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:50.959896) 2014-01-08 07:43:10.959929 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:50.959896) 2014-01-08 07:43:10.959938 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:50.959896) 2014-01-08 07:43:11.960308 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:51.960294) 2014-01-08 07:43:11.960351 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:51.960294) 2014-01-08 07:43:11.960360 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:51.960294) 2014-01-08 07:43:12.572678 7fd4c6ce2700 0 auth: could not find secret_id=2612 2014-01-08 07:43:12.572762 7fd4c6ce2700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2612 2014-01-08 07:43:12.572784 7fd4c6ce2700 0 -- 192.168.1.37:6809/21737 >> 192.168.1.45:6810/7008666 pipe(0xfc14b980 sd=182 :6809 s=0 pgs=0 cs=0 l=0 c=0x188dd540).accept: got bad authorizer 2014-01-08 07:43:13.167514 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:53.167505) 2014-01-08 07:43:13.167550 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:53.167505) 2014-01-08 07:43:13.167559 7fd4d1bd5700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:53.167505) 2014-01-08 07:43:13.225528 7fd4bc3fc700 -1 osd.7 249643 heartbeat_check: no reply from osd.8 ever on either front or back, first ping sent 2014-01-08 07:40:31.405962 (cutoff 2014-01-08 07:42:53.225526) 2014-01-08 07:43:13.225538 7fd4bc3fc700 -1 osd.7 249643 heartbeat_check: no reply from osd.9 ever on either front or back, first ping sent 2014-01-08 06:52:53.963591 (cutoff 2014-01-08 07:42:53.225526) 2014-01-08 07:43:13.225543 7fd4bc3fc700 -1 osd.7 249643 heartbeat_check: no reply from osd.10 ever on either front or back, first ping sent 2014-01-08 07:41:55.631466 (cutoff 2014-01-08 07:42:53.225526) [root@ceph-node2 ceph]# ceph status cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a health HEALTH_WARN 6 pgs degraded; 262 pgs down; 564 pgs peering; 14 pgs stale; 564 pgs stuck inactive; 12 pgs stuck stale; 570 pgs stuck unclean; recovery 10/49174 objects degraded (0.020%); mds cluster is degraded; mds ceph-mon1 is laggy; 2/10 in osds are down; crush map has non-optimal tunables monmap e3: 3 mons at {ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0}, election epoch 4218, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3 mdsmap e8461: 1/1/1 up {0=ceph-mon1=up:replay(laggy or crashed)} osdmap e249808: 10 osds: 8 up, 10 in pgmap v583944: 576 pgs, 6 pools, 98505 MB data, 24587 objects 342 GB used, 5424 GB / 5767 GB avail 10/49174 objects degraded (0.020%) 6 active+clean 184 peering 223 down+peering 6 active+degraded 12 stale+peering 2 stale+down+peering 106 remapped+peering 37 down+remapped+peering [root@ceph-node2 ceph]# |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
