If you're running ntpd, then I believe your clocks were too skewed for the
authentication to work. Once ntpd got the clocks syncing, authentication
would start working again.
You can query ntpd for how skewed the clock is relative to the NTP servers:
clewis at ceph2:~$ sudo ntpq -p
remote refid st t when poll reach delay offset
jitter
==============================================================================
+192.168.0.1 132.239.1.6 2 u 519 1024 377 0.290 0.106 0.315
*192.168.0.2 132.239.1.6 2 u 847 1024 377 0.410 -0.732 0.096
LOCAL(0) .LOCL. 10 l 60d 64 0 0.000 0.000
0.000
Offset is the number of milli-seconds that your local clock differs from
the ntp server. Repeat on all our monitors.
A difference of more than 50ms between the monitors will cause the clock
skew warnings. I'm not sure how far out it needs to be to cause
authentication problems.
On Thu, Aug 28, 2014 at 3:00 AM, yuelongguang <fastsync at 163.com> wrote:
>
> the next day it returnes to normal.
> i have no idea.
>
>
>
>
> At 2014-08-27 00:38:29, "Michael" <michael at onlinefusion.co.uk> wrote:
>
> How far out are your clocks? It's showing a clock skew, if they're too far
> out it can cause issues with cephx.
> Otherwise you're probably going to need to check your cephx auth keys.
>
> -Michael
>
> On 26/08/2014 12:26, yuelongguang wrote:
>
> hi,all
>
> i have 5 osds and 3 mons. its status is ok then.
>
> to be mentioned , this cluster has no any data. i just deploy it and to
> be familar with some command lines.
> what is the probpem and how to fix?
>
> thanks
>
>
> ---environment-----
> ceph-release-1-0.el6.noarch
> ceph-deploy-1.5.11-0.noarch
> ceph-0.81.0-5.el6.x86_64
> ceph-libs-0.81.0-5.el6.x86_64
> ---------ceph -s ----------------------
> [root at cephosd1-mona ~]# ceph -s
> cluster 508634f6-20c9-43bb-bc6f-b777f4bb1651
> health HEALTH_WARN 183 pgs peering; 183 pgs stuck inactive; 183 pgs
> stuck unclean; clock skew detected on mon.cephosd2-monb, mon.cephosd3-monc
> monmap e13: 3 mons at {cephosd1-mona=
> 10.154.249.3:6789/0,cephosd2-monb=10.154.249.4:6789/0,cephosd3-monc=10.154.249.5:6789/0},
> election epoch 74, quorum 0,1,2 cephosd1-mona,cephosd2-monb,cephosd3-monc
> osdmap e151: 5 osds: 5 up, 5 in
> pgmap v499: 384 pgs, 4 pools, 0 bytes data, 0 objects
> 201 MB used, 102143 MB / 102344 MB avail
> 167 peering
> 201 active+clean
> 16 remapped+peering
>
>
> --------------log--------------------------osd.0------------
> 2014-08-26 19:16:13.926345 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:16:13.926355 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x4dc2a80 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d5960).accept: got bad authorizer
> 2014-08-26 19:16:28.928023 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:16:28.928050 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x4dc2800 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d56a0).accept: got bad authorizer
> 2014-08-26 19:16:28.929139 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:16:28.929237 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38071 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:16:43.930846 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:16:43.930899 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x4dc2580 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d0b00).accept: got bad authorizer
> 2014-08-26 19:16:43.932204 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:16:43.932230 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38073 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:16:58.933526 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:16:58.935094 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x4dc2300 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d0840).accept: got bad authorizer
> 2014-08-26 19:16:58.936239 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:16:58.936261 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38074 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:17:13.937335 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:17:13.937368 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x4dc2080 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d1b80).accept: got bad authorizer
> 2014-08-26 19:17:13.937923 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:17:13.937933 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38075 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:17:28.939439 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:17:28.939455 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x4dc1e00 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d5540).accept: got bad authorizer
> 2014-08-26 19:17:28.939716 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:17:28.939731 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38076 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:17:43.941398 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:17:43.941426 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x4dc1b80 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d0dc0).accept: got bad authorizer
> 2014-08-26 19:17:43.941539 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:17:43.941553 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38077 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:17:58.944113 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:17:58.944257 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x4dc0c80 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x3c2dd80).accept: got bad authorizer
> 2014-08-26 19:17:58.944450 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:17:58.944466 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38078 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:18:13.946363 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:18:13.946388 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x4dc1180 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d6b40).accept: got bad authorizer
> 2014-08-26 19:18:13.946509 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:18:13.946518 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38079 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:18:28.948562 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:18:28.948602 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x4dc1400 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d69e0).accept: got bad authorizer
> 2014-08-26 19:18:28.949006 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:18:28.949042 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38080 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:18:43.950637 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:18:43.950727 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edbe80 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d6880).accept: got bad authorizer
> 2014-08-26 19:18:43.952035 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:18:43.952048 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38082 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:18:58.953732 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:18:58.954129 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3ede900 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d6720).accept: got bad authorizer
> 2014-08-26 19:18:58.954967 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:18:58.954981 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38083 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:19:13.956551 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:19:13.956692 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edf300 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d1a20).accept: got bad authorizer
> 2014-08-26 19:19:13.956929 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:19:13.956942 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38084 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:19:28.958546 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:19:28.958584 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3eddf00 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d5ac0).accept: got bad authorizer
> 2014-08-26 19:19:28.959598 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:19:28.959623 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38085 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
> 2014-08-26 19:19:43.960489 7f114a8d2700 0 cephx: verify_authorizer could
> not decrypt ticket info: error: decryptor.MessageEnd::Exception:
> StreamTransformationFilter: invalid PKCS #7 block padding found
> 2014-08-26 19:19:43.960516 7f114a8d2700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edad00 sd=25 :6800 s=0 pgs=0 cs=0 l=0
> c=0x45d2c00).accept: got bad authorizer
> 2014-08-26 19:19:43.961823 7f114c009700 0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2014-08-26 19:19:43.961851 7f114c009700 0 -- 11.154.249.2:6800/1667 >>
> 11.154.249.7:6800/1599 pipe(0x3edb700 sd=24 :38086 s=1 pgs=0 cs=0 l=0
> c=0x45d23c0).failed verifying authorize reply
>
>
>
>
> _______________________________________________
> ceph-users mailing listceph-users at lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140903/15337a04/attachment.htm>
