On Fri, Jan 26, 2018 at 12:14 PM Cary <dynamic.cary@xxxxxxxxx> wrote:
Hello,
We are running Luminous 12.2.2. 6 OSD hosts with 12 1TB OSDs, and 64GB
RAM. Each host has a SSD for Bluestore's block.wal and block.db.
There are 5 monitor nodes as well with 32GB RAM. All servers have
Gentoo with kernel, 4.12.12-gentoo.
When I export an image using:
rbd export pool-name/volume-name /location/image-name.raw
Message similar to below are displayed. The signature check fails
randomly. And sometimes a message about a bad authorizer, but not
everytime.
The image is still exported successfully.
2018-01-24 17:35:15.616080 7fc8d4024700 0 cephx:
verify_authorizer_reply bad nonce got 4552544084014661633 expected
4552499520046621785 sent 4552499520046621784
2018-01-24 17:35:15.616098 7fc8d4024700 0 --
172.21.32.16:0/1412094654 >> 172.21.32.6:6802/6219 conn(0x7fc8b0078a50
:-1 s=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=0 cs=0
l=1)._process_connection failed verifying authorize reply
2018-01-24 17:35:15.699004 7fc8d4024700 0 SIGN: MSG 2 Message
signature does not match contents.
2018-01-24 17:35:15.699020 7fc8d4024700 0 SIGN: MSG 2Signature on message:
2018-01-24 17:35:15.699021 7fc8d4024700 0 SIGN: MSG 2 sig:
8189090775647585001
2018-01-24 17:35:15.699047 7fc8d4024700 0 SIGN: MSG 2Locally
calculated signature:
2018-01-24 17:35:15.699048 7fc8d4024700 0 SIGN: MSG 2
sig_check:140500325643792
2018-01-24 17:35:15.699049 7fc8d4024700 0 Signature failed.
2018-01-24 17:35:15.699050 7fc8d4024700 0 --
172.21.32.16:0/1412094654 >> 172.21.32.2:6807/153106
conn(0x7fc8bc020870 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH
pgs=26018 cs=1 l=1).process Signature check failed
Does anyone know what could cause this, and what I can do to fix it.
That's in the cephx authentication code and it's indicating that the secure signature sent with the message isn't what the local node thinks it should be. That's pretty odd (a bit flip or something that could actually change it ought to trigger the messaging checksums directly) and I'm not quite sure how it could happen.
But, as you've noticed, it retries and apparently succeeds. How did you notice this?
-Greg
Thank you,
Cary
-Dynamic
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
