Re: Ceph Monitor cephx issues

[Shinobu Kinjo <skinjo@xxxxxxxxxx>]

Good to know.

Sorry for such an inconvenient for you anyway.

Regards,

On Sun, Jan 8, 2017 at 2:19 PM, Alex Evonosky <alex.evonosky@xxxxxxxxx> wrote:

Since this was a test lab, I totally purged the whole cluster and re-deployed..  working good now, thank you.

Alex F. Evonosky

On Sat, Jan 7, 2017 at 9:14 PM, Alex Evonosky <alex.evonosky@xxxxxxxxx> wrote:

Thank you.. 

After sending the post, I totally removed the mon and issued the build with ceph-deploy:

In the logs now:

2017-01-07 21:12:38.113534 7fa9613fd700  0 cephx: verify_reply couldn't decrypt with error: error decoding block for decryption

2017-01-07 21:12:38.113546 7fa9613fd700  0 -- 10.10.10.138:6789/0 >> 10.10.10.103:6789/0 pipe(0x55feb2e90000 sd=12 :50266 s=1 pgs=0 cs=0 l=0 c=0x55feb2ca0a80).failed verifying authorize reply

2017-01-07 21:12:38.114529 7fa95787b700  0 cephx: verify_reply couldn't decrypt with error: error decoding block for decryption

2017-01-07 21:12:38.114567 7fa95787b700  0 -- 10.10.10.138:6789/0 >> 10.10.10.252:6789/0 pipe(0x55feb2e91400 sd=11 :38690 s=1 pgs=0 cs=0 l=0 c=0x55feb2ca0c00).failed verifying authorize reply

2017-01-07 21:12:40.114522 7fa9613fd700  0 cephx: verify_reply couldn't decrypt with error: error decoding block for decryption

2017-01-07 21:12:40.114542 7fa9613fd700  0 -- 10.10.10.138:6789/0 >> 10.10.10.103:6789/0 pipe(0x55feb2e90000 sd=11 :50278 s=1 pgs=0 cs=0 l=0 c=0x55feb2ca0a80).failed verifying authorize reply

2017-01-07 21:12:40.115706 7fa95787b700  0 cephx: verify_reply couldn't decrypt with error: error decoding block for decryption

2017-01-07 21:12:40.115721 7fa95787b700  0 -- 10.10.10.138:6789/0 >> 10.10.10.252:6789/0 pipe(0x55feb2e91400 sd=12 :38702 s=1 pgs=0 cs=0 l=0 c=0x55feb2ca0c00).failed verifying authorize reply

2017-01-07 21:12:41.621916 7fa956f79700  0 cephx: verify_authorizer could not decrypt ticket info: error: NSS AES final round failed: -8190

2017-01-07 21:12:41.621929 7fa956f79700  0 mon.alex-desktop@1(probing) e0 ms_verify_authorizer bad authorizer from mon 10.10.10.103:6789/0

2017-01-07 21:12:41.621944 7fa956f79700  0 -- 10.10.10.138:6789/0 >> 10.10.10.103:6789/0 pipe(0x55feb2fb5400 sd=21 :6789 s=0 pgs=0 cs=0 l=0 c=0x55feb2ca1500).accept: got bad authorizer

$ sudo ceph -s

    cluster f5aba719-4856-4ae2-a5d4-f9ff0f614b60

     health HEALTH_WARN

            512 pgs degraded

            348 pgs stale

            512 pgs stuck unclean

            512 pgs undersized

            6 requests are blocked > 32 sec

            recovery 25013/50026 objects degraded (50.000%)

            mds cluster is degraded

            1 mons down, quorum 0,2 alpha,toshiba-laptop

     monmap e17: 3 mons at {alex-desktop=10.10.10.138:6789/0,alpha=10.10.10.103:6789/0,toshiba-laptop=10.10.10.252:6789/0}

            election epoch 806, quorum 0,2 alpha,toshiba-laptop

      fsmap e201858: 1/1/1 up {0=1=up:replay}

     osdmap e200229: 3 osds: 2 up, 2 in; 85 remapped pgs

            flags sortbitwise

      pgmap v4088774: 512 pgs, 4 pools, 50883 MB data, 25013 objects

            59662 MB used, 476 GB / 563 GB avail

            25013/50026 objects degraded (50.000%)

                 348 stale+active+undersized+degraded

                 164 active+undersized+degraded

root@alex-desktop:/var/lib/ceph/mon/ceph-alex-desktop# ls -ls

total 8

0 -rw-r--r-- 1 ceph ceph    0 Jan  7 21:11 done

4 -rw------- 1 ceph ceph   77 Jan  7 21:05 keyring

4 drwxr-xr-x 2 ceph ceph 4096 Jan  7 21:10 store.db

0 -rw-r--r-- 1 ceph ceph    0 Jan  7 21:05 systemd

Very odd...  never seen this issue on the other monitor deployments...

Alex F. Evonosky

On Sat, Jan 7, 2017 at 8:54 PM, Shinobu Kinjo <skinjo@xxxxxxxxxx> wrote:

Using ``ceph-deploy`` will save your life:

 # https://github.com/ceph/ceph/blob/master/doc/start/quick-ceph-deploy.rst
  * Please look at: Adding Monitors

If you are using centos or similar, the latest package is available here:

 # http://download.ceph.com/rpm-jewel/el7/noarch/ceph-deploy-1.5.37-0.noarch.rpm

Regards,

On Sun, Jan 8, 2017 at 9:53 AM, Alex Evonosky <alex.evonosky@xxxxxxxxx> wrote:

Thank you for the reply!

I followed this article:

http://docs.ceph.com/docs/jewel/rados/operations/add-or-rm-mons/

Under the section: ADDING A MONITOR (MANUAL)

Alex F. Evonosky

On Sat, Jan 7, 2017 at 6:36 PM, Shinobu Kinjo <skinjo@xxxxxxxxxx> wrote:

How did you add a third MON?

Regards,

On Sun, Jan 8, 2017 at 7:01 AM, Alex Evonosky <alex.evonosky@xxxxxxxxx> wrote:
> Anyone see this before?
>
>
> 2017-01-07 16:55:11.406047 7f095b379700  0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2017-01-07 16:55:11.406053 7f095b379700  0 -- 10.10.10.138:6789/0 >>
> 10.10.10.252:6789/0 pipe(0x55cf8d028000 sd=11 :47548 s=1 pgs=0 cs=0 l=0
> c=0x55cf8ce28f00).failed verifying authorize reply
>
>
>
> Two monitors are up just fine, just trying to add a third and a quorum
> cannot be met.  NTP is running and no iptables running at all on internal
> cluster.
>
>
> Thank you.
> -Alex
>
>
>
>

> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com