I was upgrading from 0.58 to ceph version 0.59
+(cbae6a435c62899f857775f66659de052fb0e759)
Upgrading from 0.57 to 0.58 was an easy one, so I was suprised with the problems
It seems to me, that I make an fatal error, that I dont understand.
I had 5 working mons (mon.{0-4]). After the upgrade of the first node I
lost the mon.4 with the cephx error. Then I upgraded all of the nodes and
I lost the mon.0 with the starting error.
After some restarts it looks like the other mons lost any quorum
so ceph -s or any kind of ceph commands didn't work anymore.
So I made today the decision to reinstall the test "cluster".
-Steffen
Btw. ceph rbd, adding/removing osds works great.
On Fri, Mar 22, 2013 at 10:01:10AM +0000, Joao Eduardo Luis wrote:
> On 03/21/2013 03:47 PM, Steffen Thorhauer wrote:
> >I think, I was impatient and should wait for the v.59 announcement. It
> >seems I should upgrading all monitors.
> > After upgrading all nodes I have on 2 monitors errors like:
> >=== mon.0 ===
> >Starting Ceph mon.0 on u124-161-ceph...
> >mon fs missing 'monmap/latest' and 'mkfs/monmap'
> >failed: 'ulimit -n 8192; /usr/bin/ceph-mon -i 0 --pid-file
> >/var/run/ceph/mon.0.pid -c /etc/ceph/ceph.conf '
> >
> >Steffen
>
> Which version are you upgrading from?
>
> Also, could you provide us with some logs of those monitors with 'debug
> mon = 20' ?
>
> -Joao
>
> >
> >
> >On 03/21/2013 02:22 PM, Steffen Thorhauer wrote:
> >>Hi,
> >>I just upgraded one node of my ceph "cluster". I wanted upgrade node
> >>after node.
> >>osd on this node has no problem. but the mon (mon.4) has
> >>authorization problems.
> >>I did'nt change any config, just made an apt-get upgrade .
> >>ceph -s
> >> health HEALTH_WARN 1 mons down, quorum 0,1,2,3 0,1,2,3
> >> monmap e2: 5 mons at
> >>{0=10.37.124.161:6789/0,1=10.37.124.162:6789/0,2=10.37.124.163:6789/0,3=10.37.124.164:6789/0,4=10.37.124.167:6789/0},
> >>election epoch 162, quorum 0,1,2,3 0,1,2,3
> >> osdmap e4839: 16 osds: 16 up, 16 in
> >> pgmap v195213: 3144 pgs: 3144 active+clean; 255 GB data, 820 GB
> >>used, 778 GB / 1599 GB avail
> >> mdsmap e54723: 1/1/1 up {0=0=up:active}, 3 up:standby
> >>
> >>
> >>but the mon.4 log file look like:
> >>
> >>2013-03-21 12:45:15.701747 7f45412c6780 2 mon.4@-1(probing) e2 init
> >>2013-03-21 12:45:15.702051 7f45412c6780 10 mon.4@-1(probing) e2 bootstrap
> >>2013-03-21 12:45:15.702094 7f45412c6780 10 mon.4@-1(probing) e2
> >>unregister_cluster_logger - not registered
> >>2013-03-21 12:45:15.702121 7f45412c6780 10 mon.4@-1(probing) e2
> >>cancel_probe_timeout (none scheduled)
> >>2013-03-21 12:45:15.702147 7f45412c6780 0 mon.4@-1(probing) e2 my
> >>rank is now 4 (was -1)
> >>2013-03-21 12:45:15.702190 7f45412c6780 10 mon.4@4(probing) e2 reset_sync
> >>2013-03-21 12:45:15.702213 7f45412c6780 10 mon.4@4(probing) e2 reset
> >>2013-03-21 12:45:15.702238 7f45412c6780 10 mon.4@4(probing) e2
> >>timecheck_finish
> >>2013-03-21 12:45:15.702286 7f45412c6780 10 mon.4@4(probing) e2
> >>cancel_probe_timeout (none scheduled)
> >>2013-03-21 12:45:15.702312 7f45412c6780 10 mon.4@4(probing) e2
> >>reset_probe_timeout 0x24d6580 after 2 seconds
> >>2013-03-21 12:45:15.702387 7f45412c6780 10 mon.4@4(probing) e2 probing
> >>other monitors
> >>2013-03-21 12:45:15.703459 7f453a15f700 10 mon.4@4(probing) e2
> >>ms_get_authorizer for mon
> >>2013-03-21 12:45:15.703641 7f453a15f700 10 cephx: build_service_ticket
> >>service mon secret_id 18446744073709551615 ticket_info.ticket.name=mon.
> >>2013-03-21 12:45:15.703642 7f453a361700 10 mon.4@4(probing) e2
> >>ms_get_authorizer for mon
> >>2013-03-21 12:45:15.703694 7f453a361700 10 cephx: build_service_ticket
> >>service mon secret_id 18446744073709551615 ticket_info.ticket.name=mon.
> >>2013-03-21 12:45:15.703869 7f453a260700 10 mon.4@4(probing) e2
> >>ms_get_authorizer for mon
> >>2013-03-21 12:45:15.703957 7f453a260700 10 cephx: build_service_ticket
> >>service mon secret_id 18446744073709551615 ticket_info.ticket.name=mon.
> >>2013-03-21 12:45:15.704244 7f453a05e700 10 mon.4@4(probing) e2
> >>ms_get_authorizer for mon
> >>2013-03-21 12:45:15.704306 7f453a05e700 10 cephx: build_service_ticket
> >>service mon secret_id 18446744073709551615 ticket_info.ticket.name=mon.
> >>2013-03-21 12:45:15.704323 7f453a361700 0 cephx: verify_reply
> >>coudln't decrypt with error: error decoding block for decryption
> >>2013-03-21 12:45:15.704333 7f453a361700 0 -- 10.37.124.167:6789/0 >>
> >>10.37.124.161:6789/0 pipe(0x24f3c80 sd=29 :42310 s=1 pgs=0 cs=0
> >>l=0).failed verifying authorize reply
> >>2013-03-21 12:45:15.704404 7f453a361700 0 -- 10.37.124.167:6789/0 >>
> >>10.37.124.161:6789/0 pipe(0x24f3c80 sd=29 :42310 s=1 pgs=0 cs=0
> >>l=0).fault
> >>2013-03-21 12:45:15.704429 7f453a15f700 0 cephx: verify_reply
> >>coudln't decrypt with error: error decoding block for decryption
> >>2013-03-21 12:45:15.704483 7f453a15f700 0 -- 10.37.124.167:6789/0 >>
> >>10.37.124.163:6789/0 pipe(0x24f3500 sd=31 :60255 s=1 pgs=0 cs=0
> >>l=0).failed verifying authorize reply
> >>2013-03-21 12:45:15.704517 7f453a260700 0 cephx: verify_reply
> >>coudln't decrypt with error: error decoding block for decryption
> >>2013-03-21 12:45:15.704578 7f453a15f700 0 -- 10.37.124.167:6789/0 >>
> >>10.37.124.163:6789/0 pipe(0x24f3500 sd=31 :60255 s=1 pgs=0 cs=0
> >>l=0).fault
> >>2013-03-21 12:45:15.704529 7f453a260700 0 -- 10.37.124.167:6789/0 >>
> >>10.37.124.162:6789/0 pipe(0x24f3a00 sd=30 :55445 s=1 pgs=0 cs=0
> >>l=0).failed verifying authorize reply
> >>
> >>What now??
> >>
> >>Regards,
> >> Steffen
> >>
> >>
> >>_______________________________________________
> >>ceph-users mailing list
> >>ceph-users@xxxxxxxxxxxxxx
> >>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >_______________________________________________
> >ceph-users mailing list
> >ceph-users@xxxxxxxxxxxxxx
> >http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
______________________________________________________________________
Steffen Thorhauer
email: sth@xxxxxxxxxxxxxxxxxxxxxxx
url: http://wwwiti.cs.uni-magdeburg.de/~thorhaue
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
