Re: ceph-mgr fails to restart after upgrade to mimic

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Thanks to everyone who has tried to help so far. I have filed a bug report on this issue at http://tracker.ceph.com/issues/37835. I hope we can get this fixed so I can finish this upgrade.

On Fri, Jan 4, 2019 at 7:26 AM Randall Smith <rbsmith@xxxxxxxxx> wrote:
Greetings,

I'm upgrading my cluster from luminous to mimic. I've upgraded my monitors and am attempting to upgrade the mgrs. Unfortunately, after an upgrade the mgr daemon exits immediately with error code 1.

I've tried running ceph-mgr in debug mode to try to see what's happening but the output (below) is a bit cryptic for me. It looks like authentication might be failing but it was working prior to the upgrade.

I do have "auth supported = cephx" in the global section of ceph.conf.

What do I need to do to fix this?

Thanks.

/usr/bin/ceph-mgr -f --cluster ceph --id 8 --setuser ceph --setgroup ceph -d --debug_ms 5                                                                                          
2019-01-04 07:01:38.457 7f808f83f700  2 Event(0x30c42c0 nevent=5000 time_id=1).set_owner idx=0 owner=140190140331776                                                                                 
2019-01-04 07:01:38.457 7f808f03e700  2 Event(0x30c4500 nevent=5000 time_id=1).set_owner idx=1 owner=140190131939072                                                                                 
2019-01-04 07:01:38.457 7f808e83d700  2 Event(0x30c4e00 nevent=5000 time_id=1).set_owner idx=2 owner=140190123546368                                                                                 
2019-01-04 07:01:38.457 7f809dd5b380  1  Processor -- start                                                                                                                                          
2019-01-04 07:01:38.477 7f809dd5b380  1 -- - start start                                                                                                                                             
2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.147:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6780 con 0                                                                      
2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.148:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6a00 con 0
2019-01-04 07:01:38.481 7f808e83d700  1 -- 192.168.253.148:0/1359135487 learned_addr learned my addr 192.168.253.148:0/1359135487
2019-01-04 07:01:38.481 7f808e83d700  2 -- 192.168.253.148:0/1359135487 >> 192.168.253.148:6789/0 conn(0x332d500 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_a$
ked_seq 0 vs out_seq 0
2019-01-04 07:01:38.481 7f808f03e700  2 -- 192.168.253.148:0/1359135487 >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_a$
ked_seq 0 vs out_seq 0
2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487 >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 seq
1 0x30c5440 mon_map magic: 0 v1
2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487 >> 192.168.253.148:6789/0 conn(0x332d500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2 seq
1 0x30c5680 mon_map magic: 0 v1
2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487 >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 seq
2 0x32a6780 auth_reply(proto 2 0 (0) Success) v1
2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487 >> 192.168.253.148:6789/0 conn(0x332d500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2 seq
2 0x32a6a00 auth_reply(proto 2 0 (0) Success) v1
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== mon.1 192.168.253.147:6789/0 1 ==== mon_map magic: 0 v1 ==== 370+0+0 (3034216899 0 0) 0x30c5440 con 0x332ce00
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== mon.2 192.168.253.148:6789/0 1 ==== mon_map magic: 0 v1 ==== 370+0+0 (3034216899 0 0) 0x30c5680 con 0x332d500
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== mon.1 192.168.253.147:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3430158761 0 0) 0x32a6780 con 0x33$
ce00
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 --> 192.168.253.147:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x32a6f00 con 0
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== mon.2 192.168.253.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3242503871 0 0) 0x32a6a00 con 0x33$
d500
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 --> 192.168.253.148:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x32a6780 con 0
2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487 >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 seq
3 0x32a6f00 auth_reply(proto 2 -22 (22) Invalid argument) v1
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== mon.1 192.168.253.147:6789/0 3 ==== auth_reply(proto 2 -22 (22) Invalid argument) v1 ==== 24+0+0 (882932531 0 0) 0x32a6f$
0 con 0x332ce00
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_OPEN pgs=74172 cs=1 l=1).mark_down
2019-01-04 07:01:38.481 7f808e03c700  2 -- 192.168.253.148:0/1359135487 >> 192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_OPEN pgs=74172 cs=1 l=1)._stop
2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487 >> 192.168.253.148:6789/0 conn(0x332d500 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2 seq
3 0x32a6780 auth_reply(proto 2 -22 (22) Invalid argument) v1
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== mon.2 192.168.253.148:6789/0 3 ==== auth_reply(proto 2 -22 (22) Invalid argument) v1 ==== 24+0+0 (1359424806 0 0) 0x32a6$
80 con 0x332d500
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 >> 192.168.253.148:6789/0 conn(0x332d500 :-1 s=STATE_OPEN pgs=74275 cs=1 l=1).mark_down
2019-01-04 07:01:38.481 7f808e03c700  2 -- 192.168.253.148:0/1359135487 >> 192.168.253.148:6789/0 conn(0x332d500 :-1 s=STATE_OPEN pgs=74275 cs=1 l=1)._stop

2019-01-04 07:01:38.481 7f809dd5b380  1 -- 192.168.253.148:0/1359135487 shutdown_connections
2019-01-04 07:01:38.481 7f809dd5b380  5 -- 192.168.253.148:0/1359135487 shutdown_connections mark down 192.168.253.148:6789/0 0x332d500
2019-01-04 07:01:38.481 7f809dd5b380  5 -- 192.168.253.148:0/1359135487 shutdown_connections mark down 192.168.253.147:6789/0 0x332ce00
2019-01-04 07:01:38.481 7f809dd5b380  5 -- 192.168.253.148:0/1359135487 shutdown_connections delete 0x332ce00
2019-01-04 07:01:38.481 7f809dd5b380  5 -- 192.168.253.148:0/1359135487 shutdown_connections delete 0x332d500
2019-01-04 07:01:38.485 7f809dd5b380  1 -- 192.168.253.148:0/1359135487 shutdown_connections
2019-01-04 07:01:38.485 7f809dd5b380  1 -- 192.168.253.148:0/1359135487 wait complete.
2019-01-04 07:01:38.485 7f809dd5b380  1 -- 192.168.253.148:0/1359135487 >> 192.168.253.148:0/1359135487 conn(0x332c000 :-1 s=STATE_NONE pgs=0 cs=0 l=0).mark_down
2019-01-04 07:01:38.485 7f809dd5b380  2 -- 192.168.253.148:0/1359135487 >> 192.168.253.148:0/1359135487 conn(0x332c000 :-1 s=STATE_NONE pgs=0 cs=0 l=0)._stop
failed to fetch mon config (--no-mon-config to skip)

--
Randall Smith
Computing Services
Adams State University
http://www.adams.edu/
719-587-7741


--
Randall Smith
Computing Services
Adams State University
http://www.adams.edu/
719-587-7741
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux