Hi,
thanks for the reply. I updated one of the new mons. And after a
resonably long init phase (inconsistent state), I am now seeing these:
2014-02-28 01:05:12.344648 7fe9d05cb700 0 cephx: verify_reply coudln't
decrypt with error: error decoding block for decryption
2014-02-28 01:05:12.345599 7fe9d05cb700 0 -- X.Y.Z.207:6789/0 >>
X.Y.Z.201:6789/0 pipe(0x14e1400 sd=21 :49082 s=1 pgs=5421935 cs=12
l=0).failed verifying authorize reply
with .207 being the updated mon and .201 being one of the "old" alive
mons. I guess they don't understand each other? I would rather not try
to update the mons running on servers that also host OSDs, especially
since there seem to be communication issues between those versions... or
am I reading this wrong?
KR,
Marc
On 28/02/2014 01:32, Gregory Farnum wrote:
> On Thu, Feb 27, 2014 at 4:25 PM, Marc <mail@xxxxxxxxxx> wrote:
>> Hi,
>>
>> I was handed a Ceph cluster that had just lost quorum due to 2/3 mons
>> (b,c) running out of disk space (using up 15GB each). We were trying to
>> rescue this cluster without service downtime. As such we freed up some
>> space to keep mon b running a while longer, which succeeded, quorum
>> restored (a,b), mon c remained offline. Even though we have freed up
>> some space on mon c's disk also, that mon just won't start. It's log
>> file does say
>>
>> ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process
>> ceph-mon, pid 27846
>>
>> and thats all she wrote. Even when starting ceph-mon with -d mind you.
>>
>> So we had a cluster with 2/3 mons up and wanted to add another mon since
>> it was only a matter of time til mon b failed again due to disk space.
>>
>> As such I added mon.g to the cluster, which took a long while to sync,
>> but now reports running.
>>
>> Then mon.h got added for the same reason. mon.h fails to start much the
>> same as mon.c does.
>>
>> Still that should leave us with 3/5 mons up. However running "ceph
>> daemon mon.{g,h} mon_status" on the respective node also blocks. The
>> only output we get from those are fault messages.
>>
>> Ok so now mon.g apparantly crashed:
>>
>> 2014-02-28 00:11:48.861263 7f4728042700 -1 mon/Monitor.cc: In function
>> 'void Monitor::sync_timeout(entity_inst_t&)' thread 7f4728042700 time
>> 2014-02-28 00:11:48.782305 mon/Monitor.cc: 1099: FAILED
>> assert(sync_state == SYNC_STATE_CHUNKS)
>>
>> ... and now blocks trying to start much like c and h.
>>
>> Long story short: is it possible to add .61.9 mons to a cluster running
>> .61.2 on the 2 alive mons and all the osds? I'm guessing this is the
>> last shot at trying to rescue the cluster without downtime.
> That should be fine, and is likely (though not guaranteed) to resolve
> your sync issues -- although it's pretty unfortunate that you're that
> far behind on the point releases; they fixed a whole lot of sync
> issues and related things and you might need to upgrade the existing
> monitors too in order to get the fixes you need... :/
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
