On 23/08/2013, at 8:53 PM, Allan Latham <alatham@xxxxxxxxxxxxxxxx> wrote:
> Hi Honza
>
> It may of course be a pacemaker problem. I have no idea where to look
> for clues -:)
>
>> What version of pacemaker you are using? Are you using cpg based
>> one, or plugin based one
>
> root@v100 /root # dpkg -l|grep pacemaker
> ii pacemaker 1.1.7-1 amd64 HA cluster resource manager
>
> I have no idea if it's cpg or plugin. We like to stick with whatever
> comes with the latest Debian unless there is some very good reason not
> to - simply to keep future maintenance to a minimum.
It depends on your configuration, not necessarily how pacemaker was built.
If you have a block like:
service {
name: pacemaker
ver:....
}
Then its a plugin based cluster.
>
> The Debian expermental/unstable version is 1.1.10+ but has 9 pages of
> dependencies - I really don't want to go there.
That sounds... strange. The upstream dependancy list didn't really change in that timeframe.
>
> Any ideas how I can get pacemaker diagnostics
man crm_report :)
> - the corosync log seems
> to contain some items which I would think come from pacemaker. Is there
> anything else to look at? Is there a 'debug' level?
Yes. We read it from corosync (man corosync.conf)
>
> All the best and many many thanks for all the help.
>
> Allan
>
>
> On 23/08/13 11:08, Jan Friesse wrote:
>> Allan,
>> sorry for late reply, but there were more urgent things.
>>
>> Anyway,
>>> I wonder also if 5 servers in a ring with wan links is simply too
>> many. Do the possible problems increase by N**2?
>>
>> If you were using 20 sec timeout it should just work. I mean, token
>> timeout should be set to sum of rtt between nodes in ring + token hold
>> timeout * no_nodes. It really shouldn't be N**2.
>>
>> Drop-in replacement is always in same X.Y series, so 1.4.x it's 1.4.6,
>> same for 2.3.x it's 2.3.1. When Y is changing, everything works well in
>> most cases but there may be bigger change (like 1.3.x -> 1.4.x). When X
>> is changing (so 1.x.y -> 2.x.y) there are HUGE changes. For example
>> running 2.3.1 and 1.4.6 on same net (same broadcast addr, port, ...)
>> will simply not work.
>>
>>
>> Allan Latham napsal(a):
>>> Hi Honza
>>>
>>> I decided for 1.4.6 as I already had the .deb packages I made for the
>>> ipv6 tests.
>>>
>>> All works well except that when I kill the server running as DC which is
>>> also running the active HA servers.
>>>
>>> The setup is two real servers running proxmox each with one KVM virutal
>>> machine.
>>>
>>> Each VM runs corosync 1.4.6 and a virtual server from Hetzner runs the
>>> same thus giving me a 3 node corosync/pacemaker cluster. The VS from
>>> Hetzner never runs any real service - it's just for quorum purposes. The
>>> two KVMs run ip changeover, drbd and mysql.
>>>
>>> I have done many tests with controlled handover - i.e. 'reboot' - all
>>> works very well.
>>>
>>> Most time when I kill a node this works too - i.e. 'stop' on proxmox or
>>> the Hetzner control panel.
>>>
>>> I have a long token time (20 seconds) to allow for short network outages
>>> so that we don't reconfigure for every little glitch.
>>>
>>> Not fully reproducible but twice now I have killed the server which was
>>> both DC and had the active services running and hit a problem. Most
>>> times it works.
>>>
>>> The problem is that the remaining two nodes do not see the killed node
>>> go offline (crm status shows everything online and quorum DC on the dead
>>> node). Nothing works anymore e.g. crm resource cleanup xyz just hangs.
>>>
>>
>> Are you 100% sure it's corosync problem? I mean, can't it be pacemaker
>> one? What version of pacemaker you are using? Are you using cpg based
>> one, or plugin based one (this is known to be very problematic, that's
>> why Andrew implemented cpg based one)?
>>
>>> The corosync log however shows the old DC disappearing and the new DC
>>> being negotiated correctly (to my eyes at least). But this doesn't
>>> appear to have any effect.
>>>
>>> A final part of the bug is that corosync refuses to shutdown during the
>>> reboot process - only a hard reboot works.
>>>
>>
>> Corosync (1.x) provides functionality to register shutdown callback and
>> if application which uses this functionality refuses shutdown, corosync
>> will refuse shutdown. Maybe this is the case.
>>
>>> This is very similar to problems I've seen on live systems before we
>>> stopped using wan links. I would love to get a fix for this as it
>>> completely kills HA. When we are in this state nothing works until the
>>> ops hard reboot all nodes.
>>>
>>> Question: what exactly do you need from me the next time this happens?
>>>
>>
>> - Please send me corosync.log from all nodes
>> - corosync-objctl -a from all nodes (this will allow us to find out,
>> what membership corosync sees).
>> - From nodes which cannot be restarted result of corosync-blackbox
>>
>> Regards,
>> Honza
>>
>>> All the best
>>>
>>> Allan
>>>
>>>
>>> On 21/08/13 14:20, Allan Latham wrote:
>>>> Hi Honza
>>>>
>>>> I'd like to compile the latest (and best) version which could work as a
>>>> drop in replacement for what is in Debian Wheezy:
>>>>
>>>> root@h89 /root # corosync -v
>>>> Corosync Cluster Engine, version '1.4.2'
>>>> Copyright (c) 2006-2009 Red Hat, Inc.
>>>>
>>>> root@h89 /root # dpkg -l |grep corosync
>>>> ii corosync 1.4.2-3
>>>> amd64 Standards-based cluster framework (daemon and modules)
>>>>
>>>> Which version do you recommend?
>>>>
>>>> Or is there a compatible .deb somewhere?
>>>>
>>>> In particular the bug with pointopoint and udpu is biting me!
>>>>
>>>> All the best
>>>>
>>>> Allan
>>>>
>>>>
>>>
>>>
>>
>>
>
> _______________________________________________
> discuss mailing list
> discuss@xxxxxxxxxxxx
> http://lists.corosync.org/mailman/listinfo/discuss
Attachment:
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ discuss mailing list discuss@xxxxxxxxxxxx http://lists.corosync.org/mailman/listinfo/discuss
