On 01/10/2012 11:47 PM, Andrew Beekhof wrote:
> On Tue, Jan 10, 2012 at 9:08 PM, Fabio M. Di Nitto <fdinitto@xxxxxxxxxx> wrote:
>> Hi all,
>>
>> in some recent discussions, it come up the issue on how to configure
>> quorum module. As I don´t really have a complete solution yet, I need to
>> seek advice in the community :)
>>
>> Problem:
>>
>> it would be very nice if corosync.conf could be simply scp´ed/copied
>> between nodes and everything works as expected on all nodes.
>> Issue being that some quorum bits are, at this point in time, node
>> specific. It means that to alter some values, it is necessary to edit
>> corosync.conf on the specific node.
>> On top of that, it would be nice if expected_votes could be
>> automatically calculated based on votes: values.
>>
>> The current quorum configuration (based on topic-quorum patches):
>>
>> quorum {
>> provider: corosync_votequorum
>> expected_votes: 8
>> votes: 1
>> two_node: 0
>> wait_for_all: 0
>> last_man_standing: 0
>> auto_tie_breaker: 0
>> }
>>
>> totem {
>> nodeid: xxx
>> }
>>
>> The 2 values that cannot be copied around are quorum.votes and totem.nodeid.
>>
>> In current votequorum/totem incarnation, votes/expected_votes/nodeid are
>> all broadcasted to all nodes. so each node that joins the cluster
>> becomes aware of the other peers values.
>>
>> As a consequence of the current config format, auto_tie_breaker feature,
>> requires wait_for_all to work (in order to have the complete list of
>> nodeids, see auto_tie_breaker implementation in topic-quorum branch for
>> details).
>>
>> Honza and I quickly explored options to add those values into the node
>> list of udpu, but that´s limiting because it doesn´t work well in
>> multicast and/or broadcast and it has integration issues with RRP.
>>
>> Also adding lists to quorum {} involves a certain level of duplicated
>> information.
>>
>> For example:
>>
>> quorum {
>> nodeid_list: x y z...
>> node.x.votes: ..
>> node.y.votes: ..
>> }
>>
>> that IMHO is all but nice to look at.
>>
>> So the question of changing the config format also raise the following
>> questions:
>>
>> 1) do we really need to support an auto_tie_breaker feature without
>> wait_for_all? if NO, then we don´t need the list of nodeids upfront.
>>
>> 2) do we really care about votes other than 1?
>
> That was also my question when reading the above.
> It always struck me as troublesome to get right, just giving one of 4
> nodes an extra vote (for example) will still give you a tie under the
> wrong conditions.
>
> Seems (to me) like a habit people got into when clusters went to
> pieces without quorum and that we have "better" solutions today (like
> the token registry).
> So my vote is drop it.
That was my take too in the beginning but apparently there are some use
cases that require votes != 1.
>
>> If NO, then votes: can
>> simply be dropped from corosync.conf defaults, and in case an override
>> is necessary, it can be done specific to the node. This solution poses
>> the problem that expected_votes need to be set in corosync.conf (one
>> liner in the config file vs different liners) but it might be slightly
>> more tricky to calculate if votes are not balanced.
>
> Any chance the value could be incremented based on the number of nodes
> ever seen?
> Ie. if count(active peers) > expected votes, update the config file.
expected_votes is already calculated that way. If you configure 8 but
all of a sudden you see 9 nodes, then expected_votes is incremented.
The above is true also if one node starts voting differently (1 -> X)
then expected_votes is updated across the cluster automagically.
Writing to file is unnecessary operation with votequorum current
incarnation.
>
> That way most people could simply ignore the setting until they wanted
> to remove a node.
Not that simple no.
There are several cases where expected_votes is required to be known
upfront specially when handling partitions and startups.
Let say you have 8 nodes cluster. quorum expected to be 5.
Switch between 4 nodes and 4 nodes is dead or mulfunctioning. By using
an incremental expected_votes, you can effectively start 2 clusters.
Both clusters would be quorate, with expected_votes set to 4 and quorum
to 3. No guarantee those will merge. I doubt we want this situation to
ever exists.
also, it would break the wait_for_all feature (or WFA would need to
require expected_votes .. either way).
Fabio
_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss
