On 02/23/2012 04:30 PM, Christine Caulfield wrote:
> ACK with some changes as discussed on IRC
Thanks for the review, I am going to implement the qdevice_register
locking directly and that will save some 255 bytes on the network,
plus simplify some of the code that we were discussing about on IRC.
I'll try to send a patch by Monday.
Fabio
>
> Reviewed-By: Christine Caulfield <ccaulfie@xxxxxxxxxx>
>
> On 23/02/12 11:04, Fabio M. Di Nitto wrote:
>> From: "Fabio M. Di Nitto"<fdinitto@xxxxxxxxxx>
>>
>> qdevice is a very special node in the cluster and it adds a certain
>> amount of complexity and special cases across the code.
>>
>> most of the qdevice data are shared across the cluster (name/votes)
>> but effectively each node has a different view of the qdevice
>> (registered/unregistered/voting/etc.)
>>
>> with this change, we align the qdevice view across the node,
>> exchanging more data between nodes and we fix how qdevice behaves
>> and it is configured.
>>
>> The only side effect is that the amount of data transmitted on wire
>> is slightly higher.
>>
>> The qdevice API is still disabled by default. This means that
>> the amount of real changes in current code are a lot smaller
>> than it appears by this patch.
>>
>> TODO: documentation/man pages needs to be updated once
>> this change is in (and behavior finalized).
>>
>> User visible changes:
>>
>> - configuration (coroparse, exec/votequorum):
>> the quorum device section is now standalone within the quorum.
>>
>> quorum {
>> provider: corosync_votequorum
>> device {
>> model: (name)
>> timeout: (millisec)
>> votes:
>> }
>> }
>>
>> the keyword "model:" is mandatory to enable qdevice in configuration
>> and should express the name of the script/daemon that will provide
>> the qdevice. Looking into the future, an init script or systemd
>> service will look for that name in /path/to/be/decided/name
>> and start/stop qdevice.
>>
>> timeout: defines the maximum interval the qdevice implementation
>> has available between poll (see votequorum_qdevice_poll.3) before
>> the device is considered dead and votes discarded
>>
>> votes: is now a configuration parameter and not an API call.
>> quorum devices don't care what they need to vote.
>> votes is autocalculated when a nodelist is available and all
>> nodes in the list vote 1. Otherwise this parameter is mandatory.
>>
>> - configuration (exec/votequorum):
>> startup and runtime configuration changes have been improved.
>> errors at startup are considered fatal. errors at runtime
>> have different exit paths.
>>
>> startup:
>>
>> * quorum.two_node and qdevice are incompatible.
>> * quorum.expected_votes requires quorum.device.votes.
>> * quorum.expected_votes - quorum.device.votes cannot be lower
>> than 2.
>> * qdevice and last_man_standing are mutually exclusive.
>> * qdevice and auto_tie_breaker are mutually exclusive.
>>
>> runtime config changes:
>>
>> * quorum.two_node and qdevice are incompatible:
>> if quorum device is alive, two_node is disabled.
>> if quorum device is not alive and node count is 2, two_node is
>> enabled, and quorum device cannot be registered
>>
>> * if either last_man_standing or auto_tie_breaker were enabled
>> at startup, and at runtime quorum device is configured,
>> quorum device registration will be blocked.
>>
>> * if quorum.expected_votes is configured but not quorum.device.votes,
>> quorum device registration will be blocked.
>>
>> * if quorum.device.votes is not configured and we cannot
>> automatically calculate it, quorum device registration will be
>> blocked.
>>
>> * An error in configuring quorum.expected_votes and
>> quorum.device.votes
>> will block quorum device registration.
>>
>> blocking quorum device registation, also means dropping the votes.
>>
>> quorum.device.votes (either set or automatically calculated) is now
>> used to determine current expected_votes in the cluster.
>>
>> - logging (exec/votequorum):
>>
>> all errors from configuration are treated as WARNING/CRITICAL.
>>
>> lots of extra DEBUG output is added (see internal changes too).
>>
>> - corosync-quorumtool (tools/corosync-quorumtool):
>>
>> * added option to forcefully kick out a quorum device from the local
>> node. This is for emergency recovery only and it is only
>> available when qdevice API is built-in.
>>
>> * Improved status output, specifically add node state and qdevice
>> information
>>
>> [root@fedora-master-node2 coro]# corosync-quorumtool -s
>> Version: 1.99.4.12-9c7d-dirty
>> Quorum type: corosync_votequorum
>> Nodes: 2
>> Ring ID: 132
>> Quorate: Yes
>> Node votes: 1
>> Node state: Member
>> Expected votes: 3
>> Highest expected: 3
>> Total votes: 3
>> Quorum: 2
>> Flags: Quorate Qdevice
>> Nodeid Votes Name
>> 1 1 fedora-master-node1.int.fabbione.net
>> 2 1 fedora-master-node2.int.fabbione.net
>> 0 1 QDEVICE (Voting)
>>
>> * allow to print status for any node in the cluster known to
>> local node.
>>
>> [root@fedora-master-node1 coro]# corosync-quorumtool -s
>> Version: 1.99.4.12-9c7d-dirty
>> Quorum type: corosync_votequorum
>> Nodes: 2
>> Ring ID: 144
>> Quorate: Yes
>> Node votes: 1
>> Node state: Member
>> Expected votes: 3
>> Highest expected: 3
>> Total votes: 2
>> Quorum: 2
>> Flags: Quorate
>> Nodeid Votes Name
>> 1 1 fedora-master-node1.int.fabbione.net
>> 2 1 fedora-master-node2.int.fabbione.net
>>
>> [root@fedora-master-node1 coro]# corosync-quorumtool -s -n 2
>> Version: 1.99.4.12-9c7d-dirty
>> Quorum type: corosync_votequorum
>> Nodes: 2
>> Ring ID: 144
>> Quorate: Yes
>> Node votes: 1
>> Node state: Member
>> Expected votes: 3
>> Highest expected: 3
>> Total votes: 3
>> Quorum: 2
>> Flags: Quorate Qdevice
>> Nodeid Votes Name
>> 1 1 fedora-master-node1.int.fabbione.net
>> 2 1 fedora-master-node2.int.fabbione.net
>> 0 1 QDEVICE (Voting)
>>
>> Internal changes:
>>
>> - change qdevice to not run all time, but only when necessary.
>> - change votequorum_nodeinfo on wire data to include
>> qdevice name and use flags instead of uint8_t.
>> - allocate nodeid 0 to qdevice since it's the only real
>> nodeid that be reserved.
>> - change send_nodeinfo to allow to send nodeinfo for any node
>> so that we can share qdevice info across the cluster
>> (and this might be useful in future if we need to sync
>> internal cluster view).
>> - add votequorum api call to update qdevice name
>> - add runtime data if quorum device has been forcefully disabled
>> by config error
>> - add qdevice votes to expected_votes calculation (this
>> is probably the biggest difference vs cman)
>> - change votequorum_read_nodelist_configuration so that
>> we can autocalculate votes for qdevice (we need the nodecount
>> vs votes).
>> - add all checks for startup/runtime config (see above).
>> - maintain qdevice information for each node. This is necessary
>> because the qdevice cluster_node is local only and
>> lay down the road to improve user error configuration in future
>> (for example we will be able to block 2 different qdevice model
>> to be used in the cluster at the same time).
>> - do not make qdevice part of the membership_list received from
>> totem. None of our users care about it and it is not a real node.
>> - change onwire message handlers to deal with "data for this node from
>> any node"
>> case and undersand nodeid 0 for qdevice info
>> - always allocate qdevice at startup. this simplifies code a lot.
>> - dispatch qdevice nodeinfo on membership changes.
>> - inform libvotequorum users when a qdevice is registered
>> - improve substantially qdevice api and add a simple (for now)
>> barrier based on qdevice name. Plan is that only qdevice
>> with the same name can register across the cluster.
>> - qdevice getinfo can now report status for qdevice on any node.
>> - change slightly the way the qdevice API is built-in/out:
>> only the libvotequorum calls are #ifdef'out now. Doing so in
>> the core is too complex and would make the code unreadable
>> with the risk of missing a bit or two effectively introducing
>> an on-wire incompatibility if we will ever turn the API on.
>> - probably added some bugs on the way...
>>
>> TODO: update qdevice_* API once the above is settled and test
>> qdevice integration with other features.
>>
>> Signed-off-by: Fabio M. Di Nitto<fdinitto@xxxxxxxxxx>
>> ---
>> exec/coroparse.c | 17 +-
>> exec/votequorum.c | 728
>> +++++++++++++++++++++++++++----------
>> include/corosync/ipc_votequorum.h | 31 ++-
>> include/corosync/votequorum.h | 26 +-
>> lib/votequorum.c | 46 ++-
>> test/testvotequorum2.c | 23 +-
>> tools/corosync-quorumtool.c | 133 ++++++-
>> 7 files changed, 766 insertions(+), 238 deletions(-)
>>
>> diff --git a/exec/coroparse.c b/exec/coroparse.c
>> index e6a5cb8..1d97323 100644
>> --- a/exec/coroparse.c
>> +++ b/exec/coroparse.c
>> @@ -87,6 +87,7 @@ enum main_cp_cb_data_state {
>> MAIN_CP_CB_DATA_STATE_LOGGING_DAEMON,
>> MAIN_CP_CB_DATA_STATE_MEMBER,
>> MAIN_CP_CB_DATA_STATE_QUORUM,
>> + MAIN_CP_CB_DATA_STATE_QDEVICE,
>> MAIN_CP_CB_DATA_STATE_NODELIST,
>> MAIN_CP_CB_DATA_STATE_NODELIST_NODE,
>> };
>> @@ -388,7 +389,6 @@ static int main_config_parser_cb(const char *path,
>> case MAIN_CP_CB_DATA_STATE_QUORUM:
>> if ((strcmp(path, "quorum.expected_votes") == 0) ||
>> (strcmp(path, "quorum.votes") == 0) ||
>> - (strcmp(path, "quorum.quorumdev_poll") == 0) ||
>> (strcmp(path, "quorum.last_man_standing_window") ==
>> 0) ||
>> (strcmp(path, "quorum.leaving_timeout") == 0)) {
>> if (safe_atoi(value,&i) != 0) {
>> @@ -410,6 +410,15 @@ static int main_config_parser_cb(const char *path,
>> add_as_string = 0;
>> }
>> break;
>> + case MAIN_CP_CB_DATA_STATE_QDEVICE:
>> + if ((strcmp(path, "quorum.device.timeout") == 0) ||
>> + (strcmp(path, "quorum.device.votes") == 0)) {
>> + if (safe_atoi(value,&i) != 0) {
>> + goto atoi_error;
>> + }
>> + icmap_set_uint32(path, i);
>> + add_as_string = 0;
>> + }
>> case MAIN_CP_CB_DATA_STATE_TOTEM:
>> if ((strcmp(path, "totem.version") == 0) ||
>> (strcmp(path, "totem.nodeid") == 0) ||
>> @@ -671,6 +680,9 @@ static int main_config_parser_cb(const char *path,
>> if (strcmp(path, "quorum") == 0) {
>> data->state = MAIN_CP_CB_DATA_STATE_QUORUM;
>> }
>> + if (strcmp(path, "quorum.device") == 0) {
>> + data->state = MAIN_CP_CB_DATA_STATE_QDEVICE;
>> + }
>> if (strcmp(path, "nodelist") == 0) {
>> data->state = MAIN_CP_CB_DATA_STATE_NODELIST;
>> data->node_number = 0;
>> @@ -858,6 +870,9 @@ static int main_config_parser_cb(const char *path,
>> case MAIN_CP_CB_DATA_STATE_QUORUM:
>> data->state = MAIN_CP_CB_DATA_STATE_NORMAL;
>> break;
>> + case MAIN_CP_CB_DATA_STATE_QDEVICE:
>> + data->state = MAIN_CP_CB_DATA_STATE_QUORUM;
>> + break;
>> case MAIN_CP_CB_DATA_STATE_NODELIST:
>> data->state = MAIN_CP_CB_DATA_STATE_NORMAL;
>> break;
>> diff --git a/exec/votequorum.c b/exec/votequorum.c
>> index f7d708f..83f4488 100644
>> --- a/exec/votequorum.c
>> +++ b/exec/votequorum.c
>> @@ -39,8 +39,6 @@
>> #include<stdint.h>
>>
>> #include<qb/qbipc_common.h>
>> -#include<qb/qbdefs.h>
>> -#include<qb/qbutil.h>
>>
>> #include "quorum.h"
>> #include<corosync/corodefs.h>
>> @@ -64,12 +62,12 @@ static struct corosync_api_v1 *corosync_api;
>> * votequorum global config vars
>> */
>>
>> -#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> -#define DEFAULT_QDEV_POLL 10000
>> +#define DEFAULT_QDEVICE_TIMEOUT 10000
>>
>> -static unsigned int quorumdev_poll = DEFAULT_QDEV_POLL;
>> -static char quorum_device_name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
>> -#endif
>> +static struct cluster_node *qdevice = NULL;
>> +static unsigned int qdevice_timeout = DEFAULT_QDEVICE_TIMEOUT;
>> +static uint8_t qdevice_can_operate = 1;
>> +static uint8_t qdevice_is_registered = 0;
>>
>> static uint8_t two_node = 0;
>>
>> @@ -85,21 +83,18 @@ static uint32_t last_man_standing_window =
>> DEFAULT_LMS_WIN;
>>
>> static uint8_t leave_remove = 0;
>> static uint32_t ev_barrier = 0;
>> +
>> /*
>> * votequorum_exec defines/structs/forward definitions
>> */
>>
>> struct req_exec_quorum_nodeinfo {
>> - struct qb_ipc_request_header header __attribute__((aligned(8)));
>> + struct qb_ipc_request_header header __attribute__((aligned(8)));
>> + uint32_t nodeid;
>> uint32_t votes;
>> uint32_t expected_votes;
>> - uint16_t flags;
>> - uint8_t quorate;
>> - uint8_t wait_for_all_status;
>> - uint8_t first_trans;
>> - uint8_t _pad0;
>> - uint8_t _pad1;
>> - uint8_t _pad2;
>> + uint32_t flags;
>> + char qdevice_name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
>> } __attribute__((packed));
>>
>> struct req_exec_quorum_reconfigure {
>> @@ -137,10 +132,14 @@ static int
>> votequorum_exec_send_reconfigure(uint8_t param, unsigned int nodeid,
>> * votequorum internal node status/view
>> */
>>
>> -#define NODE_FLAGS_LEAVING 1
>> +#define NODE_FLAGS_QUORATE 1
>> +#define NODE_FLAGS_LEAVING 2
>> +#define NODE_FLAGS_WFASTATUS 4
>> +#define NODE_FLAGS_FIRST 8
>> +#define NODE_FLAGS_QDEVICE 16
>> +#define NODE_FLAGS_QDEVICE_STATE 32
>>
>> -#define NODEID_US 0
>> -#define NODEID_QDEVICE UINT32_MAX
>> +#define NODEID_QDEVICE 0
>>
>> typedef enum {
>> NODESTATE_MEMBER=1,
>> @@ -149,13 +148,13 @@ typedef enum {
>> } nodestate_t;
>>
>> struct cluster_node {
>> - int node_id;
>> + int node_id;
>> nodestate_t state;
>> - uint32_t votes;
>> - uint32_t expected_votes;
>> - uint16_t flags;
>> - unsigned long long int last_hello; /* Only used for quorum
>> devices */
>> - struct list_head list;
>> + uint32_t votes;
>> + uint32_t expected_votes;
>> + uint32_t flags;
>> + char qdevice_name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
>> + struct list_head list;
>> };
>>
>> /*
>> @@ -164,14 +163,12 @@ struct cluster_node {
>>
>> static uint8_t quorum;
>> static uint8_t cluster_is_quorate;
>> -static uint8_t first_trans = 1;
>>
>> /*
>> * votequorum membership data
>> */
>>
>> static struct cluster_node *us;
>> -static struct cluster_node *quorum_device = NULL;
>> static struct list_head cluster_members_list;
>> static unsigned int quorum_members[PROCESSOR_COUNT_MAX+1];
>> static int quorum_members_entries = 0;
>> @@ -194,9 +191,8 @@ static struct list_head trackers_list;
>> * votequorum timers
>> */
>>
>> -#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> -static corosync_timer_handle_t quorum_device_timer;
>> -#endif
>> +static corosync_timer_handle_t qdevice_timer;
>> +static int qdevice_timer_set = 0;
>> static corosync_timer_handle_t last_man_standing_timer;
>> static int last_man_standing_timer_set = 0;
>>
>> @@ -219,6 +215,7 @@ static quorum_set_quorate_fn_t quorum_callback;
>>
>> static char *votequorum_exec_init_fn (struct corosync_api_v1 *api);
>> static int votequorum_exec_exit_fn (void);
>> +static int votequorum_exec_send_nodeinfo(uint32_t nodeid);
>>
>> static void message_handler_req_exec_votequorum_nodeinfo (
>> const void *message,
>> @@ -265,19 +262,20 @@ static void
>> message_handler_req_lib_votequorum_trackstart (void *conn,
>> static void message_handler_req_lib_votequorum_trackstop (void *conn,
>> const void *message);
>>
>> -#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> static void message_handler_req_lib_votequorum_qdevice_register
>> (void *conn,
>> - const void *message);
>> + const void *message);
>>
>> static void message_handler_req_lib_votequorum_qdevice_unregister
>> (void *conn,
>> - const void *message);
>> + const void *message);
>> +
>> +static void message_handler_req_lib_votequorum_qdevice_update (void
>> *conn,
>> + const void *message);
>>
>> static void message_handler_req_lib_votequorum_qdevice_poll (void
>> *conn,
>> - const void *message);
>> + const void *message);
>>
>> static void message_handler_req_lib_votequorum_qdevice_getinfo (void
>> *conn,
>> - const void *message);
>> -#endif
>> + const void *message);
>>
>> static struct corosync_lib_handler quorum_lib_service[] =
>> {
>> @@ -300,7 +298,6 @@ static struct corosync_lib_handler
>> quorum_lib_service[] =
>> { /* 4 */
>> .lib_handler_fn =
>> message_handler_req_lib_votequorum_trackstop,
>> .flow_control = COROSYNC_LIB_FLOW_CONTROL_NOT_REQUIRED
>> -#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> },
>> { /* 5 */
>> .lib_handler_fn =
>> message_handler_req_lib_votequorum_qdevice_register,
>> @@ -311,13 +308,16 @@ static struct corosync_lib_handler
>> quorum_lib_service[] =
>> .flow_control = COROSYNC_LIB_FLOW_CONTROL_NOT_REQUIRED
>> },
>> { /* 7 */
>> - .lib_handler_fn =
>> message_handler_req_lib_votequorum_qdevice_poll,
>> + .lib_handler_fn =
>> message_handler_req_lib_votequorum_qdevice_update,
>> .flow_control = COROSYNC_LIB_FLOW_CONTROL_NOT_REQUIRED
>> },
>> { /* 8 */
>> + .lib_handler_fn =
>> message_handler_req_lib_votequorum_qdevice_poll,
>> + .flow_control = COROSYNC_LIB_FLOW_CONTROL_NOT_REQUIRED
>> + },
>> + { /* 9 */
>> .lib_handler_fn =
>> message_handler_req_lib_votequorum_qdevice_getinfo,
>> .flow_control = COROSYNC_LIB_FLOW_CONTROL_NOT_REQUIRED
>> -#endif
>> }
>> };
>>
>> @@ -399,7 +399,7 @@ static struct cluster_node *allocate_node(unsigned
>> int nodeid)
>> if (cl) {
>> memset(cl, 0, sizeof(struct cluster_node));
>> cl->node_id = nodeid;
>> - if (nodeid) {
>> + if (nodeid != NODEID_QDEVICE) {
>> node_add_ordered(cl);
>> }
>> }
>> @@ -416,14 +416,14 @@ static struct cluster_node
>> *find_node_by_nodeid(unsigned int nodeid)
>>
>> ENTER();
>>
>> - if (nodeid == NODEID_US) {
>> + if (nodeid == us->node_id) {
>> LEAVE();
>> return us;
>> }
>>
>> if (nodeid == NODEID_QDEVICE) {
>> LEAVE();
>> - return quorum_device;
>> + return qdevice;
>> }
>>
>> list_iterate(tmp,&cluster_members_list) {
>> @@ -483,6 +483,11 @@ static int check_low_node_id_partition(void)
>> static void update_wait_for_all_status(uint8_t wfa_status)
>> {
>> wait_for_all_status = wfa_status;
>> + if (wait_for_all_status) {
>> + us->flags |= NODE_FLAGS_WFASTATUS;
>> + } else {
>> + us->flags&= ~NODE_FLAGS_WFASTATUS;
>> + }
>> icmap_set_uint8("runtime.votequorum.wait_for_all_status",
>> wait_for_all_status);
>> }
>> @@ -498,6 +503,12 @@ static void update_ev_barrier(uint32_t
>> expected_votes)
>> icmap_set_uint32("runtime.votequorum.ev_barrier", ev_barrier);
>> }
>>
>> +static void update_qdevice_can_operate(uint8_t status)
>> +{
>> + qdevice_can_operate = status;
>> + icmap_set_uint8("runtime.votequorum.qdevice_can_operate",
>> qdevice_can_operate);
>> +}
>> +
>> /*
>> * quorum calculation core bits
>> */
>> @@ -534,8 +545,13 @@ static int calculate_quorum(int allow_decrease,
>> unsigned int max_expected, unsig
>> }
>> }
>>
>> - if (quorum_device&& quorum_device->state == NODESTATE_MEMBER) {
>> - total_votes += quorum_device->votes;
>> + if (qdevice_is_registered) {
>> + log_printf(LOGSYS_LEVEL_DEBUG, "node %u state=%d, votes=%u",
>> + qdevice->node_id, qdevice->state, qdevice->votes);
>> + if (qdevice->state == NODESTATE_MEMBER) {
>> + total_votes += qdevice->votes;
>> + total_nodes++;
>> + }
>> }
>>
>> if (max_expected> 0) {
>> @@ -623,6 +639,11 @@ static void are_we_quorate(unsigned int total_votes)
>> }
>>
>> cluster_is_quorate = quorate;
>> + if (cluster_is_quorate) {
>> + us->flags |= NODE_FLAGS_QUORATE;
>> + } else {
>> + us->flags&= ~NODE_FLAGS_QUORATE;
>> + }
>>
>> if (wait_for_all) {
>> if (quorate) {
>> @@ -640,7 +661,9 @@ static void are_we_quorate(unsigned int total_votes)
>> LEAVE();
>> }
>>
>> -/* Recalculate cluster quorum, set quorate and notify changes */
>> +/*
>> + * Recalculate cluster quorum, set quorate and notify changes
>> + */
>> static void recalculate_quorum(int allow_decrease, int
>> by_current_nodes)
>> {
>> unsigned int total_votes = 0;
>> @@ -660,6 +683,13 @@ static void recalculate_quorum(int
>> allow_decrease, int by_current_nodes)
>> }
>> }
>>
>> + if (qdevice->votes) {
>> + total_votes += qdevice->votes;
>> + if (by_current_nodes) {
>> + cluster_members++;
>> + }
>> + }
>> +
>> /*
>> * Keep expected_votes at the highest number of votes in the
>> cluster
>> */
>> @@ -682,12 +712,14 @@ static void recalculate_quorum(int
>> allow_decrease, int by_current_nodes)
>> */
>>
>> static int votequorum_read_nodelist_configuration(uint32_t *votes,
>> + uint32_t *nodes,
>> uint32_t *expected_votes)
>> {
>> icmap_iter_t iter;
>> const char *iter_key;
>> char tmp_key[ICMAP_KEYNAME_MAXLEN];
>> uint32_t our_pos, node_pos;
>> + uint32_t nodecount = 0;
>> uint32_t nodelist_expected_votes = 0;
>> uint32_t node_votes = 0;
>> int res = 0;
>> @@ -697,7 +729,7 @@ static int
>> votequorum_read_nodelist_configuration(uint32_t *votes,
>> if (icmap_get_uint32("nodelist.local_node_pos",&our_pos) !=
>> CS_OK) {
>> log_printf(LOGSYS_LEVEL_DEBUG,
>> "No nodelist defined or our node is not in the
>> nodelist");
>> - return -1;
>> + return 0;
>> }
>>
>> iter = icmap_iter_init("nodelist.node.");
>> @@ -713,6 +745,8 @@ static int
>> votequorum_read_nodelist_configuration(uint32_t *votes,
>> continue;
>> }
>>
>> + nodecount++;
>> +
>> snprintf(tmp_key, ICMAP_KEYNAME_MAXLEN,
>> "nodelist.node.%u.quorum_votes", node_pos);
>> if (icmap_get_uint32(tmp_key,&node_votes) != CS_OK) {
>> node_votes = 1;
>> @@ -726,30 +760,103 @@ static int
>> votequorum_read_nodelist_configuration(uint32_t *votes,
>> }
>>
>> *expected_votes = nodelist_expected_votes;
>> + *nodes = nodecount;
>>
>> icmap_iter_finalize(iter);
>>
>> LEAVE();
>>
>> + return 1;
>> +}
>> +
>> +static int votequorum_qdevice_is_configured(uint32_t *qdevice_votes)
>> +{
>> + char *qdevice_model = NULL;
>> +
>> + ENTER();
>> +
>> + if ((icmap_get_string("quorum.device.model",&qdevice_model) ==
>> CS_OK)&&
>> + (strlen(qdevice_model))) {
>> + free(qdevice_model);
>> + if (icmap_get_uint32("quorum.device.votes", qdevice_votes) !=
>> CS_OK) {
>> + *qdevice_votes = -1;
>> + }
>> + if
>> (icmap_get_uint32("quorum.device.timeout",&qdevice_timeout) != CS_OK) {
>> + qdevice_timeout = DEFAULT_QDEVICE_TIMEOUT;
>> + }
>> + update_qdevice_can_operate(1);
>> + return 1;
>> + }
>> +
>> + LEAVE();
>> +
>> return 0;
>> }
>>
>> /*
>> * votequorum_readconfig_static is executed before
>> * votequorum_readconfig_dynamic
>> + *
>> + * errors from _static are fatal
>> + * errors from _dynamic need to be handled at runtime without
>> self-destruction
>> + *
>> + * TODO: static/dynamic shares a lot of checks _and_
>> + * there are probably more options we can change at runtime
>> + * than we allow now. Review this in 2.1
>> */
>>
>> static char *votequorum_readconfig_static(void)
>> {
>> - uint32_t node_votes, node_expected_votes, expected_votes;
>> + uint32_t node_votes = 0, qdevice_votes = 0;
>> + uint32_t node_expected_votes = 0, expected_votes = 0;
>> + uint32_t node_count = 0;
>> + int have_nodelist, have_qdevice;
>>
>> ENTER();
>>
>> log_printf(LOGSYS_LEVEL_DEBUG, "Reading static configuration");
>>
>> + /*
>> + * gather basic data here
>> + */
>> + icmap_get_uint32("quorum.expected_votes",&expected_votes);
>> + have_nodelist =
>> votequorum_read_nodelist_configuration(&node_votes,&node_count,&node_expected_votes);
>>
>> + have_qdevice = votequorum_qdevice_is_configured(&qdevice_votes);
>> icmap_get_uint8("quorum.two_node",&two_node);
>> update_two_node();
>>
>> + /*
>> + * do basic config verification
>> + */
>> + if ((!have_nodelist)&& (!expected_votes)) {
>> + return ((char *)"configuration error: nodelist or
>> quorum.expected_votes must be configured!");
>> + }
>> +
>> + if ((two_node)&& (have_qdevice)) {
>> + return ((char *)"configuration error: two_node and quorum
>> device cannot be configured at the same time!");
>> + }
>> +
>> + if ((expected_votes)&& (have_qdevice)&& (qdevice_votes == -1)) {
>> + return ((char *)"configuration error: quorum.device.votes
>> must be specified when quorum.expected_votes is set");
>> + }
>> +
>> + if ((have_qdevice)&&
>> + (qdevice_votes == -1)&&
>> + (have_nodelist)&&
>> + (node_count != node_expected_votes)) {
>> + return ((char *)"configuration error: quorum.device.votes
>> must be specified when not all nodes votes 1");
>> + }
>> +
>> + if ((qdevice_votes> 0)&& (expected_votes)) {
>> + int delta = expected_votes - qdevice_votes;
>> + if (delta< 2) {
>> + return ((char *)"configuration error: quorum.device.votes
>> is too high or expected_votes is too low");
>> + }
>> + }
>> +
>> + /*
>> + * Enable special features
>> + */
>> if (two_node) {
>> wait_for_all = 1;
>> }
>> @@ -760,9 +867,12 @@ static char *votequorum_readconfig_static(void)
>> icmap_get_uint8("quorum.last_man_standing",&last_man_standing);
>>
>> icmap_get_uint32("quorum.last_man_standing_window",&last_man_standing_window);
>>
>>
>> - if
>> ((votequorum_read_nodelist_configuration(&node_votes,&node_expected_votes))&&
>>
>> - (icmap_get_uint32("quorum.expected_votes",&expected_votes) !=
>> CS_OK)) {
>> - return ((char *)"configuration error: nodelist or
>> quorum.expected_votes must be configured!");
>> + if ((have_qdevice)&& (last_man_standing)) {
>> + return ((char *)"configuration error: quorum device is not
>> compatible with last_man_standing feature");
>> + }
>> +
>> + if ((have_qdevice)&& (auto_tie_breaker)) {
>> + return ((char *)"configuration error: quorum device is not
>> compatible with auto_tie_breaker feature");
>> }
>>
>> if (wait_for_all) {
>> @@ -776,43 +886,152 @@ static char *votequorum_readconfig_static(void)
>>
>> static void votequorum_readconfig_dynamic(void)
>> {
>> - int cluster_members = 0;
>> - struct list_head *tmp;
>> + uint32_t node_votes = 0, qdevice_votes = 0;
>> + uint32_t node_expected_votes = 0, expected_votes = 0;
>> + uint32_t node_count = 0;
>> + int have_nodelist, have_qdevice;
>>
>> ENTER();
>>
>> log_printf(LOGSYS_LEVEL_DEBUG, "Reading dynamic configuration");
>>
>> - if
>> (votequorum_read_nodelist_configuration(&us->votes,&us->expected_votes))
>> {
>> - us->votes = 1;
>> - icmap_get_uint32("quorum.votes",&us->votes);
>> + /*
>> + * gather basic data here
>> + * TODO: make it common with static!!!
>> + */
>> + icmap_get_uint32("quorum.expected_votes",&expected_votes);
>> + have_nodelist =
>> votequorum_read_nodelist_configuration(&node_votes,&node_count,&node_expected_votes);
>>
>> + have_qdevice = votequorum_qdevice_is_configured(&qdevice_votes);
>> + icmap_get_uint8("quorum.two_node",&two_node);
>> +
>> + /*
>> + * do config verification and enablement
>> + */
>> +
>> + if ((!have_nodelist)&& (!expected_votes)) {
>> + log_printf(LOGSYS_LEVEL_CRIT, "configuration error: nodelist
>> or quorum.expected_votes must be configured!");
>> + log_printf(LOGSYS_LEVEL_CRIT, "will continue with current
>> runtime data");
>> + goto out;
>> }
>>
>> - icmap_get_uint32("quorum.expected_votes",&us->expected_votes);
>> + /*
>> + * two_node and qdevice are not compatible in the same config.
>> + * try to make an educated guess of what to do
>> + */
>>
>> - update_ev_barrier(us->expected_votes);
>> + if ((two_node)&& (have_qdevice)) {
>> + log_printf(LOGSYS_LEVEL_CRIT, "configuration error: two_node
>> and quorum device cannot be configured at the same time!");
>> + if (qdevice_is_registered) {
>> + log_printf(LOGSYS_LEVEL_CRIT, "quorum device is
>> registered, disabling two_node");
>> + two_node = 0;
>> + } else {
>> + log_printf(LOGSYS_LEVEL_CRIT, "quorum device is not
>> registered, allowing two_node");
>> + update_qdevice_can_operate(0);
>> + }
>> + }
>> +
>> + /*
>> + * quorum device is not compatible with last_man_standing and
>> auto_tie_breaker
>> + * neither lms or atb can be set at runtime, so there is no need
>> to check for
>> + * runtime incompatibilities, but qdevice can be configured
>> _after_ LMS and ATB have
>> + * been enabled at startup.
>> + */
>>
>> -#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> - if (icmap_get_uint32("quorum.quorumdev_poll",&quorumdev_poll) !=
>> CS_OK) {
>> - quorumdev_poll = DEFAULT_QDEV_POLL;
>> + if ((have_qdevice)&& (last_man_standing)) {
>> + log_printf(LOGSYS_LEVEL_CRIT, "configuration error:
>> quorum.device is not compatible with last_man_standing");
>> + log_printf(LOGSYS_LEVEL_CRIT, "disabling quorum device
>> operations");
>> + update_qdevice_can_operate(0);
>> + }
>> +
>> + if ((have_qdevice)&& (auto_tie_breaker)) {
>> + log_printf(LOGSYS_LEVEL_CRIT, "configuration error:
>> quorum.device is not compatible with auto_tie_breaker");
>> + log_printf(LOGSYS_LEVEL_CRIT, "disabling quorum device
>> operations");
>> + update_qdevice_can_operate(0);
>> }
>> -#endif
>>
>> - icmap_get_uint8("quorum.two_node",&two_node);
>> - update_two_node();
>> /*
>> - * two_node mode is invalid if there are more than 2 nodes in the
>> cluster!
>> + * if user specifies quorum.expected_votes + quorum.device but
>> NOT the device.votes
>> + * we don't know what the quorum device should vote.
>> */
>> - list_iterate(tmp,&cluster_members_list) {
>> - cluster_members++;
>> - }
>>
>> - if (two_node&& cluster_members> 2) {
>> - log_printf(LOGSYS_LEVEL_WARNING, "quorum.two_node was set but
>> there are more than 2 nodes in the cluster. It will be ignored.");
>> - two_node = 0;
>> - update_two_node();
>> + if ((expected_votes)&& (have_qdevice)&& (qdevice_votes == -1)) {
>> + log_printf(LOGSYS_LEVEL_CRIT, "configuration error:
>> quorum.device.votes must be specified when quorum.expected_votes is
>> set");
>> + log_printf(LOGSYS_LEVEL_CRIT, "disabling quorum device
>> operations");
>> + update_qdevice_can_operate(0);
>> + }
>> +
>> + /*
>> + * if user specifies a node list with uneven votes and no
>> device.votes
>> + * we cannot autocalculate the votes
>> + */
>> +
>> + if ((have_qdevice)&&
>> + (qdevice_votes == -1)&&
>> + (have_nodelist)&&
>> + (node_count != node_expected_votes)) {
>> + log_printf(LOGSYS_LEVEL_CRIT, "configuration error:
>> quorum.device.votes must be specified when not all nodes votes 1");
>> + log_printf(LOGSYS_LEVEL_CRIT, "disabling quorum device
>> operations");
>> + update_qdevice_can_operate(0);
>> + }
>> +
>> + /*
>> + * validate quorum device votes vs expected_votes
>> + */
>> +
>> + if ((qdevice_votes> 0)&& (expected_votes)) {
>> + int delta = expected_votes - qdevice_votes;
>> + if (delta< 2) {
>> + log_printf(LOGSYS_LEVEL_CRIT, "configuration error:
>> quorum.device.votes is too high or expected_votes is too low");
>> + log_printf(LOGSYS_LEVEL_CRIT, "disabling quorum device
>> operations");
>> + update_qdevice_can_operate(0);
>> + }
>> + }
>> +
>> + /*
>> + * automatically calculate device votes and adjust expected_votes
>> from nodelist
>> + */
>> +
>> + if ((have_qdevice)&&
>> + (qdevice_votes == -1)&&
>> + (!expected_votes)&&
>> + (have_nodelist)&&
>> + (node_count == node_expected_votes)) {
>> + qdevice_votes = node_expected_votes - 1;
>> + node_expected_votes = node_expected_votes + qdevice_votes;
>> + }
>> +
>> + /*
>> + * set this node votes and expected_votes
>> + */
>> +
>> + if (have_nodelist) {
>> + us->votes = node_votes;
>> + us->expected_votes = node_expected_votes;
>> + } else {
>> + us->votes = 1;
>> + icmap_get_uint32("quorum.votes",&us->votes);
>> + }
>> +
>> + if (expected_votes) {
>> + us->expected_votes = expected_votes;
>> + }
>> +
>> + /*
>> + * set qdevice votes
>> + */
>> +
>> + if (!have_qdevice) {
>> + qdevice->votes = 0;
>> + }
>> +
>> + if (qdevice_votes != -1) {
>> + qdevice->votes = qdevice_votes;
>> }
>>
>> + update_ev_barrier(us->expected_votes);
>> + update_two_node();
>> +
>> +out:
>> LEAVE();
>> }
>>
>> @@ -825,11 +1044,13 @@ static void votequorum_refresh_config(
>> {
>> unsigned int old_votes;
>> unsigned int old_expected;
>> + unsigned int old_qdevice_votes;
>>
>> ENTER();
>>
>> old_votes = us->votes;
>> old_expected = us->expected_votes;
>> + old_qdevice_votes = qdevice->votes;
>>
>> /*
>> * Reload the configuration
>> @@ -845,6 +1066,9 @@ static void votequorum_refresh_config(
>> if (old_expected != us->expected_votes) {
>>
>> votequorum_exec_send_reconfigure(VOTEQUORUM_RECONFIG_PARAM_EXPECTED_VOTES,
>> us->node_id, us->expected_votes);
>> }
>> + if (old_qdevice_votes != qdevice->votes) {
>> +
>> votequorum_exec_send_reconfigure(VOTEQUORUM_RECONFIG_PARAM_NODE_VOTES,
>> NODEID_QDEVICE, qdevice->votes);
>> + }
>>
>> LEAVE();
>> }
>> @@ -902,23 +1126,29 @@ static int
>> votequorum_exec_send_reconfigure(uint8_t param, unsigned int nodeid,
>> return ret;
>> }
>>
>> -static int votequorum_exec_send_nodeinfo(void)
>> +static int votequorum_exec_send_nodeinfo(uint32_t nodeid)
>> {
>> struct req_exec_quorum_nodeinfo req_exec_quorum_nodeinfo;
>> struct iovec iov[1];
>> + struct cluster_node *node;
>> int ret;
>>
>> ENTER();
>>
>> - req_exec_quorum_nodeinfo.votes = us->votes;
>> - req_exec_quorum_nodeinfo.expected_votes = us->expected_votes;
>> - req_exec_quorum_nodeinfo.flags = us->flags;
>> - req_exec_quorum_nodeinfo.quorate = cluster_is_quorate;
>> - req_exec_quorum_nodeinfo.wait_for_all_status = wait_for_all_status;
>> - req_exec_quorum_nodeinfo.first_trans = first_trans;
>> - req_exec_quorum_nodeinfo._pad0 = 0;
>> - req_exec_quorum_nodeinfo._pad1 = 0;
>> - req_exec_quorum_nodeinfo._pad2 = 0;
>> + node = find_node_by_nodeid(nodeid);
>> + if (!node) {
>> + return -1;
>> + }
>> +
>> + req_exec_quorum_nodeinfo.nodeid = nodeid;
>> + req_exec_quorum_nodeinfo.votes = node->votes;
>> + req_exec_quorum_nodeinfo.expected_votes = node->expected_votes;
>> + req_exec_quorum_nodeinfo.flags = node->flags;
>> + if (qdevice_is_registered) {
>> + strcpy(req_exec_quorum_nodeinfo.qdevice_name,
>> qdevice->qdevice_name);
>> + } else {
>> + memset(req_exec_quorum_nodeinfo.qdevice_name, 0,
>> VOTEQUORUM_MAX_QDEVICE_NAME_LEN);
>> + }
>>
>> req_exec_quorum_nodeinfo.header.id =
>> SERVICE_ID_MAKE(VOTEQUORUM_SERVICE,
>> MESSAGE_REQ_EXEC_VOTEQUORUM_NODEINFO);
>> req_exec_quorum_nodeinfo.header.size =
>> sizeof(req_exec_quorum_nodeinfo);
>> @@ -948,7 +1178,7 @@ static int
>> votequorum_exec_send_quorum_notification(void *conn, uint64_t context
>> node = list_entry(tmp, struct cluster_node, list);
>> cluster_members++;
>> }
>> - if (quorum_device) {
>> + if (qdevice_is_registered) {
>> cluster_members++;
>> }
>>
>> @@ -968,9 +1198,9 @@ static int
>> votequorum_exec_send_quorum_notification(void *conn, uint64_t context
>> res_lib_votequorum_notification->node_list[i].nodeid =
>> node->node_id;
>> res_lib_votequorum_notification->node_list[i++].state =
>> node->state;
>> }
>> - if (quorum_device) {
>> - res_lib_votequorum_notification->node_list[i].nodeid = 0;
>> - res_lib_votequorum_notification->node_list[i++].state =
>> quorum_device->state | 0x80;
>> + if (qdevice_is_registered) {
>> + res_lib_votequorum_notification->node_list[i].nodeid =
>> NODEID_QDEVICE;
>> + res_lib_votequorum_notification->node_list[i++].state =
>> qdevice->state;
>> }
>> res_lib_votequorum_notification->header.id =
>> MESSAGE_RES_VOTEQUORUM_NOTIFICATION;
>> res_lib_votequorum_notification->header.size = size;
>> @@ -1027,16 +1257,17 @@ static void
>> exec_votequorum_nodeinfo_endian_convert (void *message)
>>
>> ENTER();
>>
>> + nodeinfo->nodeid = swab32(nodeinfo->nodeid);
>> nodeinfo->votes = swab32(nodeinfo->votes);
>> nodeinfo->expected_votes = swab32(nodeinfo->expected_votes);
>> - nodeinfo->flags = swab16(nodeinfo->flags);
>> + nodeinfo->flags = swab32(nodeinfo->flags);
>>
>> LEAVE();
>> }
>>
>> static void message_handler_req_exec_votequorum_nodeinfo (
>> const void *message,
>> - unsigned int nodeid)
>> + unsigned int sender_nodeid)
>> {
>> const struct req_exec_quorum_nodeinfo *req_exec_quorum_nodeinfo
>> = message;
>> struct cluster_node *node = NULL;
>> @@ -1047,16 +1278,15 @@ static void
>> message_handler_req_exec_votequorum_nodeinfo (
>> int new_node = 0;
>> int allow_downgrade = 0;
>> int by_node = 0;
>> + unsigned int nodeid = req_exec_quorum_nodeinfo->nodeid;
>>
>> ENTER();
>>
>> - log_printf(LOGSYS_LEVEL_DEBUG, "got nodeinfo message from cluster
>> node %u", nodeid);
>> - log_printf(LOGSYS_LEVEL_DEBUG, "nodeinfo message[%u]: votes: %d,
>> expected: %d wfa: %d quorate: %d flags: %d",
>> + log_printf(LOGSYS_LEVEL_DEBUG, "got nodeinfo message from cluster
>> node %u", sender_nodeid);
>> + log_printf(LOGSYS_LEVEL_DEBUG, "nodeinfo message[%u]: votes: %d,
>> expected: %d flags: %d",
>> nodeid,
>> req_exec_quorum_nodeinfo->votes,
>> req_exec_quorum_nodeinfo->expected_votes,
>> - req_exec_quorum_nodeinfo->wait_for_all_status,
>> - req_exec_quorum_nodeinfo->quorate,
>> req_exec_quorum_nodeinfo->flags);
>>
>> node = find_node_by_nodeid(nodeid);
>> @@ -1083,41 +1313,74 @@ static void
>> message_handler_req_exec_votequorum_nodeinfo (
>> }
>>
>> /* Update node state */
>> - node->votes = req_exec_quorum_nodeinfo->votes;
>> node->flags = req_exec_quorum_nodeinfo->flags;
>>
>> + if (nodeid != NODEID_QDEVICE) {
>> + node->votes = req_exec_quorum_nodeinfo->votes;
>> +
>> + memset(node->qdevice_name, 0, VOTEQUORUM_MAX_QDEVICE_NAME_LEN);
>> + if (node->flags& NODE_FLAGS_QDEVICE) {
>> + int state = 0;
>> +
>> + if (node->flags& NODE_FLAGS_QDEVICE_STATE) {
>> + state = 1;
>> + }
>> +
>> + strcpy(node->qdevice_name,
>> req_exec_quorum_nodeinfo->qdevice_name);
>> + log_printf(LOGSYS_LEVEL_DEBUG, "Got qdevice data from
>> node[%u]: state %u name %s",
>> + node->node_id, state, node->qdevice_name);
>> + }
>> + } else {
>> + if ((!cluster_is_quorate)&&
>> + (req_exec_quorum_nodeinfo->flags& NODE_FLAGS_QUORATE)) {
>> + node->votes = req_exec_quorum_nodeinfo->votes;
>> + } else {
>> + node->votes = max(node->votes,
>> req_exec_quorum_nodeinfo->votes);
>> + }
>> + }
>> +
>> if (node->flags& NODE_FLAGS_LEAVING) {
>> node->state = NODESTATE_LEAVING;
>> allow_downgrade = 1;
>> by_node = 1;
>> } else {
>> - node->state = NODESTATE_MEMBER;
>> + if (nodeid != NODEID_QDEVICE) {
>> + node->state = NODESTATE_MEMBER;
>> + } else {
>> + /*
>> + * qdevice status is only local to the node
>> + */
>> + node->state = old_state;
>> + }
>> }
>>
>> - if ((!cluster_is_quorate)&&
>> - (req_exec_quorum_nodeinfo->quorate)) {
>> - allow_downgrade = 1;
>> - us->expected_votes = req_exec_quorum_nodeinfo->expected_votes;
>> - }
>> + if (nodeid != NODEID_QDEVICE) {
>> + if ((!cluster_is_quorate)&&
>> + (req_exec_quorum_nodeinfo->flags& NODE_FLAGS_QUORATE)) {
>> + allow_downgrade = 1;
>> + us->expected_votes =
>> req_exec_quorum_nodeinfo->expected_votes;
>> + }
>>
>> - if (req_exec_quorum_nodeinfo->quorate) {
>> - node->expected_votes = req_exec_quorum_nodeinfo->expected_votes;
>> - } else {
>> - node->expected_votes = us->expected_votes;
>> - }
>> + if (req_exec_quorum_nodeinfo->flags& NODE_FLAGS_QUORATE) {
>> + node->expected_votes =
>> req_exec_quorum_nodeinfo->expected_votes;
>> + } else {
>> + node->expected_votes = us->expected_votes;
>> + }
>>
>> - if ((last_man_standing)&& (req_exec_quorum_nodeinfo->votes> 1)) {
>> - log_printf(LOGSYS_LEVEL_WARNING, "Last Man Standing feature
>> is supported only when all"
>> - "cluster nodes votes are set to 1. Disabling
>> LMS.");
>> - last_man_standing = 0;
>> - if (last_man_standing_timer_set) {
>> - corosync_api->timer_delete(last_man_standing_timer);
>> - last_man_standing_timer_set = 0;
>> + if ((last_man_standing)&& (req_exec_quorum_nodeinfo->votes>
>> 1)) {
>> + log_printf(LOGSYS_LEVEL_WARNING, "Last Man Standing
>> feature is supported only when all"
>> + "cluster nodes votes are set to 1.
>> Disabling LMS.");
>> + last_man_standing = 0;
>> + if (last_man_standing_timer_set) {
>> + corosync_api->timer_delete(last_man_standing_timer);
>> + last_man_standing_timer_set = 0;
>> + }
>> }
>> }
>>
>> if (new_node ||
>> - req_exec_quorum_nodeinfo->first_trans ||
>> + nodeid == NODEID_QDEVICE ||
>> + req_exec_quorum_nodeinfo->flags& NODE_FLAGS_FIRST ||
>> old_votes != node->votes ||
>> old_expected != node->expected_votes ||
>> old_flags != node->flags ||
>> @@ -1126,8 +1389,8 @@ static void
>> message_handler_req_exec_votequorum_nodeinfo (
>> }
>>
>> if ((wait_for_all)&&
>> - (!req_exec_quorum_nodeinfo->wait_for_all_status)&&
>> - (req_exec_quorum_nodeinfo->quorate)) {
>> + (!(req_exec_quorum_nodeinfo->flags& NODE_FLAGS_WFASTATUS))&&
>> + (req_exec_quorum_nodeinfo->flags& NODE_FLAGS_QUORATE)) {
>> update_wait_for_all_status(0);
>> }
>>
>> @@ -1156,7 +1419,8 @@ static void
>> message_handler_req_exec_votequorum_reconfigure (
>>
>> ENTER();
>>
>> - log_printf(LOGSYS_LEVEL_DEBUG, "got reconfigure message from
>> cluster node %u", nodeid);
>> + log_printf(LOGSYS_LEVEL_DEBUG, "got reconfigure message from
>> cluster node %u for %u",
>> + nodeid, req_exec_quorum_reconfigure->nodeid);
>>
>> node = find_node_by_nodeid(req_exec_quorum_reconfigure->nodeid);
>> if (!node) {
>> @@ -1196,7 +1460,7 @@ static int votequorum_exec_exit_fn (void)
>>
>> if (leave_remove) {
>> us->flags |= NODE_FLAGS_LEAVING;
>> - ret = votequorum_exec_send_nodeinfo();
>> + ret = votequorum_exec_send_nodeinfo(us->node_id);
>> }
>>
>> LEAVE();
>> @@ -1227,6 +1491,16 @@ static char *votequorum_exec_init_fn (struct
>> corosync_api_v1 *api)
>>
>> us->state = NODESTATE_MEMBER;
>> us->votes = 1;
>> + us->flags |= NODE_FLAGS_FIRST;
>> +
>> + qdevice = allocate_node(NODEID_QDEVICE);
>> + if (!qdevice) {
>> + LEAVE();
>> + return ((char *)"Could not allocate node.");
>> + }
>> + qdevice->state = NODESTATE_DEAD;
>> + qdevice->votes = 0;
>> + memset(qdevice->qdevice_name, 0, VOTEQUORUM_MAX_QDEVICE_NAME_LEN);
>>
>> votequorum_readconfig_dynamic();
>> recalculate_quorum(0, 0);
>> @@ -1239,7 +1513,7 @@ static char *votequorum_exec_init_fn (struct
>> corosync_api_v1 *api)
>> /*
>> * Start us off with one node
>> */
>> - votequorum_exec_send_nodeinfo();
>> + votequorum_exec_send_nodeinfo(us->node_id);
>>
>> LEAVE();
>>
>> @@ -1275,7 +1549,7 @@ static void votequorum_confchg_fn (
>> ENTER();
>>
>> if (member_list_entries> 1) {
>> - first_trans = 0;
>> + us->flags&= ~NODE_FLAGS_FIRST;
>> }
>>
>> if (left_list_entries) {
>> @@ -1304,10 +1578,8 @@ static void votequorum_confchg_fn (
>> if (member_list_entries) {
>> memcpy(quorum_members, member_list, sizeof(unsigned int) *
>> member_list_entries);
>> quorum_members_entries = member_list_entries;
>> - if (quorum_device) {
>> - quorum_members[quorum_members_entries++] = 0;
>> - }
>> - votequorum_exec_send_nodeinfo();
>> + votequorum_exec_send_nodeinfo(us->node_id);
>> + votequorum_exec_send_nodeinfo(NODEID_QDEVICE);
>> }
>>
>> memcpy(&quorum_ringid, ring_id, sizeof(*ring_id));
>> @@ -1392,29 +1664,26 @@ static int quorum_lib_exit_fn (void *conn)
>> * library internal functions
>> */
>>
>> -#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> -static void quorum_device_timer_fn(void *arg)
>> +static void qdevice_timer_fn(void *arg)
>> {
>> ENTER();
>>
>> - if (!quorum_device || quorum_device->state == NODESTATE_DEAD) {
>> + if ((!qdevice_is_registered) ||
>> + (qdevice->state == NODESTATE_DEAD) ||
>> + (!qdevice_timer_set)) {
>> LEAVE();
>> return;
>> }
>>
>> - if ((quorum_device->last_hello / QB_TIME_NS_IN_SEC) +
>> quorumdev_poll/1000<
>> - (qb_util_nano_current_get () / QB_TIME_NS_IN_SEC)) {
>> - quorum_device->state = NODESTATE_DEAD;
>> - log_printf(LOGSYS_LEVEL_INFO, "lost contact with quorum
>> device");
>> - recalculate_quorum(0, 0);
>> - } else {
>> - corosync_api->timer_add_duration((unsigned long
>> long)quorumdev_poll*1000000, quorum_device,
>> - quorum_device_timer_fn,&quorum_device_timer);
>> - }
>> + qdevice->state = NODESTATE_DEAD;
>> + us->flags&= ~NODE_FLAGS_QDEVICE_STATE;
>> + log_printf(LOGSYS_LEVEL_INFO, "lost contact with quorum device
>> %s", qdevice->qdevice_name);
>> + votequorum_exec_send_nodeinfo(us->node_id);
>> +
>> + qdevice_timer_set = 0;
>>
>> LEAVE();
>> }
>> -#endif
>>
>> /*
>> * Library Handler Functions
>> @@ -1448,12 +1717,14 @@ static void
>> message_handler_req_lib_votequorum_getinfo (void *conn, const void *
>> }
>> }
>>
>> - if (quorum_device&& quorum_device->state == NODESTATE_MEMBER) {
>> - total_votes += quorum_device->votes;
>> + if (((qdevice_is_registered)&& (qdevice->state ==
>> NODESTATE_MEMBER)) ||
>> + ((node->flags& NODE_FLAGS_QDEVICE)&& (node->flags&
>> NODE_FLAGS_QDEVICE_STATE))) {
>> + total_votes += qdevice->votes;
>> }
>>
>> - res_lib_votequorum_getinfo.votes = us->votes;
>> - res_lib_votequorum_getinfo.expected_votes = us->expected_votes;
>> + res_lib_votequorum_getinfo.state = node->state;
>> + res_lib_votequorum_getinfo.votes = node->votes;
>> + res_lib_votequorum_getinfo.expected_votes =
>> node->expected_votes;
>> res_lib_votequorum_getinfo.highest_expected = highest_expected;
>>
>> res_lib_votequorum_getinfo.quorum = quorum;
>> @@ -1479,6 +1750,9 @@ static void
>> message_handler_req_lib_votequorum_getinfo (void *conn, const void *
>> if (leave_remove) {
>> res_lib_votequorum_getinfo.flags |=
>> VOTEQUORUM_INFO_LEAVE_REMOVE;
>> }
>> + if (node->flags& NODE_FLAGS_QDEVICE) {
>> + res_lib_votequorum_getinfo.flags |= VOTEQUORUM_INFO_QDEVICE;
>> + }
>> } else {
>> error = CS_ERR_NOT_EXIST;
>> }
>> @@ -1564,10 +1838,6 @@ static void
>> message_handler_req_lib_votequorum_setvotes (void *conn, const void
>> goto error_exit;
>> }
>>
>> - if (!nodeid) {
>> - nodeid = corosync_api->totem_nodeid_get();
>> - }
>> -
>>
>> votequorum_exec_send_reconfigure(VOTEQUORUM_RECONFIG_PARAM_NODE_VOTES,
>> nodeid,
>> req_lib_votequorum_setvotes->votes);
>>
>> @@ -1645,9 +1915,8 @@ static void
>> message_handler_req_lib_votequorum_trackstop (void *conn,
>> LEAVE();
>> }
>>
>> -#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> static void message_handler_req_lib_votequorum_qdevice_register
>> (void *conn,
>> - const void *message)
>> + const void *message)
>> {
>> const struct req_lib_votequorum_qdevice_register
>> *req_lib_votequorum_qdevice_register = message;
>> struct res_lib_votequorum_status res_lib_votequorum_status;
>> @@ -1655,16 +1924,33 @@ static void
>> message_handler_req_lib_votequorum_qdevice_register (void *conn,
>>
>> ENTER();
>>
>> - if (quorum_device) {
>> - error = CS_ERR_EXIST;
>> + if (!qdevice_can_operate) {
>> + log_printf(LOGSYS_LEVEL_INFO, "Registration of quorum device
>> is disabled by incorrect corosync.conf. See logs for more information");
>> + error = CS_ERR_ACCESS;
>> + goto out;
>> + }
>> +
>> + if (qdevice_is_registered) {
>> + if ((!strncmp(req_lib_votequorum_qdevice_register->name,
>> + qdevice->qdevice_name, VOTEQUORUM_MAX_QDEVICE_NAME_LEN))) {
>> + goto out;
>> + } else {
>> + log_printf(LOGSYS_LEVEL_WARNING,
>> + "A new qdevice with different name (new: %s old:
>> %s) is trying to re-register!",
>> + req_lib_votequorum_qdevice_register->name,
>> qdevice->qdevice_name);
>> + error = CS_ERR_EXIST;
>> + goto out;
>> + }
>> } else {
>> - quorum_device = allocate_node(0);
>> - quorum_device->state = NODESTATE_DEAD;
>> - quorum_device->votes =
>> req_lib_votequorum_qdevice_register->votes;
>> - strcpy(quorum_device_name,
>> req_lib_votequorum_qdevice_register->name);
>> - list_add(&quorum_device->list,&cluster_members_list);
>> + qdevice_is_registered = 1;
>> + strcpy(qdevice->qdevice_name,
>> req_lib_votequorum_qdevice_register->name);
>> + us->flags |= NODE_FLAGS_QDEVICE;
>> + votequorum_exec_send_nodeinfo(NODEID_QDEVICE);
>> + votequorum_exec_send_nodeinfo(us->node_id);
>> }
>>
>> +out:
>> +
>> res_lib_votequorum_status.header.size =
>> sizeof(res_lib_votequorum_status);
>> res_lib_votequorum_status.header.id =
>> MESSAGE_RES_VOTEQUORUM_STATUS;
>> res_lib_votequorum_status.header.error = error;
>> @@ -1674,24 +1960,62 @@ static void
>> message_handler_req_lib_votequorum_qdevice_register (void *conn,
>> }
>>
>> static void message_handler_req_lib_votequorum_qdevice_unregister
>> (void *conn,
>> - const void *message)
>> + const void *message)
>> {
>> + const struct req_lib_votequorum_qdevice_unregister
>> *req_lib_votequorum_qdevice_unregister = message;
>> struct res_lib_votequorum_status res_lib_votequorum_status;
>> cs_error_t error = CS_OK;
>>
>> ENTER();
>>
>> - if (quorum_device) {
>> - struct cluster_node *node = quorum_device;
>> + if (qdevice_is_registered) {
>> + if (strncmp(req_lib_votequorum_qdevice_unregister->name,
>> qdevice->qdevice_name, VOTEQUORUM_MAX_QDEVICE_NAME_LEN)) {
>> + error = CS_ERR_INVALID_PARAM;
>> + goto out;
>> + }
>> + if (qdevice_timer_set) {
>> + corosync_api->timer_delete(qdevice_timer);
>> + qdevice_timer_set = 0;
>> + }
>> + qdevice_is_registered = 0;
>> + us->flags&= ~NODE_FLAGS_QDEVICE;
>> + us->flags&= ~NODE_FLAGS_QDEVICE_STATE;
>> + votequorum_exec_send_nodeinfo(us->node_id);
>> + } else {
>> + error = CS_ERR_NOT_EXIST;
>> + }
>>
>> - quorum_device = NULL;
>> - list_del(&node->list);
>> - free(node);
>> - recalculate_quorum(0, 0);
>> +out:
>> + res_lib_votequorum_status.header.size =
>> sizeof(res_lib_votequorum_status);
>> + res_lib_votequorum_status.header.id = MESSAGE_RES_VOTEQUORUM_STATUS;
>> + res_lib_votequorum_status.header.error = error;
>> + corosync_api->ipc_response_send(conn,&res_lib_votequorum_status,
>> sizeof(res_lib_votequorum_status));
>> +
>> + LEAVE();
>> +}
>> +
>> +static void message_handler_req_lib_votequorum_qdevice_update (void
>> *conn,
>> + const void *message)
>> +{
>> + const struct req_lib_votequorum_qdevice_update
>> *req_lib_votequorum_qdevice_update = message;
>> + struct res_lib_votequorum_status res_lib_votequorum_status;
>> + cs_error_t error = CS_OK;
>> +
>> + ENTER();
>> +
>> + if (qdevice_is_registered) {
>> + if (strncmp(req_lib_votequorum_qdevice_update->oldname,
>> qdevice->qdevice_name, VOTEQUORUM_MAX_QDEVICE_NAME_LEN)) {
>> + error = CS_ERR_INVALID_PARAM;
>> + goto out;
>> + }
>> + memset(qdevice->qdevice_name, 0,
>> VOTEQUORUM_MAX_QDEVICE_NAME_LEN);
>> + strcpy(qdevice->qdevice_name,
>> req_lib_votequorum_qdevice_update->newname);
>> + votequorum_exec_send_nodeinfo(us->node_id);
>> } else {
>> error = CS_ERR_NOT_EXIST;
>> }
>>
>> +out:
>> res_lib_votequorum_status.header.size =
>> sizeof(res_lib_votequorum_status);
>> res_lib_votequorum_status.header.id =
>> MESSAGE_RES_VOTEQUORUM_STATUS;
>> res_lib_votequorum_status.header.error = error;
>> @@ -1701,7 +2025,7 @@ static void
>> message_handler_req_lib_votequorum_qdevice_unregister (void *conn,
>> }
>>
>> static void message_handler_req_lib_votequorum_qdevice_poll (void
>> *conn,
>> - const void *message)
>> + const void *message)
>> {
>> const struct req_lib_votequorum_qdevice_poll
>> *req_lib_votequorum_qdevice_poll = message;
>> struct res_lib_votequorum_status res_lib_votequorum_status;
>> @@ -1709,30 +2033,43 @@ static void
>> message_handler_req_lib_votequorum_qdevice_poll (void *conn,
>>
>> ENTER();
>>
>> - if (quorum_device) {
>> - if (req_lib_votequorum_qdevice_poll->state) {
>> - quorum_device->last_hello = qb_util_nano_current_get ();
>> - if (quorum_device->state == NODESTATE_DEAD) {
>> - quorum_device->state = NODESTATE_MEMBER;
>> - recalculate_quorum(0, 0);
>> + if (!qdevice_can_operate) {
>> + error = CS_ERR_ACCESS;
>> + goto out;
>> + }
>>
>> - corosync_api->timer_add_duration((unsigned long
>> long)quorumdev_poll*1000000, quorum_device,
>> -
>> quorum_device_timer_fn,&quorum_device_timer);
>> + if (qdevice_is_registered) {
>> + if (strncmp(req_lib_votequorum_qdevice_poll->name,
>> qdevice->qdevice_name, VOTEQUORUM_MAX_QDEVICE_NAME_LEN)) {
>> + error = CS_ERR_INVALID_PARAM;
>> + goto out;
>> + }
>> + if (qdevice_timer_set) {
>> + corosync_api->timer_delete(qdevice_timer);
>> + qdevice_timer_set = 0;
>> + }
>> +
>> + if (req_lib_votequorum_qdevice_poll->state) {
>> + if (qdevice->state == NODESTATE_DEAD) {
>> + qdevice->state = NODESTATE_MEMBER;
>> + us->flags |= NODE_FLAGS_QDEVICE_STATE;
>> + votequorum_exec_send_nodeinfo(us->node_id);
>> }
>> } else {
>> - if (quorum_device->state == NODESTATE_MEMBER) {
>> - quorum_device->state = NODESTATE_DEAD;
>> - recalculate_quorum(0, 0);
>> - corosync_api->timer_delete(quorum_device_timer);
>> + if (qdevice->state == NODESTATE_MEMBER) {
>> + qdevice->state = NODESTATE_DEAD;
>> + us->flags&= ~NODE_FLAGS_QDEVICE_STATE;
>> + votequorum_exec_send_nodeinfo(us->node_id);
>> }
>> }
>> +
>> + corosync_api->timer_add_duration((unsigned long
>> long)qdevice_timeout*1000000, qdevice,
>> + qdevice_timer_fn,&qdevice_timer);
>> + qdevice_timer_set = 1;
>> } else {
>> error = CS_ERR_NOT_EXIST;
>> }
>>
>> - /*
>> - * send status
>> - */
>> +out:
>> res_lib_votequorum_status.header.size =
>> sizeof(res_lib_votequorum_status);
>> res_lib_votequorum_status.header.id =
>> MESSAGE_RES_VOTEQUORUM_STATUS;
>> res_lib_votequorum_status.header.error = error;
>> @@ -1742,24 +2079,50 @@ static void
>> message_handler_req_lib_votequorum_qdevice_poll (void *conn,
>> }
>>
>> static void message_handler_req_lib_votequorum_qdevice_getinfo (void
>> *conn,
>> - const void *message)
>> + const void *message)
>> {
>> + const struct req_lib_votequorum_qdevice_getinfo
>> *req_lib_votequorum_qdevice_getinfo = message;
>> struct res_lib_votequorum_qdevice_getinfo
>> res_lib_votequorum_qdevice_getinfo;
>> cs_error_t error = CS_OK;
>> + struct cluster_node *node;
>> + uint32_t nodeid = req_lib_votequorum_qdevice_getinfo->nodeid;
>>
>> ENTER();
>>
>> - if (quorum_device) {
>> - log_printf(LOGSYS_LEVEL_DEBUG, "got qdevice_getinfo state
>> %d", quorum_device->state);
>> - res_lib_votequorum_qdevice_getinfo.votes = quorum_device->votes;
>> - if (quorum_device->state == NODESTATE_MEMBER) {
>> - res_lib_votequorum_qdevice_getinfo.state = 1;
>> + if ((nodeid != us->node_id)&&
>> + (nodeid != NODEID_QDEVICE)) {
>> + node =
>> find_node_by_nodeid(req_lib_votequorum_qdevice_getinfo->nodeid);
>> + if ((node)&&
>> + (node->flags& NODE_FLAGS_QDEVICE)) {
>> + int state = 0;
>> +
>> + if (node->flags& NODE_FLAGS_QDEVICE_STATE) {
>> + state = 1;
>> + }
>> + log_printf(LOGSYS_LEVEL_DEBUG, "got qdevice_getinfo node
>> %u state %d", nodeid, state);
>> + res_lib_votequorum_qdevice_getinfo.votes = qdevice->votes;
>> + if (state) {
>> + res_lib_votequorum_qdevice_getinfo.state = 1;
>> + } else {
>> + res_lib_votequorum_qdevice_getinfo.state = 0;
>> + }
>> + strcpy(res_lib_votequorum_qdevice_getinfo.name,
>> node->qdevice_name);
>> } else {
>> - res_lib_votequorum_qdevice_getinfo.state = 0;
>> + error = CS_ERR_NOT_EXIST;
>> }
>> - strcpy(res_lib_votequorum_qdevice_getinfo.name,
>> quorum_device_name);
>> } else {
>> - error = CS_ERR_NOT_EXIST;
>> + if (qdevice_is_registered) {
>> + log_printf(LOGSYS_LEVEL_DEBUG, "got qdevice_getinfo node
>> %u state %d", us->node_id, qdevice->state);
>> + res_lib_votequorum_qdevice_getinfo.votes = qdevice->votes;
>> + if (qdevice->state == NODESTATE_MEMBER) {
>> + res_lib_votequorum_qdevice_getinfo.state = 1;
>> + } else {
>> + res_lib_votequorum_qdevice_getinfo.state = 0;
>> + }
>> + strcpy(res_lib_votequorum_qdevice_getinfo.name,
>> qdevice->qdevice_name);
>> + } else {
>> + error = CS_ERR_NOT_EXIST;
>> + }
>> }
>>
>> res_lib_votequorum_qdevice_getinfo.header.size =
>> sizeof(res_lib_votequorum_qdevice_getinfo);
>> @@ -1769,4 +2132,3 @@ static void
>> message_handler_req_lib_votequorum_qdevice_getinfo (void *conn,
>>
>> LEAVE();
>> }
>> -#endif
>> diff --git a/include/corosync/ipc_votequorum.h
>> b/include/corosync/ipc_votequorum.h
>> index e63bbc5..b633ece 100644
>> --- a/include/corosync/ipc_votequorum.h
>> +++ b/include/corosync/ipc_votequorum.h
>> @@ -48,6 +48,7 @@ enum req_votequorum_types {
>> ,
>> MESSAGE_REQ_VOTEQUORUM_QDEVICE_REGISTER,
>> MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER,
>> + MESSAGE_REQ_VOTEQUORUM_QDEVICE_UPDATE,
>> MESSAGE_REQ_VOTEQUORUM_QDEVICE_POLL,
>> MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO
>> #endif
>> @@ -62,23 +63,39 @@ enum res_votequorum_types {
>> MESSAGE_RES_VOTEQUORUM_EXPECTEDVOTES_NOTIFICATION
>> };
>>
>> -struct req_lib_votequorum_setvotes {
>> +struct req_lib_votequorum_qdevice_register {
>> struct qb_ipc_request_header header __attribute__((aligned(8)));
>> - unsigned int votes;
>> - int nodeid;
>> + char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
>> };
>>
>> -struct req_lib_votequorum_qdevice_register {
>> +struct req_lib_votequorum_qdevice_unregister {
>> struct qb_ipc_request_header header __attribute__((aligned(8)));
>> - unsigned int votes;
>> char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
>> };
>>
>> +struct req_lib_votequorum_qdevice_update {
>> + struct qb_ipc_request_header header __attribute__((aligned(8)));
>> + char oldname[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
>> + char newname[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
>> +};
>> +
>> struct req_lib_votequorum_qdevice_poll {
>> struct qb_ipc_request_header header __attribute__((aligned(8)));
>> + char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
>> int state;
>> };
>>
>> +struct req_lib_votequorum_qdevice_getinfo {
>> + struct qb_ipc_request_header header __attribute__((aligned(8)));
>> + int nodeid;
>> +};
>> +
>> +struct req_lib_votequorum_setvotes {
>> + struct qb_ipc_request_header header __attribute__((aligned(8)));
>> + unsigned int votes;
>> + int nodeid;
>> +};
>> +
>> struct req_lib_votequorum_setexpected {
>> struct qb_ipc_request_header header __attribute__((aligned(8)));
>> unsigned int expected_votes;
>> @@ -109,10 +126,12 @@ struct res_lib_votequorum_status {
>> #define VOTEQUORUM_INFO_LAST_MAN_STANDING 8
>> #define VOTEQUORUM_INFO_AUTO_TIE_BREAKER 16
>> #define VOTEQUORUM_INFO_LEAVE_REMOVE 32
>> +#define VOTEQUORUM_INFO_QDEVICE 64
>>
>> struct res_lib_votequorum_getinfo {
>> struct qb_ipc_response_header header __attribute__((aligned(8)));
>> - int nodeid;
>> + unsigned int nodeid;
>> + unsigned int state;
>> unsigned int votes;
>> unsigned int expected_votes;
>> unsigned int highest_expected;
>> diff --git a/include/corosync/votequorum.h
>> b/include/corosync/votequorum.h
>> index 8173c45..98b7b76 100644
>> --- a/include/corosync/votequorum.h
>> +++ b/include/corosync/votequorum.h
>> @@ -42,19 +42,16 @@ extern "C" {
>>
>> typedef uint64_t votequorum_handle_t;
>>
>> -#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> -#define VOTEQUORUM_MAX_QDEVICE_NAME_LEN 255
>> -#endif
>> -
>> #define VOTEQUORUM_INFO_FLAG_TWONODE 1
>> #define VOTEQUORUM_INFO_FLAG_QUORATE 2
>> #define VOTEQUORUM_INFO_WAIT_FOR_ALL 4
>> #define VOTEQUORUM_INFO_LAST_MAN_STANDING 8
>> #define VOTEQUORUM_INFO_AUTO_TIE_BREAKER 16
>> #define VOTEQUORUM_INFO_LEAVE_REMOVE 32
>> +#define VOTEQUORUM_INFO_QDEVICE 64
>>
>> -#define VOTEQUORUM_NODEID_US 0
>> -#define VOTEQUORUM_NODEID_QDEVICE -1
>> +#define VOTEQUORUM_NODEID_QDEVICE 0
>> +#define VOTEQUORUM_MAX_QDEVICE_NAME_LEN 255
>>
>> #define NODESTATE_MEMBER 1
>> #define NODESTATE_DEAD 2
>> @@ -64,6 +61,7 @@ typedef uint64_t votequorum_handle_t;
>>
>> struct votequorum_info {
>> unsigned int node_id;
>> + unsigned int state;
>> unsigned int node_votes;
>> unsigned int node_expected_votes;
>> unsigned int highest_expected;
>> @@ -187,20 +185,29 @@ cs_error_t votequorum_context_set (
>> */
>> cs_error_t votequorum_qdevice_register (
>> votequorum_handle_t handle,
>> - const char *name,
>> - unsigned int votes);
>> + const char *name);
>>
>> /**
>> * Unregister a quorum device
>> */
>> cs_error_t votequorum_qdevice_unregister (
>> - votequorum_handle_t handle);
>> + votequorum_handle_t handle,
>> + const char *name);
>> +
>> +/**
>> + * Update registered name of a quorum device
>> + */
>> +cs_error_t votequorum_qdevice_update (
>> + votequorum_handle_t handle,
>> + const char *oldname,
>> + const char *newname);
>>
>> /**
>> * Poll a quorum device
>> */
>> cs_error_t votequorum_qdevice_poll (
>> votequorum_handle_t handle,
>> + const char *name,
>> unsigned int state);
>>
>> /**
>> @@ -208,6 +215,7 @@ cs_error_t votequorum_qdevice_poll (
>> */
>> cs_error_t votequorum_qdevice_getinfo (
>> votequorum_handle_t handle,
>> + unsigned int nodeid,
>> struct votequorum_qdevice_info *info);
>>
>> #endif
>> diff --git a/lib/votequorum.c b/lib/votequorum.c
>> index 32f5a6b..133e706 100644
>> --- a/lib/votequorum.c
>> +++ b/lib/votequorum.c
>> @@ -174,6 +174,7 @@ cs_error_t votequorum_getinfo (
>> error = res_lib_votequorum_getinfo.header.error;
>>
>> info->node_id = res_lib_votequorum_getinfo.nodeid;
>> + info->state = res_lib_votequorum_getinfo.state;
>> info->node_votes = res_lib_votequorum_getinfo.votes;
>> info->node_expected_votes =
>> res_lib_votequorum_getinfo.expected_votes;
>> info->highest_expected =
>> res_lib_votequorum_getinfo.highest_expected;
>> @@ -539,8 +540,7 @@ error_put:
>> #ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> cs_error_t votequorum_qdevice_register (
>> votequorum_handle_t handle,
>> - const char *name,
>> - unsigned int votes)
>> + const char *name)
>> {
>> cs_error_t error;
>> struct votequorum_inst *votequorum_inst;
>> @@ -548,8 +548,9 @@ cs_error_t votequorum_qdevice_register (
>> struct req_lib_votequorum_qdevice_register
>> req_lib_votequorum_qdevice_register;
>> struct res_lib_votequorum_status res_lib_votequorum_status;
>>
>> - if (strlen(name)>= VOTEQUORUM_MAX_QDEVICE_NAME_LEN)
>> + if (strlen(name)>= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
>> return CS_ERR_INVALID_PARAM;
>> + }
>>
>> error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db,
>> handle, (void *)&votequorum_inst));
>> if (error != CS_OK) {
>> @@ -560,7 +561,6 @@ cs_error_t votequorum_qdevice_register (
>> req_lib_votequorum_qdevice_register.header.size = sizeof (struct
>> req_lib_votequorum_qdevice_register);
>> req_lib_votequorum_qdevice_register.header.id =
>> MESSAGE_REQ_VOTEQUORUM_QDEVICE_REGISTER;
>> strcpy(req_lib_votequorum_qdevice_register.name, name);
>> - req_lib_votequorum_qdevice_register.votes = votes;
>>
>> iov.iov_base = (char *)&req_lib_votequorum_qdevice_register;
>> iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_register);
>> @@ -586,6 +586,7 @@ error_exit:
>>
>> cs_error_t votequorum_qdevice_poll (
>> votequorum_handle_t handle,
>> + const char *name,
>> unsigned int state)
>> {
>> cs_error_t error;
>> @@ -594,14 +595,18 @@ cs_error_t votequorum_qdevice_poll (
>> struct req_lib_votequorum_qdevice_poll
>> req_lib_votequorum_qdevice_poll;
>> struct res_lib_votequorum_status res_lib_votequorum_status;
>>
>> + if (strlen(name)>= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
>> + return CS_ERR_INVALID_PARAM;
>> + }
>> +
>> error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db,
>> handle, (void *)&votequorum_inst));
>> if (error != CS_OK) {
>> return (error);
>> }
>>
>> -
>> req_lib_votequorum_qdevice_poll.header.size = sizeof (struct
>> req_lib_votequorum_qdevice_poll);
>> req_lib_votequorum_qdevice_poll.header.id =
>> MESSAGE_REQ_VOTEQUORUM_QDEVICE_POLL;
>> + strcpy(req_lib_votequorum_qdevice_poll.name, name);
>> req_lib_votequorum_qdevice_poll.state = state;
>>
>> iov.iov_base = (char *)&req_lib_votequorum_qdevice_poll;
>> @@ -627,24 +632,30 @@ error_exit:
>> }
>>
>> cs_error_t votequorum_qdevice_unregister (
>> - votequorum_handle_t handle)
>> + votequorum_handle_t handle,
>> + const char *name)
>> {
>> cs_error_t error;
>> struct votequorum_inst *votequorum_inst;
>> struct iovec iov;
>> - struct req_lib_votequorum_general req_lib_votequorum_general;
>> + struct req_lib_votequorum_qdevice_unregister
>> req_lib_votequorum_qdevice_unregister;
>> struct res_lib_votequorum_status res_lib_votequorum_status;
>>
>> + if (strlen(name)>= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
>> + return CS_ERR_INVALID_PARAM;
>> + }
>> +
>> error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db,
>> handle, (void *)&votequorum_inst));
>> if (error != CS_OK) {
>> return (error);
>> }
>>
>> - req_lib_votequorum_general.header.size = sizeof (struct
>> req_lib_votequorum_general);
>> - req_lib_votequorum_general.header.id =
>> MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER;
>> + req_lib_votequorum_qdevice_unregister.header.size = sizeof
>> (struct req_lib_votequorum_qdevice_unregister);
>> + req_lib_votequorum_qdevice_unregister.header.id =
>> MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER;
>> + strcpy(req_lib_votequorum_qdevice_unregister.name, name);
>>
>> - iov.iov_base = (char *)&req_lib_votequorum_general;
>> - iov.iov_len = sizeof (struct req_lib_votequorum_general);
>> + iov.iov_base = (char *)&req_lib_votequorum_qdevice_unregister;
>> + iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_unregister);
>>
>> error = qb_to_cs_error(qb_ipcc_sendv_recv (
>> votequorum_inst->c,
>> @@ -667,12 +678,13 @@ error_exit:
>>
>> cs_error_t votequorum_qdevice_getinfo (
>> votequorum_handle_t handle,
>> + unsigned int nodeid,
>> struct votequorum_qdevice_info *qinfo)
>> {
>> cs_error_t error;
>> struct votequorum_inst *votequorum_inst;
>> struct iovec iov;
>> - struct req_lib_votequorum_general req_lib_votequorum_general;
>> + struct req_lib_votequorum_qdevice_getinfo
>> req_lib_votequorum_qdevice_getinfo;
>> struct res_lib_votequorum_qdevice_getinfo
>> res_lib_votequorum_qdevice_getinfo;
>>
>> error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db,
>> handle, (void *)&votequorum_inst));
>> @@ -680,12 +692,12 @@ cs_error_t votequorum_qdevice_getinfo (
>> return (error);
>> }
>>
>> + req_lib_votequorum_qdevice_getinfo.header.size = sizeof (struct
>> req_lib_votequorum_qdevice_getinfo);
>> + req_lib_votequorum_qdevice_getinfo.header.id =
>> MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO;
>> + req_lib_votequorum_qdevice_getinfo.nodeid = nodeid;
>>
>> - req_lib_votequorum_general.header.size = sizeof (struct
>> req_lib_votequorum_general);
>> - req_lib_votequorum_general.header.id =
>> MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO;
>> -
>> - iov.iov_base = (char *)&req_lib_votequorum_general;
>> - iov.iov_len = sizeof (struct req_lib_votequorum_general);
>> + iov.iov_base = (char *)&req_lib_votequorum_qdevice_getinfo;
>> + iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_getinfo);
>>
>> error = qb_to_cs_error(qb_ipcc_sendv_recv (
>> votequorum_inst->c,
>> diff --git a/test/testvotequorum2.c b/test/testvotequorum2.c
>> index 39a7cbf..560f8ca 100644
>> --- a/test/testvotequorum2.c
>> +++ b/test/testvotequorum2.c
>> @@ -50,7 +50,7 @@ static void print_info(int ok_to_fail)
>> struct votequorum_qdevice_info qinfo;
>> int err;
>>
>> - if ( (err=votequorum_qdevice_getinfo(handle,&qinfo)) != CS_OK)
>> + if ( (err=votequorum_qdevice_getinfo(handle,
>> VOTEQUORUM_NODEID_QDEVICE,&qinfo)) != CS_OK)
>> fprintf(stderr, "votequorum_qdevice_getinfo error %d: %s\n",
>> err, ok_to_fail?"OK":"FAILED");
>> else {
>> printf("qdevice votes %d\n", qinfo.votes);
>> @@ -63,6 +63,7 @@ static void print_info(int ok_to_fail)
>>
>> int main(int argc, char *argv[])
>> {
>> + int ret = 0;
>> #ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> int pollcount=0, polltime=1;
>> int err;
>> @@ -82,24 +83,34 @@ int main(int argc, char *argv[])
>> }
>>
>> if (argc>= 2) {
>> - if ( (err=votequorum_qdevice_register(handle, "QDEVICE", 4))
>> != CS_OK)
>> + if ( (err=votequorum_qdevice_register(handle, "QDEVICE")) !=
>> CS_OK) {
>> fprintf(stderr, "qdevice_register FAILED: %d\n", err);
>> + ret = -1;
>> + goto out;
>> + }
>>
>> while (pollcount--) {
>> print_info(0);
>> - if ((err=votequorum_qdevice_poll(handle, 1)) != CS_OK)
>> + if ((err=votequorum_qdevice_poll(handle, "QDEVICE", 1))
>> != CS_OK) {
>> fprintf(stderr, "qdevice poll FAILED: %d\n", err);
>> + ret = -1;
>> + goto out;
>> + }
>> print_info(0);
>> sleep(polltime);
>> }
>> - if ((err= votequorum_qdevice_unregister(handle)) != CS_OK)
>> + if ((err= votequorum_qdevice_unregister(handle, "QDEVICE"))
>> != CS_OK) {
>> fprintf(stderr, "qdevice unregister FAILED: %d\n", err);
>> + ret = -1;
>> + goto out;
>> + }
>> }
>> print_info(1);
>>
>> +out:
>> votequorum_finalize(handle);
>> #else
>> - fprintf(stderr, "qdevice support is not built in
>> corosync/votequorum\n");
>> + fprintf(stderr, "qdevice support is not built in
>> corosync/votequorum\n");
>> #endif
>> - return 0;
>> + return ret;
>> }
>> diff --git a/tools/corosync-quorumtool.c b/tools/corosync-quorumtool.c
>> index 0a21a3c..f60d7fd 100644
>> --- a/tools/corosync-quorumtool.c
>> +++ b/tools/corosync-quorumtool.c
>> @@ -63,7 +63,8 @@ typedef enum {
>> CMD_SHOWSTATUS,
>> CMD_SETVOTES,
>> CMD_SETEXPECTED,
>> - CMD_MONITOR
>> + CMD_MONITOR,
>> + CMD_UNREGISTER_QDEVICE
>> } command_t;
>>
>> /*
>> @@ -108,6 +109,8 @@ static votequorum_callbacks_t v_callbacks = {
>> .votequorum_notify_fn = NULL,
>> .votequorum_expectedvotes_notify_fn = NULL
>> };
>> +static uint32_t our_nodeid = 0;
>> +static uint8_t display_qdevice = 0;
>>
>> /*
>> * cfg bits
>> @@ -132,6 +135,7 @@ static void show_usage(const char *name)
>> printf(" -e<expected> change expected votes for the cluster
>> *\n");
>> printf(" -H show nodeids in hexadecimal rather than
>> decimal\n");
>> printf(" -i show node IP addresses instead of the
>> resolved name\n");
>> + printf(" -f forcefully unregister a quorum device
>> *DANGEROUS*\n");
>> printf(" -h show this help text\n");
>> printf("\n");
>> printf(" * Starred items only work if votequorum is the quorum
>> provider for corosync\n");
>> @@ -283,7 +287,7 @@ static void quorum_notification_fn(
>> }
>> }
>>
>> -static void display_nodes_data(nodeid_format_t nodeid_format,
>> name_format_t name_format)
>> +static void display_nodes_data(uint32_t nodeid, nodeid_format_t
>> nodeid_format, name_format_t name_format)
>> {
>> int i;
>>
>> @@ -305,13 +309,51 @@ static void display_nodes_data(nodeid_format_t
>> nodeid_format, name_format_t name
>> printf("%s\n", node_name(g_view_list[i], name_format));
>> }
>> }
>> +
>> if (g_view_list_entries) {
>> free(g_view_list);
>> g_view_list = NULL;
>> }
>> +
>> +#if EXPERIMENTAL_QUORUM_DEVICE_API
>> + if ((display_qdevice)&& (v_handle)) {
>> + int err;
>> + struct votequorum_qdevice_info qinfo;
>> +
>> + err = votequorum_qdevice_getinfo(v_handle, nodeid,&qinfo);
>> + if (err != CS_OK) {
>> + fprintf(stderr, "votequorum_qdevice_getinfo error: %d\n",
>> err);
>> + } else {
>> + if (nodeid_format == NODEID_FORMAT_DECIMAL) {
>> + printf("%10u ", VOTEQUORUM_NODEID_QDEVICE);
>> + } else {
>> + printf("0x%08x ", VOTEQUORUM_NODEID_QDEVICE);
>> + }
>> + printf("%3d %s (%s)\n", qinfo.votes, qinfo.name,
>> qinfo.state?"Voting":"Not voting");
>> + }
>> + }
>> +#endif
>> +}
>> +
>> +static const char *decode_state(int state)
>> +{
>> + switch(state) {
>> + case NODESTATE_MEMBER:
>> + return "Member";
>> + break;
>> + case NODESTATE_DEAD:
>> + return "Dead";
>> + break;
>> + case NODESTATE_LEAVING:
>> + return "Leaving";
>> + break;
>> + default:
>> + return "Unknown state (this is bad)";
>> + break;
>> + }
>> }
>>
>> -static int display_quorum_data(int is_quorate, int loop)
>> +static int display_quorum_data(int is_quorate, uint32_t nodeid, int
>> loop)
>> {
>> struct votequorum_info info;
>> int err;
>> @@ -335,8 +377,9 @@ static int display_quorum_data(int is_quorate, int
>> loop)
>> return CS_OK;
>> }
>>
>> - if ((err=votequorum_getinfo(v_handle, 0,&info)) == CS_OK) {
>> + if ((err=votequorum_getinfo(v_handle, nodeid,&info)) == CS_OK) {
>> printf("Node votes: %d\n", info.node_votes);
>> + printf("Node state: %s\n", decode_state(info.state));
>> printf("Expected votes: %d\n", info.node_expected_votes);
>> printf("Highest expected: %d\n", info.highest_expected);
>> printf("Total votes: %d\n", info.total_votes);
>> @@ -348,6 +391,12 @@ static int display_quorum_data(int is_quorate,
>> int loop)
>> if (info.flags& VOTEQUORUM_INFO_LAST_MAN_STANDING)
>> printf("LastManStanding ");
>> if (info.flags& VOTEQUORUM_INFO_AUTO_TIE_BREAKER)
>> printf("AutoTieBreaker ");
>> if (info.flags& VOTEQUORUM_INFO_LEAVE_REMOVE)
>> printf("LeaveRemove ");
>> + if (info.flags& VOTEQUORUM_INFO_QDEVICE) {
>> + printf("Qdevice ");
>> + display_qdevice = 1;
>> + } else {
>> + display_qdevice = 0;
>> + }
>> printf("\n");
>> } else {
>> fprintf(stderr, "votequorum_getinfo FAILED: %d\n", err);
>> @@ -361,7 +410,7 @@ static int display_quorum_data(int is_quorate, int
>> loop)
>> * 0 if not quorate
>> * -1 on error
>> */
>> -static int show_status(nodeid_format_t nodeid_format, name_format_t
>> name_format)
>> +static int show_status(uint32_t nodeid, nodeid_format_t
>> nodeid_format, name_format_t name_format)
>> {
>> int is_quorate;
>> int err;
>> @@ -395,11 +444,11 @@ quorum_err:
>> return -1;
>> }
>>
>> - err = display_quorum_data(is_quorate, 0);
>> + err = display_quorum_data(is_quorate, nodeid, 0);
>> if (err != CS_OK) {
>> return -1;
>> }
>> - display_nodes_data(nodeid_format, name_format);
>> + display_nodes_data(nodeid, nodeid_format, name_format);
>>
>> return is_quorate;
>> }
>> @@ -410,7 +459,7 @@ static int monitor_status(nodeid_format_t
>> nodeid_format, name_format_t name_form
>>
>> if (q_type == QUORUM_FREE) {
>> printf("\nQuorum is not configured - cannot monitor\n");
>> - return show_status(nodeid_format, name_format);
>> + return show_status(our_nodeid, nodeid_format, name_format);
>> }
>>
>> err=quorum_trackstart(q_handle, CS_TRACK_CHANGES);
>> @@ -429,8 +478,8 @@ static int monitor_status(nodeid_format_t
>> nodeid_format, name_format_t name_form
>> }
>> time(&t);
>> printf("date: %s", ctime((const time_t *)&t));
>> - err = display_quorum_data(g_quorate, loop);
>> - display_nodes_data(nodeid_format, name_format);
>> + err = display_quorum_data(g_quorate, our_nodeid, loop);
>> + display_nodes_data(our_nodeid, nodeid_format, name_format);
>> printf("\n");
>> loop = 1;
>> if (err != CS_OK) {
>> @@ -463,13 +512,37 @@ static int show_nodes(nodeid_format_t
>> nodeid_format, name_format_t name_format)
>> }
>> }
>>
>> - display_nodes_data(nodeid_format, name_format);
>> + display_nodes_data(our_nodeid, nodeid_format, name_format);
>>
>> result = EXIT_SUCCESS;
>> err_exit:
>> return result;
>> }
>>
>> +static int unregister_qdevice(void)
>> +{
>> +#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
>> + int err;
>> + struct votequorum_qdevice_info qinfo;
>> +
>> + err = votequorum_qdevice_getinfo(v_handle, our_nodeid,&qinfo);
>> + if (err != CS_OK) {
>> + fprintf(stderr, "votequorum_qdevice_getinfo FAILED: %d\n", err);
>> + return -1;
>> + }
>> +
>> + err = votequorum_qdevice_unregister(v_handle, qinfo.name);
>> + if (err != CS_OK) {
>> + fprintf(stderr, "votequorum_qdevice_unregister FAILED: %d\n",
>> err);
>> + return -1;
>> + }
>> + return 0;
>> +#else
>> + fprintf(stderr, "votequorum quorum device support is not
>> built-in\n");
>> + return -1;
>> +#endif
>> +}
>> +
>> /*
>> * return -1 on error
>> * 0 if OK
>> @@ -509,6 +582,11 @@ static int init_all(void) {
>> goto out;
>> }
>>
>> + if (cmap_get_uint32(cmap_handle,
>> "runtime.votequorum.this_node_id",&our_nodeid) != CS_OK) {
>> + fprintf(stderr, "Unable to retrive this node nodeid\n");
>> + goto out;
>> + }
>> +
>> return 0;
>> out:
>> return -1;
>> @@ -530,12 +608,13 @@ static void close_all(void) {
>> }
>>
>> int main (int argc, char *argv[]) {
>> - const char *options = "VHslme:v:hin:d:";
>> + const char *options = "VHslmfe:v:hin:d:";
>> char *endptr;
>> int opt;
>> int votes = 0;
>> int ret = 0;
>> - uint32_t nodeid = VOTEQUORUM_NODEID_US;
>> + uint32_t nodeid = 0;
>> + uint32_t nodeid_set = 0;
>> nodeid_format_t nodeid_format = NODEID_FORMAT_DECIMAL;
>> name_format_t address_format = ADDRESS_FORMAT_NAME;
>> command_t command_opt = CMD_UNKNOWN;
>> @@ -552,6 +631,14 @@ int main (int argc, char *argv[]) {
>>
>> while ( (opt = getopt(argc, argv, options)) != -1 ) {
>> switch (opt) {
>> + case 'f':
>> + if (using_votequorum()> 0) {
>> + command_opt = CMD_UNREGISTER_QDEVICE;
>> + } else {
>> + fprintf(stderr, "You cannot unregister quorum device,
>> corosync is not using votequorum\n");
>> + exit(2);
>> + }
>> + break;
>> case 's':
>> command_opt = CMD_SHOWSTATUS;
>> break;
>> @@ -582,15 +669,18 @@ int main (int argc, char *argv[]) {
>> break;
>> case 'n':
>> nodeid = strtol(optarg,&endptr, 0);
>> - if ((nodeid == 0&& endptr == optarg) || nodeid<= 0) {
>> + if ((nodeid == 0&& endptr == optarg) || nodeid< 0) {
>> fprintf(stderr, "The nodeid was not valid, try a
>> positive number\n");
>> + exit(2);
>> }
>> + nodeid_set = 1;
>> break;
>> case 'v':
>> if (using_votequorum()> 0) {
>> votes = strtol(optarg,&endptr, 0);
>> if ((votes == 0&& endptr == optarg) || votes< 0) {
>> fprintf(stderr, "New votes value was not valid,
>> try a positive number or zero\n");
>> + exit(2);
>> } else {
>> command_opt = CMD_SETVOTES;
>> }
>> @@ -600,10 +690,12 @@ int main (int argc, char *argv[]) {
>> exit(2);
>> }
>> break;
>> + case ':':
>> case 'h':
>> case '?':
>> default:
>> - break;
>> + command_opt = CMD_UNKNOWN;
>> + break;
>> }
>> }
>>
>> @@ -616,9 +708,15 @@ int main (int argc, char *argv[]) {
>> ret = show_nodes(nodeid_format, address_format);
>> break;
>> case CMD_SHOWSTATUS:
>> - ret = show_status(nodeid_format, address_format);
>> + if (!nodeid_set) {
>> + nodeid = our_nodeid;
>> + }
>> + ret = show_status(nodeid, nodeid_format, address_format);
>> break;
>> case CMD_SETVOTES:
>> + if (!nodeid_set) {
>> + nodeid = our_nodeid;
>> + }
>> ret = set_votes(nodeid, votes);
>> break;
>> case CMD_SETEXPECTED:
>> @@ -627,6 +725,9 @@ int main (int argc, char *argv[]) {
>> case CMD_MONITOR:
>> ret = monitor_status(nodeid_format, address_format);
>> break;
>> + case CMD_UNREGISTER_QDEVICE:
>> + ret = unregister_qdevice();
>> + break;
>> }
>>
>> close_all();
>
> _______________________________________________
> discuss mailing list
> discuss@xxxxxxxxxxxx
> http://lists.corosync.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss
