From: "Fabio M. Di Nitto" <fdinitto@xxxxxxxxxx>
qdevice is a very special node in the cluster and it adds a certain
amount of complexity and special cases across the code.
most of the qdevice data are shared across the cluster (name/votes)
but effectively each node has a different view of the qdevice
(registered/unregistered/voting/etc.)
with this change, we align the qdevice view across the node,
exchanging more data between nodes and we fix how qdevice behaves
and it is configured.
The only side effect is that the amount of data transmitted on wire
is slightly higher.
The qdevice API is still disabled by default. This means that
the amount of real changes in current code are a lot smaller
than it appears by this patch.
TODO: documentation/man pages needs to be updated once
this change is in (and behavior finalized).
User visible changes:
- configuration (coroparse, exec/votequorum):
the quorum device section is now standalone within the quorum.
quorum {
provider: corosync_votequorum
device {
model: (name)
timeout: (millisec)
votes:
}
}
the keyword "model:" is mandatory to enable qdevice in configuration
and should express the name of the script/daemon that will provide
the qdevice. Looking into the future, an init script or systemd
service will look for that name in /path/to/be/decided/name
and start/stop qdevice.
timeout: defines the maximum interval the qdevice implementation
has available between poll (see votequorum_qdevice_poll.3) before
the device is considered dead and votes discarded
votes: is now a configuration parameter and not an API call.
quorum devices don't care what they need to vote.
votes is autocalculated when a nodelist is available and all
nodes in the list vote 1. Otherwise this parameter is mandatory.
- configuration (exec/votequorum):
startup and runtime configuration changes have been improved.
errors at startup are considered fatal. errors at runtime
have different exit paths.
startup:
* quorum.two_node and qdevice are incompatible.
* quorum.expected_votes requires quorum.device.votes.
* quorum.expected_votes - quorum.device.votes cannot be lower
than 2.
* qdevice and last_man_standing are mutually exclusive.
* qdevice and auto_tie_breaker are mutually exclusive.
runtime config changes:
* quorum.two_node and qdevice are incompatible:
if quorum device is alive, two_node is disabled.
if quorum device is not alive and node count is 2, two_node is
enabled, and quorum device cannot be registered
* if either last_man_standing or auto_tie_breaker were enabled
at startup, and at runtime quorum device is configured,
quorum device registration will be blocked.
* if quorum.expected_votes is configured but not quorum.device.votes,
quorum device registration will be blocked.
* if quorum.device.votes is not configured and we cannot
automatically calculate it, quorum device registration will be blocked.
* An error in configuring quorum.expected_votes and quorum.device.votes
will block quorum device registration.
blocking quorum device registation, also means dropping the votes.
quorum.device.votes (either set or automatically calculated) is now
used to determine current expected_votes in the cluster.
- logging (exec/votequorum):
all errors from configuration are treated as WARNING/CRITICAL.
lots of extra DEBUG output is added (see internal changes too).
- corosync-quorumtool (tools/corosync-quorumtool):
* added option to forcefully kick out a quorum device from the local
node. This is for emergency recovery only and it is only
available when qdevice API is built-in.
* Improved status output, specifically add node state and qdevice
information
[root@fedora-master-node2 coro]# corosync-quorumtool -s
Version: 1.99.4.12-9c7d-dirty
Quorum type: corosync_votequorum
Nodes: 2
Ring ID: 132
Quorate: Yes
Node votes: 1
Node state: Member
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate Qdevice
Nodeid Votes Name
1 1 fedora-master-node1.int.fabbione.net
2 1 fedora-master-node2.int.fabbione.net
0 1 QDEVICE (Voting)
* allow to print status for any node in the cluster known to
local node.
[root@fedora-master-node1 coro]# corosync-quorumtool -s
Version: 1.99.4.12-9c7d-dirty
Quorum type: corosync_votequorum
Nodes: 2
Ring ID: 144
Quorate: Yes
Node votes: 1
Node state: Member
Expected votes: 3
Highest expected: 3
Total votes: 2
Quorum: 2
Flags: Quorate
Nodeid Votes Name
1 1 fedora-master-node1.int.fabbione.net
2 1 fedora-master-node2.int.fabbione.net
[root@fedora-master-node1 coro]# corosync-quorumtool -s -n 2
Version: 1.99.4.12-9c7d-dirty
Quorum type: corosync_votequorum
Nodes: 2
Ring ID: 144
Quorate: Yes
Node votes: 1
Node state: Member
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate Qdevice
Nodeid Votes Name
1 1 fedora-master-node1.int.fabbione.net
2 1 fedora-master-node2.int.fabbione.net
0 1 QDEVICE (Voting)
Internal changes:
- change qdevice to not run all time, but only when necessary.
- change votequorum_nodeinfo on wire data to include
qdevice name and use flags instead of uint8_t.
- allocate nodeid 0 to qdevice since it's the only real
nodeid that be reserved.
- change send_nodeinfo to allow to send nodeinfo for any node
so that we can share qdevice info across the cluster
(and this might be useful in future if we need to sync
internal cluster view).
- add votequorum api call to update qdevice name
- add runtime data if quorum device has been forcefully disabled
by config error
- add qdevice votes to expected_votes calculation (this
is probably the biggest difference vs cman)
- change votequorum_read_nodelist_configuration so that
we can autocalculate votes for qdevice (we need the nodecount
vs votes).
- add all checks for startup/runtime config (see above).
- maintain qdevice information for each node. This is necessary
because the qdevice cluster_node is local only and
lay down the road to improve user error configuration in future
(for example we will be able to block 2 different qdevice model
to be used in the cluster at the same time).
- do not make qdevice part of the membership_list received from
totem. None of our users care about it and it is not a real node.
- change onwire message handlers to deal with "data for this node from any node"
case and undersand nodeid 0 for qdevice info
- always allocate qdevice at startup. this simplifies code a lot.
- dispatch qdevice nodeinfo on membership changes.
- inform libvotequorum users when a qdevice is registered
- improve substantially qdevice api and add a simple (for now)
barrier based on qdevice name. Plan is that only qdevice
with the same name can register across the cluster.
- qdevice getinfo can now report status for qdevice on any node.
- change slightly the way the qdevice API is built-in/out:
only the libvotequorum calls are #ifdef'out now. Doing so in
the core is too complex and would make the code unreadable
with the risk of missing a bit or two effectively introducing
an on-wire incompatibility if we will ever turn the API on.
- probably added some bugs on the way...
TODO: update qdevice_* API once the above is settled and test
qdevice integration with other features.
Signed-off-by: Fabio M. Di Nitto <fdinitto@xxxxxxxxxx>
---
exec/coroparse.c | 17 +-
exec/votequorum.c | 728 +++++++++++++++++++++++++++----------
include/corosync/ipc_votequorum.h | 31 ++-
include/corosync/votequorum.h | 26 +-
lib/votequorum.c | 46 ++-
test/testvotequorum2.c | 23 +-
tools/corosync-quorumtool.c | 133 ++++++-
7 files changed, 766 insertions(+), 238 deletions(-)
diff --git a/exec/coroparse.c b/exec/coroparse.c
index e6a5cb8..1d97323 100644
--- a/exec/coroparse.c
+++ b/exec/coroparse.c
@@ -87,6 +87,7 @@ enum main_cp_cb_data_state {
MAIN_CP_CB_DATA_STATE_LOGGING_DAEMON,
MAIN_CP_CB_DATA_STATE_MEMBER,
MAIN_CP_CB_DATA_STATE_QUORUM,
+ MAIN_CP_CB_DATA_STATE_QDEVICE,
MAIN_CP_CB_DATA_STATE_NODELIST,
MAIN_CP_CB_DATA_STATE_NODELIST_NODE,
};
@@ -388,7 +389,6 @@ static int main_config_parser_cb(const char *path,
case MAIN_CP_CB_DATA_STATE_QUORUM:
if ((strcmp(path, "quorum.expected_votes") == 0) ||
(strcmp(path, "quorum.votes") == 0) ||
- (strcmp(path, "quorum.quorumdev_poll") == 0) ||
(strcmp(path, "quorum.last_man_standing_window") == 0) ||
(strcmp(path, "quorum.leaving_timeout") == 0)) {
if (safe_atoi(value, &i) != 0) {
@@ -410,6 +410,15 @@ static int main_config_parser_cb(const char *path,
add_as_string = 0;
}
break;
+ case MAIN_CP_CB_DATA_STATE_QDEVICE:
+ if ((strcmp(path, "quorum.device.timeout") == 0) ||
+ (strcmp(path, "quorum.device.votes") == 0)) {
+ if (safe_atoi(value, &i) != 0) {
+ goto atoi_error;
+ }
+ icmap_set_uint32(path, i);
+ add_as_string = 0;
+ }
case MAIN_CP_CB_DATA_STATE_TOTEM:
if ((strcmp(path, "totem.version") == 0) ||
(strcmp(path, "totem.nodeid") == 0) ||
@@ -671,6 +680,9 @@ static int main_config_parser_cb(const char *path,
if (strcmp(path, "quorum") == 0) {
data->state = MAIN_CP_CB_DATA_STATE_QUORUM;
}
+ if (strcmp(path, "quorum.device") == 0) {
+ data->state = MAIN_CP_CB_DATA_STATE_QDEVICE;
+ }
if (strcmp(path, "nodelist") == 0) {
data->state = MAIN_CP_CB_DATA_STATE_NODELIST;
data->node_number = 0;
@@ -858,6 +870,9 @@ static int main_config_parser_cb(const char *path,
case MAIN_CP_CB_DATA_STATE_QUORUM:
data->state = MAIN_CP_CB_DATA_STATE_NORMAL;
break;
+ case MAIN_CP_CB_DATA_STATE_QDEVICE:
+ data->state = MAIN_CP_CB_DATA_STATE_QUORUM;
+ break;
case MAIN_CP_CB_DATA_STATE_NODELIST:
data->state = MAIN_CP_CB_DATA_STATE_NORMAL;
break;
diff --git a/exec/votequorum.c b/exec/votequorum.c
index f7d708f..83f4488 100644
--- a/exec/votequorum.c
+++ b/exec/votequorum.c
@@ -39,8 +39,6 @@
#include <stdint.h>
#include <qb/qbipc_common.h>
-#include <qb/qbdefs.h>
-#include <qb/qbutil.h>
#include "quorum.h"
#include <corosync/corodefs.h>
@@ -64,12 +62,12 @@ static struct corosync_api_v1 *corosync_api;
* votequorum global config vars
*/
-#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
-#define DEFAULT_QDEV_POLL 10000
+#define DEFAULT_QDEVICE_TIMEOUT 10000
-static unsigned int quorumdev_poll = DEFAULT_QDEV_POLL;
-static char quorum_device_name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
-#endif
+static struct cluster_node *qdevice = NULL;
+static unsigned int qdevice_timeout = DEFAULT_QDEVICE_TIMEOUT;
+static uint8_t qdevice_can_operate = 1;
+static uint8_t qdevice_is_registered = 0;
static uint8_t two_node = 0;
@@ -85,21 +83,18 @@ static uint32_t last_man_standing_window = DEFAULT_LMS_WIN;
static uint8_t leave_remove = 0;
static uint32_t ev_barrier = 0;
+
/*
* votequorum_exec defines/structs/forward definitions
*/
struct req_exec_quorum_nodeinfo {
- struct qb_ipc_request_header header __attribute__((aligned(8)));
+ struct qb_ipc_request_header header __attribute__((aligned(8)));
+ uint32_t nodeid;
uint32_t votes;
uint32_t expected_votes;
- uint16_t flags;
- uint8_t quorate;
- uint8_t wait_for_all_status;
- uint8_t first_trans;
- uint8_t _pad0;
- uint8_t _pad1;
- uint8_t _pad2;
+ uint32_t flags;
+ char qdevice_name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
} __attribute__((packed));
struct req_exec_quorum_reconfigure {
@@ -137,10 +132,14 @@ static int votequorum_exec_send_reconfigure(uint8_t param, unsigned int nodeid,
* votequorum internal node status/view
*/
-#define NODE_FLAGS_LEAVING 1
+#define NODE_FLAGS_QUORATE 1
+#define NODE_FLAGS_LEAVING 2
+#define NODE_FLAGS_WFASTATUS 4
+#define NODE_FLAGS_FIRST 8
+#define NODE_FLAGS_QDEVICE 16
+#define NODE_FLAGS_QDEVICE_STATE 32
-#define NODEID_US 0
-#define NODEID_QDEVICE UINT32_MAX
+#define NODEID_QDEVICE 0
typedef enum {
NODESTATE_MEMBER=1,
@@ -149,13 +148,13 @@ typedef enum {
} nodestate_t;
struct cluster_node {
- int node_id;
+ int node_id;
nodestate_t state;
- uint32_t votes;
- uint32_t expected_votes;
- uint16_t flags;
- unsigned long long int last_hello; /* Only used for quorum devices */
- struct list_head list;
+ uint32_t votes;
+ uint32_t expected_votes;
+ uint32_t flags;
+ char qdevice_name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
+ struct list_head list;
};
/*
@@ -164,14 +163,12 @@ struct cluster_node {
static uint8_t quorum;
static uint8_t cluster_is_quorate;
-static uint8_t first_trans = 1;
/*
* votequorum membership data
*/
static struct cluster_node *us;
-static struct cluster_node *quorum_device = NULL;
static struct list_head cluster_members_list;
static unsigned int quorum_members[PROCESSOR_COUNT_MAX+1];
static int quorum_members_entries = 0;
@@ -194,9 +191,8 @@ static struct list_head trackers_list;
* votequorum timers
*/
-#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
-static corosync_timer_handle_t quorum_device_timer;
-#endif
+static corosync_timer_handle_t qdevice_timer;
+static int qdevice_timer_set = 0;
static corosync_timer_handle_t last_man_standing_timer;
static int last_man_standing_timer_set = 0;
@@ -219,6 +215,7 @@ static quorum_set_quorate_fn_t quorum_callback;
static char *votequorum_exec_init_fn (struct corosync_api_v1 *api);
static int votequorum_exec_exit_fn (void);
+static int votequorum_exec_send_nodeinfo(uint32_t nodeid);
static void message_handler_req_exec_votequorum_nodeinfo (
const void *message,
@@ -265,19 +262,20 @@ static void message_handler_req_lib_votequorum_trackstart (void *conn,
static void message_handler_req_lib_votequorum_trackstop (void *conn,
const void *message);
-#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
static void message_handler_req_lib_votequorum_qdevice_register (void *conn,
- const void *message);
+ const void *message);
static void message_handler_req_lib_votequorum_qdevice_unregister (void *conn,
- const void *message);
+ const void *message);
+
+static void message_handler_req_lib_votequorum_qdevice_update (void *conn,
+ const void *message);
static void message_handler_req_lib_votequorum_qdevice_poll (void *conn,
- const void *message);
+ const void *message);
static void message_handler_req_lib_votequorum_qdevice_getinfo (void *conn,
- const void *message);
-#endif
+ const void *message);
static struct corosync_lib_handler quorum_lib_service[] =
{
@@ -300,7 +298,6 @@ static struct corosync_lib_handler quorum_lib_service[] =
{ /* 4 */
.lib_handler_fn = message_handler_req_lib_votequorum_trackstop,
.flow_control = COROSYNC_LIB_FLOW_CONTROL_NOT_REQUIRED
-#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
},
{ /* 5 */
.lib_handler_fn = message_handler_req_lib_votequorum_qdevice_register,
@@ -311,13 +308,16 @@ static struct corosync_lib_handler quorum_lib_service[] =
.flow_control = COROSYNC_LIB_FLOW_CONTROL_NOT_REQUIRED
},
{ /* 7 */
- .lib_handler_fn = message_handler_req_lib_votequorum_qdevice_poll,
+ .lib_handler_fn = message_handler_req_lib_votequorum_qdevice_update,
.flow_control = COROSYNC_LIB_FLOW_CONTROL_NOT_REQUIRED
},
{ /* 8 */
+ .lib_handler_fn = message_handler_req_lib_votequorum_qdevice_poll,
+ .flow_control = COROSYNC_LIB_FLOW_CONTROL_NOT_REQUIRED
+ },
+ { /* 9 */
.lib_handler_fn = message_handler_req_lib_votequorum_qdevice_getinfo,
.flow_control = COROSYNC_LIB_FLOW_CONTROL_NOT_REQUIRED
-#endif
}
};
@@ -399,7 +399,7 @@ static struct cluster_node *allocate_node(unsigned int nodeid)
if (cl) {
memset(cl, 0, sizeof(struct cluster_node));
cl->node_id = nodeid;
- if (nodeid) {
+ if (nodeid != NODEID_QDEVICE) {
node_add_ordered(cl);
}
}
@@ -416,14 +416,14 @@ static struct cluster_node *find_node_by_nodeid(unsigned int nodeid)
ENTER();
- if (nodeid == NODEID_US) {
+ if (nodeid == us->node_id) {
LEAVE();
return us;
}
if (nodeid == NODEID_QDEVICE) {
LEAVE();
- return quorum_device;
+ return qdevice;
}
list_iterate(tmp, &cluster_members_list) {
@@ -483,6 +483,11 @@ static int check_low_node_id_partition(void)
static void update_wait_for_all_status(uint8_t wfa_status)
{
wait_for_all_status = wfa_status;
+ if (wait_for_all_status) {
+ us->flags |= NODE_FLAGS_WFASTATUS;
+ } else {
+ us->flags &= ~NODE_FLAGS_WFASTATUS;
+ }
icmap_set_uint8("runtime.votequorum.wait_for_all_status",
wait_for_all_status);
}
@@ -498,6 +503,12 @@ static void update_ev_barrier(uint32_t expected_votes)
icmap_set_uint32("runtime.votequorum.ev_barrier", ev_barrier);
}
+static void update_qdevice_can_operate(uint8_t status)
+{
+ qdevice_can_operate = status;
+ icmap_set_uint8("runtime.votequorum.qdevice_can_operate", qdevice_can_operate);
+}
+
/*
* quorum calculation core bits
*/
@@ -534,8 +545,13 @@ static int calculate_quorum(int allow_decrease, unsigned int max_expected, unsig
}
}
- if (quorum_device && quorum_device->state == NODESTATE_MEMBER) {
- total_votes += quorum_device->votes;
+ if (qdevice_is_registered) {
+ log_printf(LOGSYS_LEVEL_DEBUG, "node %u state=%d, votes=%u",
+ qdevice->node_id, qdevice->state, qdevice->votes);
+ if (qdevice->state == NODESTATE_MEMBER) {
+ total_votes += qdevice->votes;
+ total_nodes++;
+ }
}
if (max_expected > 0) {
@@ -623,6 +639,11 @@ static void are_we_quorate(unsigned int total_votes)
}
cluster_is_quorate = quorate;
+ if (cluster_is_quorate) {
+ us->flags |= NODE_FLAGS_QUORATE;
+ } else {
+ us->flags &= ~NODE_FLAGS_QUORATE;
+ }
if (wait_for_all) {
if (quorate) {
@@ -640,7 +661,9 @@ static void are_we_quorate(unsigned int total_votes)
LEAVE();
}
-/* Recalculate cluster quorum, set quorate and notify changes */
+/*
+ * Recalculate cluster quorum, set quorate and notify changes
+ */
static void recalculate_quorum(int allow_decrease, int by_current_nodes)
{
unsigned int total_votes = 0;
@@ -660,6 +683,13 @@ static void recalculate_quorum(int allow_decrease, int by_current_nodes)
}
}
+ if (qdevice->votes) {
+ total_votes += qdevice->votes;
+ if (by_current_nodes) {
+ cluster_members++;
+ }
+ }
+
/*
* Keep expected_votes at the highest number of votes in the cluster
*/
@@ -682,12 +712,14 @@ static void recalculate_quorum(int allow_decrease, int by_current_nodes)
*/
static int votequorum_read_nodelist_configuration(uint32_t *votes,
+ uint32_t *nodes,
uint32_t *expected_votes)
{
icmap_iter_t iter;
const char *iter_key;
char tmp_key[ICMAP_KEYNAME_MAXLEN];
uint32_t our_pos, node_pos;
+ uint32_t nodecount = 0;
uint32_t nodelist_expected_votes = 0;
uint32_t node_votes = 0;
int res = 0;
@@ -697,7 +729,7 @@ static int votequorum_read_nodelist_configuration(uint32_t *votes,
if (icmap_get_uint32("nodelist.local_node_pos", &our_pos) != CS_OK) {
log_printf(LOGSYS_LEVEL_DEBUG,
"No nodelist defined or our node is not in the nodelist");
- return -1;
+ return 0;
}
iter = icmap_iter_init("nodelist.node.");
@@ -713,6 +745,8 @@ static int votequorum_read_nodelist_configuration(uint32_t *votes,
continue;
}
+ nodecount++;
+
snprintf(tmp_key, ICMAP_KEYNAME_MAXLEN, "nodelist.node.%u.quorum_votes", node_pos);
if (icmap_get_uint32(tmp_key, &node_votes) != CS_OK) {
node_votes = 1;
@@ -726,30 +760,103 @@ static int votequorum_read_nodelist_configuration(uint32_t *votes,
}
*expected_votes = nodelist_expected_votes;
+ *nodes = nodecount;
icmap_iter_finalize(iter);
LEAVE();
+ return 1;
+}
+
+static int votequorum_qdevice_is_configured(uint32_t *qdevice_votes)
+{
+ char *qdevice_model = NULL;
+
+ ENTER();
+
+ if ((icmap_get_string("quorum.device.model", &qdevice_model) == CS_OK) &&
+ (strlen(qdevice_model))) {
+ free(qdevice_model);
+ if (icmap_get_uint32("quorum.device.votes", qdevice_votes) != CS_OK) {
+ *qdevice_votes = -1;
+ }
+ if (icmap_get_uint32("quorum.device.timeout", &qdevice_timeout) != CS_OK) {
+ qdevice_timeout = DEFAULT_QDEVICE_TIMEOUT;
+ }
+ update_qdevice_can_operate(1);
+ return 1;
+ }
+
+ LEAVE();
+
return 0;
}
/*
* votequorum_readconfig_static is executed before
* votequorum_readconfig_dynamic
+ *
+ * errors from _static are fatal
+ * errors from _dynamic need to be handled at runtime without self-destruction
+ *
+ * TODO: static/dynamic shares a lot of checks _and_
+ * there are probably more options we can change at runtime
+ * than we allow now. Review this in 2.1
*/
static char *votequorum_readconfig_static(void)
{
- uint32_t node_votes, node_expected_votes, expected_votes;
+ uint32_t node_votes = 0, qdevice_votes = 0;
+ uint32_t node_expected_votes = 0, expected_votes = 0;
+ uint32_t node_count = 0;
+ int have_nodelist, have_qdevice;
ENTER();
log_printf(LOGSYS_LEVEL_DEBUG, "Reading static configuration");
+ /*
+ * gather basic data here
+ */
+ icmap_get_uint32("quorum.expected_votes", &expected_votes);
+ have_nodelist = votequorum_read_nodelist_configuration(&node_votes, &node_count, &node_expected_votes);
+ have_qdevice = votequorum_qdevice_is_configured(&qdevice_votes);
icmap_get_uint8("quorum.two_node", &two_node);
update_two_node();
+ /*
+ * do basic config verification
+ */
+ if ((!have_nodelist) && (!expected_votes)) {
+ return ((char *)"configuration error: nodelist or quorum.expected_votes must be configured!");
+ }
+
+ if ((two_node) && (have_qdevice)) {
+ return ((char *)"configuration error: two_node and quorum device cannot be configured at the same time!");
+ }
+
+ if ((expected_votes) && (have_qdevice) && (qdevice_votes == -1)) {
+ return ((char *)"configuration error: quorum.device.votes must be specified when quorum.expected_votes is set");
+ }
+
+ if ((have_qdevice) &&
+ (qdevice_votes == -1) &&
+ (have_nodelist) &&
+ (node_count != node_expected_votes)) {
+ return ((char *)"configuration error: quorum.device.votes must be specified when not all nodes votes 1");
+ }
+
+ if ((qdevice_votes > 0) && (expected_votes)) {
+ int delta = expected_votes - qdevice_votes;
+ if (delta < 2) {
+ return ((char *)"configuration error: quorum.device.votes is too high or expected_votes is too low");
+ }
+ }
+
+ /*
+ * Enable special features
+ */
if (two_node) {
wait_for_all = 1;
}
@@ -760,9 +867,12 @@ static char *votequorum_readconfig_static(void)
icmap_get_uint8("quorum.last_man_standing", &last_man_standing);
icmap_get_uint32("quorum.last_man_standing_window", &last_man_standing_window);
- if ((votequorum_read_nodelist_configuration(&node_votes, &node_expected_votes)) &&
- (icmap_get_uint32("quorum.expected_votes", &expected_votes) != CS_OK)) {
- return ((char *)"configuration error: nodelist or quorum.expected_votes must be configured!");
+ if ((have_qdevice) && (last_man_standing)) {
+ return ((char *)"configuration error: quorum device is not compatible with last_man_standing feature");
+ }
+
+ if ((have_qdevice) && (auto_tie_breaker)) {
+ return ((char *)"configuration error: quorum device is not compatible with auto_tie_breaker feature");
}
if (wait_for_all) {
@@ -776,43 +886,152 @@ static char *votequorum_readconfig_static(void)
static void votequorum_readconfig_dynamic(void)
{
- int cluster_members = 0;
- struct list_head *tmp;
+ uint32_t node_votes = 0, qdevice_votes = 0;
+ uint32_t node_expected_votes = 0, expected_votes = 0;
+ uint32_t node_count = 0;
+ int have_nodelist, have_qdevice;
ENTER();
log_printf(LOGSYS_LEVEL_DEBUG, "Reading dynamic configuration");
- if (votequorum_read_nodelist_configuration(&us->votes, &us->expected_votes)) {
- us->votes = 1;
- icmap_get_uint32("quorum.votes", &us->votes);
+ /*
+ * gather basic data here
+ * TODO: make it common with static!!!
+ */
+ icmap_get_uint32("quorum.expected_votes", &expected_votes);
+ have_nodelist = votequorum_read_nodelist_configuration(&node_votes, &node_count, &node_expected_votes);
+ have_qdevice = votequorum_qdevice_is_configured(&qdevice_votes);
+ icmap_get_uint8("quorum.two_node", &two_node);
+
+ /*
+ * do config verification and enablement
+ */
+
+ if ((!have_nodelist) && (!expected_votes)) {
+ log_printf(LOGSYS_LEVEL_CRIT, "configuration error: nodelist or quorum.expected_votes must be configured!");
+ log_printf(LOGSYS_LEVEL_CRIT, "will continue with current runtime data");
+ goto out;
}
- icmap_get_uint32("quorum.expected_votes", &us->expected_votes);
+ /*
+ * two_node and qdevice are not compatible in the same config.
+ * try to make an educated guess of what to do
+ */
- update_ev_barrier(us->expected_votes);
+ if ((two_node) && (have_qdevice)) {
+ log_printf(LOGSYS_LEVEL_CRIT, "configuration error: two_node and quorum device cannot be configured at the same time!");
+ if (qdevice_is_registered) {
+ log_printf(LOGSYS_LEVEL_CRIT, "quorum device is registered, disabling two_node");
+ two_node = 0;
+ } else {
+ log_printf(LOGSYS_LEVEL_CRIT, "quorum device is not registered, allowing two_node");
+ update_qdevice_can_operate(0);
+ }
+ }
+
+ /*
+ * quorum device is not compatible with last_man_standing and auto_tie_breaker
+ * neither lms or atb can be set at runtime, so there is no need to check for
+ * runtime incompatibilities, but qdevice can be configured _after_ LMS and ATB have
+ * been enabled at startup.
+ */
-#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
- if (icmap_get_uint32("quorum.quorumdev_poll", &quorumdev_poll) != CS_OK) {
- quorumdev_poll = DEFAULT_QDEV_POLL;
+ if ((have_qdevice) && (last_man_standing)) {
+ log_printf(LOGSYS_LEVEL_CRIT, "configuration error: quorum.device is not compatible with last_man_standing");
+ log_printf(LOGSYS_LEVEL_CRIT, "disabling quorum device operations");
+ update_qdevice_can_operate(0);
+ }
+
+ if ((have_qdevice) && (auto_tie_breaker)) {
+ log_printf(LOGSYS_LEVEL_CRIT, "configuration error: quorum.device is not compatible with auto_tie_breaker");
+ log_printf(LOGSYS_LEVEL_CRIT, "disabling quorum device operations");
+ update_qdevice_can_operate(0);
}
-#endif
- icmap_get_uint8("quorum.two_node", &two_node);
- update_two_node();
/*
- * two_node mode is invalid if there are more than 2 nodes in the cluster!
+ * if user specifies quorum.expected_votes + quorum.device but NOT the device.votes
+ * we don't know what the quorum device should vote.
*/
- list_iterate(tmp, &cluster_members_list) {
- cluster_members++;
- }
- if (two_node && cluster_members > 2) {
- log_printf(LOGSYS_LEVEL_WARNING, "quorum.two_node was set but there are more than 2 nodes in the cluster. It will be ignored.");
- two_node = 0;
- update_two_node();
+ if ((expected_votes) && (have_qdevice) && (qdevice_votes == -1)) {
+ log_printf(LOGSYS_LEVEL_CRIT, "configuration error: quorum.device.votes must be specified when quorum.expected_votes is set");
+ log_printf(LOGSYS_LEVEL_CRIT, "disabling quorum device operations");
+ update_qdevice_can_operate(0);
+ }
+
+ /*
+ * if user specifies a node list with uneven votes and no device.votes
+ * we cannot autocalculate the votes
+ */
+
+ if ((have_qdevice) &&
+ (qdevice_votes == -1) &&
+ (have_nodelist) &&
+ (node_count != node_expected_votes)) {
+ log_printf(LOGSYS_LEVEL_CRIT, "configuration error: quorum.device.votes must be specified when not all nodes votes 1");
+ log_printf(LOGSYS_LEVEL_CRIT, "disabling quorum device operations");
+ update_qdevice_can_operate(0);
+ }
+
+ /*
+ * validate quorum device votes vs expected_votes
+ */
+
+ if ((qdevice_votes > 0) && (expected_votes)) {
+ int delta = expected_votes - qdevice_votes;
+ if (delta < 2) {
+ log_printf(LOGSYS_LEVEL_CRIT, "configuration error: quorum.device.votes is too high or expected_votes is too low");
+ log_printf(LOGSYS_LEVEL_CRIT, "disabling quorum device operations");
+ update_qdevice_can_operate(0);
+ }
+ }
+
+ /*
+ * automatically calculate device votes and adjust expected_votes from nodelist
+ */
+
+ if ((have_qdevice) &&
+ (qdevice_votes == -1) &&
+ (!expected_votes) &&
+ (have_nodelist) &&
+ (node_count == node_expected_votes)) {
+ qdevice_votes = node_expected_votes - 1;
+ node_expected_votes = node_expected_votes + qdevice_votes;
+ }
+
+ /*
+ * set this node votes and expected_votes
+ */
+
+ if (have_nodelist) {
+ us->votes = node_votes;
+ us->expected_votes = node_expected_votes;
+ } else {
+ us->votes = 1;
+ icmap_get_uint32("quorum.votes", &us->votes);
+ }
+
+ if (expected_votes) {
+ us->expected_votes = expected_votes;
+ }
+
+ /*
+ * set qdevice votes
+ */
+
+ if (!have_qdevice) {
+ qdevice->votes = 0;
+ }
+
+ if (qdevice_votes != -1) {
+ qdevice->votes = qdevice_votes;
}
+ update_ev_barrier(us->expected_votes);
+ update_two_node();
+
+out:
LEAVE();
}
@@ -825,11 +1044,13 @@ static void votequorum_refresh_config(
{
unsigned int old_votes;
unsigned int old_expected;
+ unsigned int old_qdevice_votes;
ENTER();
old_votes = us->votes;
old_expected = us->expected_votes;
+ old_qdevice_votes = qdevice->votes;
/*
* Reload the configuration
@@ -845,6 +1066,9 @@ static void votequorum_refresh_config(
if (old_expected != us->expected_votes) {
votequorum_exec_send_reconfigure(VOTEQUORUM_RECONFIG_PARAM_EXPECTED_VOTES, us->node_id, us->expected_votes);
}
+ if (old_qdevice_votes != qdevice->votes) {
+ votequorum_exec_send_reconfigure(VOTEQUORUM_RECONFIG_PARAM_NODE_VOTES, NODEID_QDEVICE, qdevice->votes);
+ }
LEAVE();
}
@@ -902,23 +1126,29 @@ static int votequorum_exec_send_reconfigure(uint8_t param, unsigned int nodeid,
return ret;
}
-static int votequorum_exec_send_nodeinfo(void)
+static int votequorum_exec_send_nodeinfo(uint32_t nodeid)
{
struct req_exec_quorum_nodeinfo req_exec_quorum_nodeinfo;
struct iovec iov[1];
+ struct cluster_node *node;
int ret;
ENTER();
- req_exec_quorum_nodeinfo.votes = us->votes;
- req_exec_quorum_nodeinfo.expected_votes = us->expected_votes;
- req_exec_quorum_nodeinfo.flags = us->flags;
- req_exec_quorum_nodeinfo.quorate = cluster_is_quorate;
- req_exec_quorum_nodeinfo.wait_for_all_status = wait_for_all_status;
- req_exec_quorum_nodeinfo.first_trans = first_trans;
- req_exec_quorum_nodeinfo._pad0 = 0;
- req_exec_quorum_nodeinfo._pad1 = 0;
- req_exec_quorum_nodeinfo._pad2 = 0;
+ node = find_node_by_nodeid(nodeid);
+ if (!node) {
+ return -1;
+ }
+
+ req_exec_quorum_nodeinfo.nodeid = nodeid;
+ req_exec_quorum_nodeinfo.votes = node->votes;
+ req_exec_quorum_nodeinfo.expected_votes = node->expected_votes;
+ req_exec_quorum_nodeinfo.flags = node->flags;
+ if (qdevice_is_registered) {
+ strcpy(req_exec_quorum_nodeinfo.qdevice_name, qdevice->qdevice_name);
+ } else {
+ memset(req_exec_quorum_nodeinfo.qdevice_name, 0, VOTEQUORUM_MAX_QDEVICE_NAME_LEN);
+ }
req_exec_quorum_nodeinfo.header.id = SERVICE_ID_MAKE(VOTEQUORUM_SERVICE, MESSAGE_REQ_EXEC_VOTEQUORUM_NODEINFO);
req_exec_quorum_nodeinfo.header.size = sizeof(req_exec_quorum_nodeinfo);
@@ -948,7 +1178,7 @@ static int votequorum_exec_send_quorum_notification(void *conn, uint64_t context
node = list_entry(tmp, struct cluster_node, list);
cluster_members++;
}
- if (quorum_device) {
+ if (qdevice_is_registered) {
cluster_members++;
}
@@ -968,9 +1198,9 @@ static int votequorum_exec_send_quorum_notification(void *conn, uint64_t context
res_lib_votequorum_notification->node_list[i].nodeid = node->node_id;
res_lib_votequorum_notification->node_list[i++].state = node->state;
}
- if (quorum_device) {
- res_lib_votequorum_notification->node_list[i].nodeid = 0;
- res_lib_votequorum_notification->node_list[i++].state = quorum_device->state | 0x80;
+ if (qdevice_is_registered) {
+ res_lib_votequorum_notification->node_list[i].nodeid = NODEID_QDEVICE;
+ res_lib_votequorum_notification->node_list[i++].state = qdevice->state;
}
res_lib_votequorum_notification->header.id = MESSAGE_RES_VOTEQUORUM_NOTIFICATION;
res_lib_votequorum_notification->header.size = size;
@@ -1027,16 +1257,17 @@ static void exec_votequorum_nodeinfo_endian_convert (void *message)
ENTER();
+ nodeinfo->nodeid = swab32(nodeinfo->nodeid);
nodeinfo->votes = swab32(nodeinfo->votes);
nodeinfo->expected_votes = swab32(nodeinfo->expected_votes);
- nodeinfo->flags = swab16(nodeinfo->flags);
+ nodeinfo->flags = swab32(nodeinfo->flags);
LEAVE();
}
static void message_handler_req_exec_votequorum_nodeinfo (
const void *message,
- unsigned int nodeid)
+ unsigned int sender_nodeid)
{
const struct req_exec_quorum_nodeinfo *req_exec_quorum_nodeinfo = message;
struct cluster_node *node = NULL;
@@ -1047,16 +1278,15 @@ static void message_handler_req_exec_votequorum_nodeinfo (
int new_node = 0;
int allow_downgrade = 0;
int by_node = 0;
+ unsigned int nodeid = req_exec_quorum_nodeinfo->nodeid;
ENTER();
- log_printf(LOGSYS_LEVEL_DEBUG, "got nodeinfo message from cluster node %u", nodeid);
- log_printf(LOGSYS_LEVEL_DEBUG, "nodeinfo message[%u]: votes: %d, expected: %d wfa: %d quorate: %d flags: %d",
+ log_printf(LOGSYS_LEVEL_DEBUG, "got nodeinfo message from cluster node %u", sender_nodeid);
+ log_printf(LOGSYS_LEVEL_DEBUG, "nodeinfo message[%u]: votes: %d, expected: %d flags: %d",
nodeid,
req_exec_quorum_nodeinfo->votes,
req_exec_quorum_nodeinfo->expected_votes,
- req_exec_quorum_nodeinfo->wait_for_all_status,
- req_exec_quorum_nodeinfo->quorate,
req_exec_quorum_nodeinfo->flags);
node = find_node_by_nodeid(nodeid);
@@ -1083,41 +1313,74 @@ static void message_handler_req_exec_votequorum_nodeinfo (
}
/* Update node state */
- node->votes = req_exec_quorum_nodeinfo->votes;
node->flags = req_exec_quorum_nodeinfo->flags;
+ if (nodeid != NODEID_QDEVICE) {
+ node->votes = req_exec_quorum_nodeinfo->votes;
+
+ memset(node->qdevice_name, 0, VOTEQUORUM_MAX_QDEVICE_NAME_LEN);
+ if (node->flags & NODE_FLAGS_QDEVICE) {
+ int state = 0;
+
+ if (node->flags & NODE_FLAGS_QDEVICE_STATE) {
+ state = 1;
+ }
+
+ strcpy(node->qdevice_name, req_exec_quorum_nodeinfo->qdevice_name);
+ log_printf(LOGSYS_LEVEL_DEBUG, "Got qdevice data from node[%u]: state %u name %s",
+ node->node_id, state, node->qdevice_name);
+ }
+ } else {
+ if ((!cluster_is_quorate) &&
+ (req_exec_quorum_nodeinfo->flags & NODE_FLAGS_QUORATE)) {
+ node->votes = req_exec_quorum_nodeinfo->votes;
+ } else {
+ node->votes = max(node->votes, req_exec_quorum_nodeinfo->votes);
+ }
+ }
+
if (node->flags & NODE_FLAGS_LEAVING) {
node->state = NODESTATE_LEAVING;
allow_downgrade = 1;
by_node = 1;
} else {
- node->state = NODESTATE_MEMBER;
+ if (nodeid != NODEID_QDEVICE) {
+ node->state = NODESTATE_MEMBER;
+ } else {
+ /*
+ * qdevice status is only local to the node
+ */
+ node->state = old_state;
+ }
}
- if ((!cluster_is_quorate) &&
- (req_exec_quorum_nodeinfo->quorate)) {
- allow_downgrade = 1;
- us->expected_votes = req_exec_quorum_nodeinfo->expected_votes;
- }
+ if (nodeid != NODEID_QDEVICE) {
+ if ((!cluster_is_quorate) &&
+ (req_exec_quorum_nodeinfo->flags & NODE_FLAGS_QUORATE)) {
+ allow_downgrade = 1;
+ us->expected_votes = req_exec_quorum_nodeinfo->expected_votes;
+ }
- if (req_exec_quorum_nodeinfo->quorate) {
- node->expected_votes = req_exec_quorum_nodeinfo->expected_votes;
- } else {
- node->expected_votes = us->expected_votes;
- }
+ if (req_exec_quorum_nodeinfo->flags & NODE_FLAGS_QUORATE) {
+ node->expected_votes = req_exec_quorum_nodeinfo->expected_votes;
+ } else {
+ node->expected_votes = us->expected_votes;
+ }
- if ((last_man_standing) && (req_exec_quorum_nodeinfo->votes > 1)) {
- log_printf(LOGSYS_LEVEL_WARNING, "Last Man Standing feature is supported only when all"
- "cluster nodes votes are set to 1. Disabling LMS.");
- last_man_standing = 0;
- if (last_man_standing_timer_set) {
- corosync_api->timer_delete(last_man_standing_timer);
- last_man_standing_timer_set = 0;
+ if ((last_man_standing) && (req_exec_quorum_nodeinfo->votes > 1)) {
+ log_printf(LOGSYS_LEVEL_WARNING, "Last Man Standing feature is supported only when all"
+ "cluster nodes votes are set to 1. Disabling LMS.");
+ last_man_standing = 0;
+ if (last_man_standing_timer_set) {
+ corosync_api->timer_delete(last_man_standing_timer);
+ last_man_standing_timer_set = 0;
+ }
}
}
if (new_node ||
- req_exec_quorum_nodeinfo->first_trans ||
+ nodeid == NODEID_QDEVICE ||
+ req_exec_quorum_nodeinfo->flags & NODE_FLAGS_FIRST ||
old_votes != node->votes ||
old_expected != node->expected_votes ||
old_flags != node->flags ||
@@ -1126,8 +1389,8 @@ static void message_handler_req_exec_votequorum_nodeinfo (
}
if ((wait_for_all) &&
- (!req_exec_quorum_nodeinfo->wait_for_all_status) &&
- (req_exec_quorum_nodeinfo->quorate)) {
+ (!(req_exec_quorum_nodeinfo->flags & NODE_FLAGS_WFASTATUS)) &&
+ (req_exec_quorum_nodeinfo->flags & NODE_FLAGS_QUORATE)) {
update_wait_for_all_status(0);
}
@@ -1156,7 +1419,8 @@ static void message_handler_req_exec_votequorum_reconfigure (
ENTER();
- log_printf(LOGSYS_LEVEL_DEBUG, "got reconfigure message from cluster node %u", nodeid);
+ log_printf(LOGSYS_LEVEL_DEBUG, "got reconfigure message from cluster node %u for %u",
+ nodeid, req_exec_quorum_reconfigure->nodeid);
node = find_node_by_nodeid(req_exec_quorum_reconfigure->nodeid);
if (!node) {
@@ -1196,7 +1460,7 @@ static int votequorum_exec_exit_fn (void)
if (leave_remove) {
us->flags |= NODE_FLAGS_LEAVING;
- ret = votequorum_exec_send_nodeinfo();
+ ret = votequorum_exec_send_nodeinfo(us->node_id);
}
LEAVE();
@@ -1227,6 +1491,16 @@ static char *votequorum_exec_init_fn (struct corosync_api_v1 *api)
us->state = NODESTATE_MEMBER;
us->votes = 1;
+ us->flags |= NODE_FLAGS_FIRST;
+
+ qdevice = allocate_node(NODEID_QDEVICE);
+ if (!qdevice) {
+ LEAVE();
+ return ((char *)"Could not allocate node.");
+ }
+ qdevice->state = NODESTATE_DEAD;
+ qdevice->votes = 0;
+ memset(qdevice->qdevice_name, 0, VOTEQUORUM_MAX_QDEVICE_NAME_LEN);
votequorum_readconfig_dynamic();
recalculate_quorum(0, 0);
@@ -1239,7 +1513,7 @@ static char *votequorum_exec_init_fn (struct corosync_api_v1 *api)
/*
* Start us off with one node
*/
- votequorum_exec_send_nodeinfo();
+ votequorum_exec_send_nodeinfo(us->node_id);
LEAVE();
@@ -1275,7 +1549,7 @@ static void votequorum_confchg_fn (
ENTER();
if (member_list_entries > 1) {
- first_trans = 0;
+ us->flags &= ~NODE_FLAGS_FIRST;
}
if (left_list_entries) {
@@ -1304,10 +1578,8 @@ static void votequorum_confchg_fn (
if (member_list_entries) {
memcpy(quorum_members, member_list, sizeof(unsigned int) * member_list_entries);
quorum_members_entries = member_list_entries;
- if (quorum_device) {
- quorum_members[quorum_members_entries++] = 0;
- }
- votequorum_exec_send_nodeinfo();
+ votequorum_exec_send_nodeinfo(us->node_id);
+ votequorum_exec_send_nodeinfo(NODEID_QDEVICE);
}
memcpy(&quorum_ringid, ring_id, sizeof(*ring_id));
@@ -1392,29 +1664,26 @@ static int quorum_lib_exit_fn (void *conn)
* library internal functions
*/
-#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
-static void quorum_device_timer_fn(void *arg)
+static void qdevice_timer_fn(void *arg)
{
ENTER();
- if (!quorum_device || quorum_device->state == NODESTATE_DEAD) {
+ if ((!qdevice_is_registered) ||
+ (qdevice->state == NODESTATE_DEAD) ||
+ (!qdevice_timer_set)) {
LEAVE();
return;
}
- if ((quorum_device->last_hello / QB_TIME_NS_IN_SEC) + quorumdev_poll/1000 <
- (qb_util_nano_current_get () / QB_TIME_NS_IN_SEC)) {
- quorum_device->state = NODESTATE_DEAD;
- log_printf(LOGSYS_LEVEL_INFO, "lost contact with quorum device");
- recalculate_quorum(0, 0);
- } else {
- corosync_api->timer_add_duration((unsigned long long)quorumdev_poll*1000000, quorum_device,
- quorum_device_timer_fn, &quorum_device_timer);
- }
+ qdevice->state = NODESTATE_DEAD;
+ us->flags &= ~NODE_FLAGS_QDEVICE_STATE;
+ log_printf(LOGSYS_LEVEL_INFO, "lost contact with quorum device %s", qdevice->qdevice_name);
+ votequorum_exec_send_nodeinfo(us->node_id);
+
+ qdevice_timer_set = 0;
LEAVE();
}
-#endif
/*
* Library Handler Functions
@@ -1448,12 +1717,14 @@ static void message_handler_req_lib_votequorum_getinfo (void *conn, const void *
}
}
- if (quorum_device && quorum_device->state == NODESTATE_MEMBER) {
- total_votes += quorum_device->votes;
+ if (((qdevice_is_registered) && (qdevice->state == NODESTATE_MEMBER)) ||
+ ((node->flags & NODE_FLAGS_QDEVICE) && (node->flags & NODE_FLAGS_QDEVICE_STATE))) {
+ total_votes += qdevice->votes;
}
- res_lib_votequorum_getinfo.votes = us->votes;
- res_lib_votequorum_getinfo.expected_votes = us->expected_votes;
+ res_lib_votequorum_getinfo.state = node->state;
+ res_lib_votequorum_getinfo.votes = node->votes;
+ res_lib_votequorum_getinfo.expected_votes = node->expected_votes;
res_lib_votequorum_getinfo.highest_expected = highest_expected;
res_lib_votequorum_getinfo.quorum = quorum;
@@ -1479,6 +1750,9 @@ static void message_handler_req_lib_votequorum_getinfo (void *conn, const void *
if (leave_remove) {
res_lib_votequorum_getinfo.flags |= VOTEQUORUM_INFO_LEAVE_REMOVE;
}
+ if (node->flags & NODE_FLAGS_QDEVICE) {
+ res_lib_votequorum_getinfo.flags |= VOTEQUORUM_INFO_QDEVICE;
+ }
} else {
error = CS_ERR_NOT_EXIST;
}
@@ -1564,10 +1838,6 @@ static void message_handler_req_lib_votequorum_setvotes (void *conn, const void
goto error_exit;
}
- if (!nodeid) {
- nodeid = corosync_api->totem_nodeid_get();
- }
-
votequorum_exec_send_reconfigure(VOTEQUORUM_RECONFIG_PARAM_NODE_VOTES, nodeid,
req_lib_votequorum_setvotes->votes);
@@ -1645,9 +1915,8 @@ static void message_handler_req_lib_votequorum_trackstop (void *conn,
LEAVE();
}
-#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
static void message_handler_req_lib_votequorum_qdevice_register (void *conn,
- const void *message)
+ const void *message)
{
const struct req_lib_votequorum_qdevice_register *req_lib_votequorum_qdevice_register = message;
struct res_lib_votequorum_status res_lib_votequorum_status;
@@ -1655,16 +1924,33 @@ static void message_handler_req_lib_votequorum_qdevice_register (void *conn,
ENTER();
- if (quorum_device) {
- error = CS_ERR_EXIST;
+ if (!qdevice_can_operate) {
+ log_printf(LOGSYS_LEVEL_INFO, "Registration of quorum device is disabled by incorrect corosync.conf. See logs for more information");
+ error = CS_ERR_ACCESS;
+ goto out;
+ }
+
+ if (qdevice_is_registered) {
+ if ((!strncmp(req_lib_votequorum_qdevice_register->name,
+ qdevice->qdevice_name, VOTEQUORUM_MAX_QDEVICE_NAME_LEN))) {
+ goto out;
+ } else {
+ log_printf(LOGSYS_LEVEL_WARNING,
+ "A new qdevice with different name (new: %s old: %s) is trying to re-register!",
+ req_lib_votequorum_qdevice_register->name, qdevice->qdevice_name);
+ error = CS_ERR_EXIST;
+ goto out;
+ }
} else {
- quorum_device = allocate_node(0);
- quorum_device->state = NODESTATE_DEAD;
- quorum_device->votes = req_lib_votequorum_qdevice_register->votes;
- strcpy(quorum_device_name, req_lib_votequorum_qdevice_register->name);
- list_add(&quorum_device->list, &cluster_members_list);
+ qdevice_is_registered = 1;
+ strcpy(qdevice->qdevice_name, req_lib_votequorum_qdevice_register->name);
+ us->flags |= NODE_FLAGS_QDEVICE;
+ votequorum_exec_send_nodeinfo(NODEID_QDEVICE);
+ votequorum_exec_send_nodeinfo(us->node_id);
}
+out:
+
res_lib_votequorum_status.header.size = sizeof(res_lib_votequorum_status);
res_lib_votequorum_status.header.id = MESSAGE_RES_VOTEQUORUM_STATUS;
res_lib_votequorum_status.header.error = error;
@@ -1674,24 +1960,62 @@ static void message_handler_req_lib_votequorum_qdevice_register (void *conn,
}
static void message_handler_req_lib_votequorum_qdevice_unregister (void *conn,
- const void *message)
+ const void *message)
{
+ const struct req_lib_votequorum_qdevice_unregister *req_lib_votequorum_qdevice_unregister = message;
struct res_lib_votequorum_status res_lib_votequorum_status;
cs_error_t error = CS_OK;
ENTER();
- if (quorum_device) {
- struct cluster_node *node = quorum_device;
+ if (qdevice_is_registered) {
+ if (strncmp(req_lib_votequorum_qdevice_unregister->name, qdevice->qdevice_name, VOTEQUORUM_MAX_QDEVICE_NAME_LEN)) {
+ error = CS_ERR_INVALID_PARAM;
+ goto out;
+ }
+ if (qdevice_timer_set) {
+ corosync_api->timer_delete(qdevice_timer);
+ qdevice_timer_set = 0;
+ }
+ qdevice_is_registered = 0;
+ us->flags &= ~NODE_FLAGS_QDEVICE;
+ us->flags &= ~NODE_FLAGS_QDEVICE_STATE;
+ votequorum_exec_send_nodeinfo(us->node_id);
+ } else {
+ error = CS_ERR_NOT_EXIST;
+ }
- quorum_device = NULL;
- list_del(&node->list);
- free(node);
- recalculate_quorum(0, 0);
+out:
+ res_lib_votequorum_status.header.size = sizeof(res_lib_votequorum_status);
+ res_lib_votequorum_status.header.id = MESSAGE_RES_VOTEQUORUM_STATUS;
+ res_lib_votequorum_status.header.error = error;
+ corosync_api->ipc_response_send(conn, &res_lib_votequorum_status, sizeof(res_lib_votequorum_status));
+
+ LEAVE();
+}
+
+static void message_handler_req_lib_votequorum_qdevice_update (void *conn,
+ const void *message)
+{
+ const struct req_lib_votequorum_qdevice_update *req_lib_votequorum_qdevice_update = message;
+ struct res_lib_votequorum_status res_lib_votequorum_status;
+ cs_error_t error = CS_OK;
+
+ ENTER();
+
+ if (qdevice_is_registered) {
+ if (strncmp(req_lib_votequorum_qdevice_update->oldname, qdevice->qdevice_name, VOTEQUORUM_MAX_QDEVICE_NAME_LEN)) {
+ error = CS_ERR_INVALID_PARAM;
+ goto out;
+ }
+ memset(qdevice->qdevice_name, 0, VOTEQUORUM_MAX_QDEVICE_NAME_LEN);
+ strcpy(qdevice->qdevice_name, req_lib_votequorum_qdevice_update->newname);
+ votequorum_exec_send_nodeinfo(us->node_id);
} else {
error = CS_ERR_NOT_EXIST;
}
+out:
res_lib_votequorum_status.header.size = sizeof(res_lib_votequorum_status);
res_lib_votequorum_status.header.id = MESSAGE_RES_VOTEQUORUM_STATUS;
res_lib_votequorum_status.header.error = error;
@@ -1701,7 +2025,7 @@ static void message_handler_req_lib_votequorum_qdevice_unregister (void *conn,
}
static void message_handler_req_lib_votequorum_qdevice_poll (void *conn,
- const void *message)
+ const void *message)
{
const struct req_lib_votequorum_qdevice_poll *req_lib_votequorum_qdevice_poll = message;
struct res_lib_votequorum_status res_lib_votequorum_status;
@@ -1709,30 +2033,43 @@ static void message_handler_req_lib_votequorum_qdevice_poll (void *conn,
ENTER();
- if (quorum_device) {
- if (req_lib_votequorum_qdevice_poll->state) {
- quorum_device->last_hello = qb_util_nano_current_get ();
- if (quorum_device->state == NODESTATE_DEAD) {
- quorum_device->state = NODESTATE_MEMBER;
- recalculate_quorum(0, 0);
+ if (!qdevice_can_operate) {
+ error = CS_ERR_ACCESS;
+ goto out;
+ }
- corosync_api->timer_add_duration((unsigned long long)quorumdev_poll*1000000, quorum_device,
- quorum_device_timer_fn, &quorum_device_timer);
+ if (qdevice_is_registered) {
+ if (strncmp(req_lib_votequorum_qdevice_poll->name, qdevice->qdevice_name, VOTEQUORUM_MAX_QDEVICE_NAME_LEN)) {
+ error = CS_ERR_INVALID_PARAM;
+ goto out;
+ }
+ if (qdevice_timer_set) {
+ corosync_api->timer_delete(qdevice_timer);
+ qdevice_timer_set = 0;
+ }
+
+ if (req_lib_votequorum_qdevice_poll->state) {
+ if (qdevice->state == NODESTATE_DEAD) {
+ qdevice->state = NODESTATE_MEMBER;
+ us->flags |= NODE_FLAGS_QDEVICE_STATE;
+ votequorum_exec_send_nodeinfo(us->node_id);
}
} else {
- if (quorum_device->state == NODESTATE_MEMBER) {
- quorum_device->state = NODESTATE_DEAD;
- recalculate_quorum(0, 0);
- corosync_api->timer_delete(quorum_device_timer);
+ if (qdevice->state == NODESTATE_MEMBER) {
+ qdevice->state = NODESTATE_DEAD;
+ us->flags &= ~NODE_FLAGS_QDEVICE_STATE;
+ votequorum_exec_send_nodeinfo(us->node_id);
}
}
+
+ corosync_api->timer_add_duration((unsigned long long)qdevice_timeout*1000000, qdevice,
+ qdevice_timer_fn, &qdevice_timer);
+ qdevice_timer_set = 1;
} else {
error = CS_ERR_NOT_EXIST;
}
- /*
- * send status
- */
+out:
res_lib_votequorum_status.header.size = sizeof(res_lib_votequorum_status);
res_lib_votequorum_status.header.id = MESSAGE_RES_VOTEQUORUM_STATUS;
res_lib_votequorum_status.header.error = error;
@@ -1742,24 +2079,50 @@ static void message_handler_req_lib_votequorum_qdevice_poll (void *conn,
}
static void message_handler_req_lib_votequorum_qdevice_getinfo (void *conn,
- const void *message)
+ const void *message)
{
+ const struct req_lib_votequorum_qdevice_getinfo *req_lib_votequorum_qdevice_getinfo = message;
struct res_lib_votequorum_qdevice_getinfo res_lib_votequorum_qdevice_getinfo;
cs_error_t error = CS_OK;
+ struct cluster_node *node;
+ uint32_t nodeid = req_lib_votequorum_qdevice_getinfo->nodeid;
ENTER();
- if (quorum_device) {
- log_printf(LOGSYS_LEVEL_DEBUG, "got qdevice_getinfo state %d", quorum_device->state);
- res_lib_votequorum_qdevice_getinfo.votes = quorum_device->votes;
- if (quorum_device->state == NODESTATE_MEMBER) {
- res_lib_votequorum_qdevice_getinfo.state = 1;
+ if ((nodeid != us->node_id) &&
+ (nodeid != NODEID_QDEVICE)) {
+ node = find_node_by_nodeid(req_lib_votequorum_qdevice_getinfo->nodeid);
+ if ((node) &&
+ (node->flags & NODE_FLAGS_QDEVICE)) {
+ int state = 0;
+
+ if (node->flags & NODE_FLAGS_QDEVICE_STATE) {
+ state = 1;
+ }
+ log_printf(LOGSYS_LEVEL_DEBUG, "got qdevice_getinfo node %u state %d", nodeid, state);
+ res_lib_votequorum_qdevice_getinfo.votes = qdevice->votes;
+ if (state) {
+ res_lib_votequorum_qdevice_getinfo.state = 1;
+ } else {
+ res_lib_votequorum_qdevice_getinfo.state = 0;
+ }
+ strcpy(res_lib_votequorum_qdevice_getinfo.name, node->qdevice_name);
} else {
- res_lib_votequorum_qdevice_getinfo.state = 0;
+ error = CS_ERR_NOT_EXIST;
}
- strcpy(res_lib_votequorum_qdevice_getinfo.name, quorum_device_name);
} else {
- error = CS_ERR_NOT_EXIST;
+ if (qdevice_is_registered) {
+ log_printf(LOGSYS_LEVEL_DEBUG, "got qdevice_getinfo node %u state %d", us->node_id, qdevice->state);
+ res_lib_votequorum_qdevice_getinfo.votes = qdevice->votes;
+ if (qdevice->state == NODESTATE_MEMBER) {
+ res_lib_votequorum_qdevice_getinfo.state = 1;
+ } else {
+ res_lib_votequorum_qdevice_getinfo.state = 0;
+ }
+ strcpy(res_lib_votequorum_qdevice_getinfo.name, qdevice->qdevice_name);
+ } else {
+ error = CS_ERR_NOT_EXIST;
+ }
}
res_lib_votequorum_qdevice_getinfo.header.size = sizeof(res_lib_votequorum_qdevice_getinfo);
@@ -1769,4 +2132,3 @@ static void message_handler_req_lib_votequorum_qdevice_getinfo (void *conn,
LEAVE();
}
-#endif
diff --git a/include/corosync/ipc_votequorum.h b/include/corosync/ipc_votequorum.h
index e63bbc5..b633ece 100644
--- a/include/corosync/ipc_votequorum.h
+++ b/include/corosync/ipc_votequorum.h
@@ -48,6 +48,7 @@ enum req_votequorum_types {
,
MESSAGE_REQ_VOTEQUORUM_QDEVICE_REGISTER,
MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER,
+ MESSAGE_REQ_VOTEQUORUM_QDEVICE_UPDATE,
MESSAGE_REQ_VOTEQUORUM_QDEVICE_POLL,
MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO
#endif
@@ -62,23 +63,39 @@ enum res_votequorum_types {
MESSAGE_RES_VOTEQUORUM_EXPECTEDVOTES_NOTIFICATION
};
-struct req_lib_votequorum_setvotes {
+struct req_lib_votequorum_qdevice_register {
struct qb_ipc_request_header header __attribute__((aligned(8)));
- unsigned int votes;
- int nodeid;
+ char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
};
-struct req_lib_votequorum_qdevice_register {
+struct req_lib_votequorum_qdevice_unregister {
struct qb_ipc_request_header header __attribute__((aligned(8)));
- unsigned int votes;
char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
};
+struct req_lib_votequorum_qdevice_update {
+ struct qb_ipc_request_header header __attribute__((aligned(8)));
+ char oldname[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
+ char newname[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
+};
+
struct req_lib_votequorum_qdevice_poll {
struct qb_ipc_request_header header __attribute__((aligned(8)));
+ char name[VOTEQUORUM_MAX_QDEVICE_NAME_LEN];
int state;
};
+struct req_lib_votequorum_qdevice_getinfo {
+ struct qb_ipc_request_header header __attribute__((aligned(8)));
+ int nodeid;
+};
+
+struct req_lib_votequorum_setvotes {
+ struct qb_ipc_request_header header __attribute__((aligned(8)));
+ unsigned int votes;
+ int nodeid;
+};
+
struct req_lib_votequorum_setexpected {
struct qb_ipc_request_header header __attribute__((aligned(8)));
unsigned int expected_votes;
@@ -109,10 +126,12 @@ struct res_lib_votequorum_status {
#define VOTEQUORUM_INFO_LAST_MAN_STANDING 8
#define VOTEQUORUM_INFO_AUTO_TIE_BREAKER 16
#define VOTEQUORUM_INFO_LEAVE_REMOVE 32
+#define VOTEQUORUM_INFO_QDEVICE 64
struct res_lib_votequorum_getinfo {
struct qb_ipc_response_header header __attribute__((aligned(8)));
- int nodeid;
+ unsigned int nodeid;
+ unsigned int state;
unsigned int votes;
unsigned int expected_votes;
unsigned int highest_expected;
diff --git a/include/corosync/votequorum.h b/include/corosync/votequorum.h
index 8173c45..98b7b76 100644
--- a/include/corosync/votequorum.h
+++ b/include/corosync/votequorum.h
@@ -42,19 +42,16 @@ extern "C" {
typedef uint64_t votequorum_handle_t;
-#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
-#define VOTEQUORUM_MAX_QDEVICE_NAME_LEN 255
-#endif
-
#define VOTEQUORUM_INFO_FLAG_TWONODE 1
#define VOTEQUORUM_INFO_FLAG_QUORATE 2
#define VOTEQUORUM_INFO_WAIT_FOR_ALL 4
#define VOTEQUORUM_INFO_LAST_MAN_STANDING 8
#define VOTEQUORUM_INFO_AUTO_TIE_BREAKER 16
#define VOTEQUORUM_INFO_LEAVE_REMOVE 32
+#define VOTEQUORUM_INFO_QDEVICE 64
-#define VOTEQUORUM_NODEID_US 0
-#define VOTEQUORUM_NODEID_QDEVICE -1
+#define VOTEQUORUM_NODEID_QDEVICE 0
+#define VOTEQUORUM_MAX_QDEVICE_NAME_LEN 255
#define NODESTATE_MEMBER 1
#define NODESTATE_DEAD 2
@@ -64,6 +61,7 @@ typedef uint64_t votequorum_handle_t;
struct votequorum_info {
unsigned int node_id;
+ unsigned int state;
unsigned int node_votes;
unsigned int node_expected_votes;
unsigned int highest_expected;
@@ -187,20 +185,29 @@ cs_error_t votequorum_context_set (
*/
cs_error_t votequorum_qdevice_register (
votequorum_handle_t handle,
- const char *name,
- unsigned int votes);
+ const char *name);
/**
* Unregister a quorum device
*/
cs_error_t votequorum_qdevice_unregister (
- votequorum_handle_t handle);
+ votequorum_handle_t handle,
+ const char *name);
+
+/**
+ * Update registered name of a quorum device
+ */
+cs_error_t votequorum_qdevice_update (
+ votequorum_handle_t handle,
+ const char *oldname,
+ const char *newname);
/**
* Poll a quorum device
*/
cs_error_t votequorum_qdevice_poll (
votequorum_handle_t handle,
+ const char *name,
unsigned int state);
/**
@@ -208,6 +215,7 @@ cs_error_t votequorum_qdevice_poll (
*/
cs_error_t votequorum_qdevice_getinfo (
votequorum_handle_t handle,
+ unsigned int nodeid,
struct votequorum_qdevice_info *info);
#endif
diff --git a/lib/votequorum.c b/lib/votequorum.c
index 32f5a6b..133e706 100644
--- a/lib/votequorum.c
+++ b/lib/votequorum.c
@@ -174,6 +174,7 @@ cs_error_t votequorum_getinfo (
error = res_lib_votequorum_getinfo.header.error;
info->node_id = res_lib_votequorum_getinfo.nodeid;
+ info->state = res_lib_votequorum_getinfo.state;
info->node_votes = res_lib_votequorum_getinfo.votes;
info->node_expected_votes = res_lib_votequorum_getinfo.expected_votes;
info->highest_expected = res_lib_votequorum_getinfo.highest_expected;
@@ -539,8 +540,7 @@ error_put:
#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
cs_error_t votequorum_qdevice_register (
votequorum_handle_t handle,
- const char *name,
- unsigned int votes)
+ const char *name)
{
cs_error_t error;
struct votequorum_inst *votequorum_inst;
@@ -548,8 +548,9 @@ cs_error_t votequorum_qdevice_register (
struct req_lib_votequorum_qdevice_register req_lib_votequorum_qdevice_register;
struct res_lib_votequorum_status res_lib_votequorum_status;
- if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN)
+ if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
return CS_ERR_INVALID_PARAM;
+ }
error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
if (error != CS_OK) {
@@ -560,7 +561,6 @@ cs_error_t votequorum_qdevice_register (
req_lib_votequorum_qdevice_register.header.size = sizeof (struct req_lib_votequorum_qdevice_register);
req_lib_votequorum_qdevice_register.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_REGISTER;
strcpy(req_lib_votequorum_qdevice_register.name, name);
- req_lib_votequorum_qdevice_register.votes = votes;
iov.iov_base = (char *)&req_lib_votequorum_qdevice_register;
iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_register);
@@ -586,6 +586,7 @@ error_exit:
cs_error_t votequorum_qdevice_poll (
votequorum_handle_t handle,
+ const char *name,
unsigned int state)
{
cs_error_t error;
@@ -594,14 +595,18 @@ cs_error_t votequorum_qdevice_poll (
struct req_lib_votequorum_qdevice_poll req_lib_votequorum_qdevice_poll;
struct res_lib_votequorum_status res_lib_votequorum_status;
+ if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
+ return CS_ERR_INVALID_PARAM;
+ }
+
error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
if (error != CS_OK) {
return (error);
}
-
req_lib_votequorum_qdevice_poll.header.size = sizeof (struct req_lib_votequorum_qdevice_poll);
req_lib_votequorum_qdevice_poll.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_POLL;
+ strcpy(req_lib_votequorum_qdevice_poll.name, name);
req_lib_votequorum_qdevice_poll.state = state;
iov.iov_base = (char *)&req_lib_votequorum_qdevice_poll;
@@ -627,24 +632,30 @@ error_exit:
}
cs_error_t votequorum_qdevice_unregister (
- votequorum_handle_t handle)
+ votequorum_handle_t handle,
+ const char *name)
{
cs_error_t error;
struct votequorum_inst *votequorum_inst;
struct iovec iov;
- struct req_lib_votequorum_general req_lib_votequorum_general;
+ struct req_lib_votequorum_qdevice_unregister req_lib_votequorum_qdevice_unregister;
struct res_lib_votequorum_status res_lib_votequorum_status;
+ if (strlen(name) >= VOTEQUORUM_MAX_QDEVICE_NAME_LEN) {
+ return CS_ERR_INVALID_PARAM;
+ }
+
error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
if (error != CS_OK) {
return (error);
}
- req_lib_votequorum_general.header.size = sizeof (struct req_lib_votequorum_general);
- req_lib_votequorum_general.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER;
+ req_lib_votequorum_qdevice_unregister.header.size = sizeof (struct req_lib_votequorum_qdevice_unregister);
+ req_lib_votequorum_qdevice_unregister.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_UNREGISTER;
+ strcpy(req_lib_votequorum_qdevice_unregister.name, name);
- iov.iov_base = (char *)&req_lib_votequorum_general;
- iov.iov_len = sizeof (struct req_lib_votequorum_general);
+ iov.iov_base = (char *)&req_lib_votequorum_qdevice_unregister;
+ iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_unregister);
error = qb_to_cs_error(qb_ipcc_sendv_recv (
votequorum_inst->c,
@@ -667,12 +678,13 @@ error_exit:
cs_error_t votequorum_qdevice_getinfo (
votequorum_handle_t handle,
+ unsigned int nodeid,
struct votequorum_qdevice_info *qinfo)
{
cs_error_t error;
struct votequorum_inst *votequorum_inst;
struct iovec iov;
- struct req_lib_votequorum_general req_lib_votequorum_general;
+ struct req_lib_votequorum_qdevice_getinfo req_lib_votequorum_qdevice_getinfo;
struct res_lib_votequorum_qdevice_getinfo res_lib_votequorum_qdevice_getinfo;
error = hdb_error_to_cs(hdb_handle_get (&votequorum_handle_t_db, handle, (void *)&votequorum_inst));
@@ -680,12 +692,12 @@ cs_error_t votequorum_qdevice_getinfo (
return (error);
}
+ req_lib_votequorum_qdevice_getinfo.header.size = sizeof (struct req_lib_votequorum_qdevice_getinfo);
+ req_lib_votequorum_qdevice_getinfo.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO;
+ req_lib_votequorum_qdevice_getinfo.nodeid = nodeid;
- req_lib_votequorum_general.header.size = sizeof (struct req_lib_votequorum_general);
- req_lib_votequorum_general.header.id = MESSAGE_REQ_VOTEQUORUM_QDEVICE_GETINFO;
-
- iov.iov_base = (char *)&req_lib_votequorum_general;
- iov.iov_len = sizeof (struct req_lib_votequorum_general);
+ iov.iov_base = (char *)&req_lib_votequorum_qdevice_getinfo;
+ iov.iov_len = sizeof (struct req_lib_votequorum_qdevice_getinfo);
error = qb_to_cs_error(qb_ipcc_sendv_recv (
votequorum_inst->c,
diff --git a/test/testvotequorum2.c b/test/testvotequorum2.c
index 39a7cbf..560f8ca 100644
--- a/test/testvotequorum2.c
+++ b/test/testvotequorum2.c
@@ -50,7 +50,7 @@ static void print_info(int ok_to_fail)
struct votequorum_qdevice_info qinfo;
int err;
- if ( (err=votequorum_qdevice_getinfo(handle, &qinfo)) != CS_OK)
+ if ( (err=votequorum_qdevice_getinfo(handle, VOTEQUORUM_NODEID_QDEVICE, &qinfo)) != CS_OK)
fprintf(stderr, "votequorum_qdevice_getinfo error %d: %s\n", err, ok_to_fail?"OK":"FAILED");
else {
printf("qdevice votes %d\n", qinfo.votes);
@@ -63,6 +63,7 @@ static void print_info(int ok_to_fail)
int main(int argc, char *argv[])
{
+ int ret = 0;
#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
int pollcount=0, polltime=1;
int err;
@@ -82,24 +83,34 @@ int main(int argc, char *argv[])
}
if (argc >= 2) {
- if ( (err=votequorum_qdevice_register(handle, "QDEVICE", 4)) != CS_OK)
+ if ( (err=votequorum_qdevice_register(handle, "QDEVICE")) != CS_OK) {
fprintf(stderr, "qdevice_register FAILED: %d\n", err);
+ ret = -1;
+ goto out;
+ }
while (pollcount--) {
print_info(0);
- if ((err=votequorum_qdevice_poll(handle, 1)) != CS_OK)
+ if ((err=votequorum_qdevice_poll(handle, "QDEVICE", 1)) != CS_OK) {
fprintf(stderr, "qdevice poll FAILED: %d\n", err);
+ ret = -1;
+ goto out;
+ }
print_info(0);
sleep(polltime);
}
- if ((err= votequorum_qdevice_unregister(handle)) != CS_OK)
+ if ((err= votequorum_qdevice_unregister(handle, "QDEVICE")) != CS_OK) {
fprintf(stderr, "qdevice unregister FAILED: %d\n", err);
+ ret = -1;
+ goto out;
+ }
}
print_info(1);
+out:
votequorum_finalize(handle);
#else
- fprintf(stderr, "qdevice support is not built in corosync/votequorum\n");
+ fprintf(stderr, "qdevice support is not built in corosync/votequorum\n");
#endif
- return 0;
+ return ret;
}
diff --git a/tools/corosync-quorumtool.c b/tools/corosync-quorumtool.c
index 0a21a3c..f60d7fd 100644
--- a/tools/corosync-quorumtool.c
+++ b/tools/corosync-quorumtool.c
@@ -63,7 +63,8 @@ typedef enum {
CMD_SHOWSTATUS,
CMD_SETVOTES,
CMD_SETEXPECTED,
- CMD_MONITOR
+ CMD_MONITOR,
+ CMD_UNREGISTER_QDEVICE
} command_t;
/*
@@ -108,6 +109,8 @@ static votequorum_callbacks_t v_callbacks = {
.votequorum_notify_fn = NULL,
.votequorum_expectedvotes_notify_fn = NULL
};
+static uint32_t our_nodeid = 0;
+static uint8_t display_qdevice = 0;
/*
* cfg bits
@@ -132,6 +135,7 @@ static void show_usage(const char *name)
printf(" -e <expected> change expected votes for the cluster *\n");
printf(" -H show nodeids in hexadecimal rather than decimal\n");
printf(" -i show node IP addresses instead of the resolved name\n");
+ printf(" -f forcefully unregister a quorum device *DANGEROUS*\n");
printf(" -h show this help text\n");
printf("\n");
printf(" * Starred items only work if votequorum is the quorum provider for corosync\n");
@@ -283,7 +287,7 @@ static void quorum_notification_fn(
}
}
-static void display_nodes_data(nodeid_format_t nodeid_format, name_format_t name_format)
+static void display_nodes_data(uint32_t nodeid, nodeid_format_t nodeid_format, name_format_t name_format)
{
int i;
@@ -305,13 +309,51 @@ static void display_nodes_data(nodeid_format_t nodeid_format, name_format_t name
printf("%s\n", node_name(g_view_list[i], name_format));
}
}
+
if (g_view_list_entries) {
free(g_view_list);
g_view_list = NULL;
}
+
+#if EXPERIMENTAL_QUORUM_DEVICE_API
+ if ((display_qdevice) && (v_handle)) {
+ int err;
+ struct votequorum_qdevice_info qinfo;
+
+ err = votequorum_qdevice_getinfo(v_handle, nodeid, &qinfo);
+ if (err != CS_OK) {
+ fprintf(stderr, "votequorum_qdevice_getinfo error: %d\n", err);
+ } else {
+ if (nodeid_format == NODEID_FORMAT_DECIMAL) {
+ printf("%10u ", VOTEQUORUM_NODEID_QDEVICE);
+ } else {
+ printf("0x%08x ", VOTEQUORUM_NODEID_QDEVICE);
+ }
+ printf("%3d %s (%s)\n", qinfo.votes, qinfo.name, qinfo.state?"Voting":"Not voting");
+ }
+ }
+#endif
+}
+
+static const char *decode_state(int state)
+{
+ switch(state) {
+ case NODESTATE_MEMBER:
+ return "Member";
+ break;
+ case NODESTATE_DEAD:
+ return "Dead";
+ break;
+ case NODESTATE_LEAVING:
+ return "Leaving";
+ break;
+ default:
+ return "Unknown state (this is bad)";
+ break;
+ }
}
-static int display_quorum_data(int is_quorate, int loop)
+static int display_quorum_data(int is_quorate, uint32_t nodeid, int loop)
{
struct votequorum_info info;
int err;
@@ -335,8 +377,9 @@ static int display_quorum_data(int is_quorate, int loop)
return CS_OK;
}
- if ((err=votequorum_getinfo(v_handle, 0, &info)) == CS_OK) {
+ if ((err=votequorum_getinfo(v_handle, nodeid, &info)) == CS_OK) {
printf("Node votes: %d\n", info.node_votes);
+ printf("Node state: %s\n", decode_state(info.state));
printf("Expected votes: %d\n", info.node_expected_votes);
printf("Highest expected: %d\n", info.highest_expected);
printf("Total votes: %d\n", info.total_votes);
@@ -348,6 +391,12 @@ static int display_quorum_data(int is_quorate, int loop)
if (info.flags & VOTEQUORUM_INFO_LAST_MAN_STANDING) printf("LastManStanding ");
if (info.flags & VOTEQUORUM_INFO_AUTO_TIE_BREAKER) printf("AutoTieBreaker ");
if (info.flags & VOTEQUORUM_INFO_LEAVE_REMOVE) printf("LeaveRemove ");
+ if (info.flags & VOTEQUORUM_INFO_QDEVICE) {
+ printf("Qdevice ");
+ display_qdevice = 1;
+ } else {
+ display_qdevice = 0;
+ }
printf("\n");
} else {
fprintf(stderr, "votequorum_getinfo FAILED: %d\n", err);
@@ -361,7 +410,7 @@ static int display_quorum_data(int is_quorate, int loop)
* 0 if not quorate
* -1 on error
*/
-static int show_status(nodeid_format_t nodeid_format, name_format_t name_format)
+static int show_status(uint32_t nodeid, nodeid_format_t nodeid_format, name_format_t name_format)
{
int is_quorate;
int err;
@@ -395,11 +444,11 @@ quorum_err:
return -1;
}
- err = display_quorum_data(is_quorate, 0);
+ err = display_quorum_data(is_quorate, nodeid, 0);
if (err != CS_OK) {
return -1;
}
- display_nodes_data(nodeid_format, name_format);
+ display_nodes_data(nodeid, nodeid_format, name_format);
return is_quorate;
}
@@ -410,7 +459,7 @@ static int monitor_status(nodeid_format_t nodeid_format, name_format_t name_form
if (q_type == QUORUM_FREE) {
printf("\nQuorum is not configured - cannot monitor\n");
- return show_status(nodeid_format, name_format);
+ return show_status(our_nodeid, nodeid_format, name_format);
}
err=quorum_trackstart(q_handle, CS_TRACK_CHANGES);
@@ -429,8 +478,8 @@ static int monitor_status(nodeid_format_t nodeid_format, name_format_t name_form
}
time(&t);
printf("date: %s", ctime((const time_t *)&t));
- err = display_quorum_data(g_quorate, loop);
- display_nodes_data(nodeid_format, name_format);
+ err = display_quorum_data(g_quorate, our_nodeid, loop);
+ display_nodes_data(our_nodeid, nodeid_format, name_format);
printf("\n");
loop = 1;
if (err != CS_OK) {
@@ -463,13 +512,37 @@ static int show_nodes(nodeid_format_t nodeid_format, name_format_t name_format)
}
}
- display_nodes_data(nodeid_format, name_format);
+ display_nodes_data(our_nodeid, nodeid_format, name_format);
result = EXIT_SUCCESS;
err_exit:
return result;
}
+static int unregister_qdevice(void)
+{
+#ifdef EXPERIMENTAL_QUORUM_DEVICE_API
+ int err;
+ struct votequorum_qdevice_info qinfo;
+
+ err = votequorum_qdevice_getinfo(v_handle, our_nodeid, &qinfo);
+ if (err != CS_OK) {
+ fprintf(stderr, "votequorum_qdevice_getinfo FAILED: %d\n", err);
+ return -1;
+ }
+
+ err = votequorum_qdevice_unregister(v_handle, qinfo.name);
+ if (err != CS_OK) {
+ fprintf(stderr, "votequorum_qdevice_unregister FAILED: %d\n", err);
+ return -1;
+ }
+ return 0;
+#else
+ fprintf(stderr, "votequorum quorum device support is not built-in\n");
+ return -1;
+#endif
+}
+
/*
* return -1 on error
* 0 if OK
@@ -509,6 +582,11 @@ static int init_all(void) {
goto out;
}
+ if (cmap_get_uint32(cmap_handle, "runtime.votequorum.this_node_id", &our_nodeid) != CS_OK) {
+ fprintf(stderr, "Unable to retrive this node nodeid\n");
+ goto out;
+ }
+
return 0;
out:
return -1;
@@ -530,12 +608,13 @@ static void close_all(void) {
}
int main (int argc, char *argv[]) {
- const char *options = "VHslme:v:hin:d:";
+ const char *options = "VHslmfe:v:hin:d:";
char *endptr;
int opt;
int votes = 0;
int ret = 0;
- uint32_t nodeid = VOTEQUORUM_NODEID_US;
+ uint32_t nodeid = 0;
+ uint32_t nodeid_set = 0;
nodeid_format_t nodeid_format = NODEID_FORMAT_DECIMAL;
name_format_t address_format = ADDRESS_FORMAT_NAME;
command_t command_opt = CMD_UNKNOWN;
@@ -552,6 +631,14 @@ int main (int argc, char *argv[]) {
while ( (opt = getopt(argc, argv, options)) != -1 ) {
switch (opt) {
+ case 'f':
+ if (using_votequorum() > 0) {
+ command_opt = CMD_UNREGISTER_QDEVICE;
+ } else {
+ fprintf(stderr, "You cannot unregister quorum device, corosync is not using votequorum\n");
+ exit(2);
+ }
+ break;
case 's':
command_opt = CMD_SHOWSTATUS;
break;
@@ -582,15 +669,18 @@ int main (int argc, char *argv[]) {
break;
case 'n':
nodeid = strtol(optarg, &endptr, 0);
- if ((nodeid == 0 && endptr == optarg) || nodeid <= 0) {
+ if ((nodeid == 0 && endptr == optarg) || nodeid < 0) {
fprintf(stderr, "The nodeid was not valid, try a positive number\n");
+ exit(2);
}
+ nodeid_set = 1;
break;
case 'v':
if (using_votequorum() > 0) {
votes = strtol(optarg, &endptr, 0);
if ((votes == 0 && endptr == optarg) || votes < 0) {
fprintf(stderr, "New votes value was not valid, try a positive number or zero\n");
+ exit(2);
} else {
command_opt = CMD_SETVOTES;
}
@@ -600,10 +690,12 @@ int main (int argc, char *argv[]) {
exit(2);
}
break;
+ case ':':
case 'h':
case '?':
default:
- break;
+ command_opt = CMD_UNKNOWN;
+ break;
}
}
@@ -616,9 +708,15 @@ int main (int argc, char *argv[]) {
ret = show_nodes(nodeid_format, address_format);
break;
case CMD_SHOWSTATUS:
- ret = show_status(nodeid_format, address_format);
+ if (!nodeid_set) {
+ nodeid = our_nodeid;
+ }
+ ret = show_status(nodeid, nodeid_format, address_format);
break;
case CMD_SETVOTES:
+ if (!nodeid_set) {
+ nodeid = our_nodeid;
+ }
ret = set_votes(nodeid, votes);
break;
case CMD_SETEXPECTED:
@@ -627,6 +725,9 @@ int main (int argc, char *argv[]) {
case CMD_MONITOR:
ret = monitor_status(nodeid_format, address_format);
break;
+ case CMD_UNREGISTER_QDEVICE:
+ ret = unregister_qdevice();
+ break;
}
close_all();
--
1.7.7.6
_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss
