First up, before I begin, I am looking to pacemaker for the future as
well and do not yet use it. So please take whatever I say about
pacemaker with a grain of sand. Andrew, on the other hand, is the author
and anything he says can be taken as authoritative on the topic.
On the future;
I also have a 2-node project/product that I am working to update in time
for the release of RHEL 7. Speaking entirely for myself, I can tell you
that I am planning to use Pacemaker from RHEL 7.0. As a Red hat
outsider, I can only speak as a member of the community, but I have
every reason to believe that the pacemaker resource manager will be the
one used from 7.0 and forward.
As for the CIB, yes, it's a local XML file stored on each node.
Synchronization occurs via updates pushed over corosync to nodes active
in the cluster. As I understand it, when a node that had been offline
connects to the cluster, it receives any updates to the CIB.
Dealing with 2-node clusters, setting aside qdisk which has an uncertain
future I believe, you can not use quorum. For this reason, it is
possible for a node to boot up, fail to reach it's peer and think it's
the only one running. It will start your HA services and voila, two
nodes offering the same services at the same time in an uncoordinated
manner. This is bad and it is called a "split-brain".
The way to avoid split-brains in 2-node clusters is to use fence
devices, aka stonith devices (exact same thing by two different names).
This is _always_ wise to use, but in 2-node clusters, it is critical.
So imagine back to your scenario;
If a node came up and tried to connect to it's peer but failed to do so,
before proceeding, it would fence (usually forcibly power off) the other
node. Only after doing so would it start the HA services. In this way,
both nodes can never be offering the same HA service at the same time.
The risk here though is a "fence loop". If you set the cluster to start
on boot and if there is a break in the connection, you can have an
initial state where, upon the break in the network, both try to fence
the other. The faster node wins, forcing the other node off and resuming
to operate on it's own. This is fine and exactly what you want. However,
now the fenced node powers back up, starts it's cluster stack, fails to
reach it's peer and fences it. It finishes starting, offers the HA
services and goes on it's way ... until the other node boots back up. :)
Personally, I avoid this by _not_ starting the cluster stack on boot. My
reasoning is that, if a node fails and gets rebooted, I want to check it
over myself before I let it back into the cluster (I get alert emails
when something like this happens). It's not a risk from an HA
perspective because it's services would have recovered on the surviving
peer long before it reboots anyway. This also has the added benefit of
avoiding a fence loop, no matter what happens.
Cheers
digimer
On 04/23/2013 02:07 PM, Michael Richmond wrote:
Andrew and Digimer,
Thank you for taking the time to respond, you have collaborated some of
what I've been putting together as the likely direction.
I am working on adapting some cluster-aware storage features for use in a
Linux cluster environment. With this kind of project it is useful to try
and predict where the Linux community is heading so that I can focus my
development work on what will be the "current" cluster stack around my
anticipated release dates. Any predictions are simply educated guesses
that may prove to be wrong, but are useful with regard to developing
plans. From my reading of various web pages and piecing things together I
found that RHEL 7 is intended to be based on Fedora 18, so I assume that
the new Pacemaker stack has a good chance of being rolled out in RHEL
7.1/7.2, or even possibly 7.0.
Hearing that there is official word that the intention is for Pacemaker to
be the official cluster stack helps me put my development plans together.
The project I am working on is focused on two-node clusters. But I also
need a persistent, cluster-wide data store to hold a small amount of state
(less than 1KB). This data store is what I refer to as a cluster-registry.
The state data records the last-known operational state for the storage
feature. This last-known state helps drive recovery operations for the
storage feature during node bring-up. This project is specifically aimed
at integrating generic functionality into the Linux cluster stack.
I have been thinking about using the cluster configuration file for this
storage which I assume is the CIB referenced by Andrew. But I can imagine
cases where the CIB file may loose updates if it does not utilize shared
storage media. My understanding is that the CIB file is stored on each
node using local disk storage.
For example, consider a two-node cluster that is configured with a quorum
disk on shared storage media. If at a given point in time NodeB is up and
NodeB is down. NodeA can form quorate and start cluster services
(including HA applications). Assume that NodeA updates the CIB to record
some state update. If NodeB starts booting but before NodeB joins the
cluster, NodeA crashes. At this point, the updated CIB only resides on
NodeA and cannot be accessed by NodeB even if NodeB can access the quorum
disk as form quorate. Effectively, NodeB cannot be aware of the update
from NodeA which will result in an implicit roll-back of any updates
performed by NodeA.
With a two-node cluster, there are two options for resolving this:
* prevent any update to the cluster registry/CIB unless all nodes are part
of the cluster. (This is not practical since it undermines some of the
reasons for building clusters.)
* store the cluster registry on shared storage so that there is one source
of truth.
It is possible that the nature of the data stored in the CIB is resilient
to the example scenario that I describe. In this case, maybe the CIB is
not an appropriate data store for my cluster registry data. In this case I
am either looking for an appropriate Linux component to use for my cluster
registry, or I will build a custom data store that provides atomic update
semantics on shared storage.
Any thoughts and/or pointers would be appreciated.
Thanks,
Michael Richmond
--
michael richmond | principal software engineer | flashsoft, sandisk |
+1.408.425.6731
On 22/4/13 4:37 PM, "Andrew Beekhof" <andrew@xxxxxxxxxxx> wrote:
On 23/04/2013, at 4:59 AM, Digimer <lists@xxxxxxxxxx> wrote:
On 04/22/2013 02:36 PM, Michael Richmond wrote:
Hello,
I am researching the new cluster stack that is scheduled to be
delivered
in Fedora 19. Does anyone on this list have a sense for the timeframe
for this new stack to be rolled into a RHEL release? (I assume the
earliest would be RHEL 7.)
On the Windows platform, Microsoft Cluster Services provides a
cluster-wide registry service that is basically a cluster-wide
key:value
store with atomic updates and support to store the registry on shared
disk. The storage on shared disk allows access and use of the registry
in cases where nodes are frequently joining and leaving the cluster.
Are there any component(s) that can be used to provide a similar
registry in the Linux cluster stack? (The current RHEL 6 stack, and/or
the new Fedora 19 stack.)
Thanks in advance for your information,
Michael Richmond
Hi Michael,
First up, Red Hat's policy of what is coming is "we'll announce on
release day". So anything else is a guess. As it is, Pacemaker is in
tech-preview in RHEL 6, and the best guess is that it will be the
official resource manager in RHEL 7, but it's just that, a guess.
I believe we're officially allowed to say that it is our _intention_ that
Pacemaker will be the one and only supported stack in RHEL7.
As for the registry question; I am not entirely sure what it is you
are asking here (sorry, not familiar with windows). I can say that
pacemaker uses something called the CIB (cluster information base) which
is an XML file containing the cluster's configuration and state. It can
be updated from any node and the changes will push to the other nodes
immediately.
How many of these attributes are you planning to have?
You can throw a few in there, but I'd not use it for 100's or 1000's of
them - its mainly designed to store the resource/service configuration.
Does this answer your question?
The current RHEL 6 cluster is corosync + cman + rgmanager. It also
uses an XML config and it can be updated from any node and push out to
the other nodes.
Perhaps a better way to help would be to ask what, exactly, you want
to build your cluster for?
Cheers
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
________________________________
PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
