Hi,
When creating the VG by default clustered, you implicitely assume that it will be used with a clustered FS on top of it (gfs, ocfs, etc...) that will handle the active/active mode.
As I do not intend to use GFS in this particular case, but ext3 and raw devices, I need to make sure the vg is exclusively activated on one node, preventing the other nodes to access it unless it is the failover procedure (node holding the VG crashed) and then re activate it exclusively on the failover node.
Thanks
Brem
2009/7/21, Christine Caulfield <ccaulfie@xxxxxxxxxx>:
Hiya,
If you make a VG non-clustered then you, by definition, forfeit all cluster locking protection on all of the LVs in that group.
In practise you should always make shared volumes clustered and non-shared volumes non-clustered. If you make a shared volume non-clustered then it's up to you to manage the protection of it.
It might be possible to put some protection in for when a volume group is changed from clustered to non-clustered, but really, you're not supposed to do that!
Look at it this way, if you create a shared VG and mark it non-clustered to start with, then you can corrupt it as much as you like by mounting its filesystems on multiple nodes. It's just the same as a SAN volume then.
On 07/21/2009 12:16 PM, brem belguebli wrote:
Hi Chrissie,
Indeed, by default when creating the VG, it is clustered, thus when
creating the LV it is active on all nodes.
To avoid data corruption, I have re created the VG as non clustered
(vgcreate -c n vgXX) then created the LV which got activated only on the
node where it got created.
Then changed the VG to clustered (vgchange -c y VGXX) and activated it
exclusively on this node.
But, I could reproduce the behaviour of bypassing the exclusive flag:
On node B, re change the VG to non clustered though it is activated
exclusively on node A.
and then activate it on node B and it works.
The thing I'm trying to point is that simply by erasing the clustered
flag you can bypass the exclusive activation.
I think a barrier is necessary to prevent this to happen, removing the
clustered flag from a VG should be possible only if the node holding the
VG exclusively is down (does the lock manager DLM report which node
holds exclusively a VG ?)
Thanks
Brem
2009/7/21, Christine Caulfield <ccaulfie@xxxxxxxxxx
<mailto:ccaulfie@xxxxxxxxxx>>:Linux-cluster@xxxxxxxxxx <mailto:Linux-cluster@xxxxxxxxxx>
Hiya,
I've just tried this on my cluster and it works fine.
What you need to remember is that lvcreate on one node will also
activate the LV on all nodes in the cluster - it does an implicit
lvchange -ay when you create it. What I can't explain is why
vgchange -ae seemed to work fine on node A, it should give the same
error as on node B because LVs are open shared on both nodes.
Its not clear to me when you tagged the VG as clustered, so that
might be contributing to the problem. When I create a new VG on
shared storage it automatically gets labelled clustered so I have
never needed to do this explicitly. If you create a non-clustered VG
you probably ought to deactivate it on all nodes first as it could
mess up the locking otherwise. This *might* be the cause of your
troubles.
The error on clvmd startup can be ignored. It's caused by clvmd
ussing a background command with --no_locking so that it can check
which LVs (if any) are already active and re-acquire locks for them
Sorry this isn't conclusive, The exact order in which things are
happening is not clear to me.
Chrissie.
On 07/21/2009 10:21 AM, brem belguebli wrote:
Hi all,
I think there is something to clarify about using CLVM across a
cluster
in a active/passive mode without GFS.
From my understanding, CLVM keeps LVM metadata coherent among the
cluster nodes and provides a cluster wide locking mechanism that can
prevent any node from trying to activate a volume group if it
has been
activated exclusively (vgchange -a e VGXXX) by another node (which
needs to be up).
I have been playing with it to check this behaviour but it
doesn't seem
to make what is expected.
I have 2 nodes (RHEL 5.3 X86_64, cluster installed and
configured) , A
and B using a SAN shared storage.
I have a LUN from this SAN seen by both nodes, pvcreate'd
/dev/mpath/mpath0 , vgcreate'd vg10 and lvcreate'd lvol1 (on one
node),
created an ext3 FS on /dev/vg10/lvol1
CLVM is running in debug mode (clvmd -d2 ) (but it complains about
locking disabled though locking set to 3 on both nodes)
On node A:
vgchange -c y vg10 returns OK (vgs --> vg10 1
1 0
wz--nc)
vgchange -a e --> OK
lvs returns lvol1 vg10 -wi-a-
On node B (while things are active on A, A is UP and member of the
cluster ):
vgchange -a e --> Error locking on node B: Volume is
busy on
another node
1 logical volume(s) in
volume group
"vg10" now active
It activates vg10 even if it sees it busy on another node .
on B, lvs returns lvol1 vg10 -wi-a-
as well as on A.
I think the main problem comes from the fact that, as it is said
when
starting CLVM in debug mode, WARNING: Locking disabled. Be careful!
This could corrupt your metadata.
IMHO, the algorithm should be as follows:
VG is tagged as clustered (vgchange -c y VGXXX)
if a node (node P) tries to activate the VG exclusively
(vgchange -a VGXXX)
ask the lock manager to check if VG is not already locked by another
node (node X)
if so, check if node X is up
if node X is down, return OK to node P
else
return NOK to node P (explicitely that VG is held exclusively by
node X)
Brem
PS: this shouldn't be a problem with GFS or other clustered FS
(OCFS,
etc...) as no node should try to activate exclusively any VG.
------------------------------------------------------------------------
--
Linux-cluster mailing list
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx <mailto:Linux-cluster@xxxxxxxxxx>
https://www.redhat.com/mailman/listinfo/linux-cluster
------------------------------------------------------------------------
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
