3rd node won't rejoin cluster

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I’ve just run into a odd problem on my production cluster.  One of the nodes got fenced (still digging through logs to find out why) and on it’s way back up, it appears to join the cluster find but the node that fenced it starts spewing out tons of these in /var/log/messages:

 

Sep 10 14:25:34 redwing gfs_controld[6119]: cpg_mcast_joined retry 176200 unknown

Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined retry 176300 unknown

Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined retry 176400 unknown

Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined retry 176500 unknown

Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined retry 176600 unknown

Sep 10 14:25:35 redwing gfs_controld[6119]: cpg_mcast_joined retry 176700 unknown

….

 

The node that got fenced just hangs at the “Starting Fencing…” part of cman, while redwing (the node that fenced it) starts to climb in load slowly but surely.  I ended up bringing down the fenced node and I’m running fine off the 2 remaining nodes.  Has anyone ran into this problem.

 

I’m running RHEL5.3 with these packages:

 

[a_arwin@redwing ~]$ rpm -qa | egrep 'cman|rgman|gfs|lvm'

lvm2-2.02.40-6.el5

kmod-gfs-0.1.31-3.el5

cman-2.0.98-1.el5_3.1

gfs-utils-0.1.18-1.el5

rgmanager-2.0.46-1.el5_3.3

gfs2-utils-0.1.53-1.el5_3.2

lvm2-cluster-2.02.40-7.el5

[a_arwin@redwing ~]$ uname -a

Linux redwing.csun.edu 2.6.18-128.1.6.el5 #1 SMP Tue Mar 24 12:05:57 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

 

Thanks ahead of time,

Arwin

--
Linux-cluster mailing list
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster

[Index of Archives]     [Corosync Cluster Engine]     [GFS]     [Linux Virtualization]     [Centos Virtualization]     [Centos]     [Linux RAID]     [Fedora Users]     [Fedora SELinux]     [Big List of Linux Books]     [Yosemite Camping]

  Powered by Linux