On Fri, 2004-12-10 at 15:36, Jonathan E Brassow wrote: > starting ccsd with the -v flag should print to the log the plugin it is > using to connect to the cluster infrastructure. Something on the order > of: > "Connected to cluster infrastructure via: CMAN/SM Plugin v1.1" > > This will at least tell you that ccsd is able to connect to the cluster > manager - and therefore, should know whether the cluster is quorate or > not. > > brassow > > On Dec 10, 2004, at 4:16 PM, Matthew B. Brookover wrote: > > > On Fri, 2004-12-10 at 00:17, David Teigland wrote: > >> This sounds similar to a problem I have if I run fence_tool without > >> ccsd > >> running. > >> > >> Check /proc/cluster/status while it's waiting to see if the cluster > >> actually has quorum or not. Also, I've added some extra checking and > >> debugging to fence_tool that should help narrow down where things are > >> stuck. Please update from cvs and rebuild at least the stuff in > >> cluster/fence; then use "fence_tool join -D". > >> > >> Usually things get stuck talking to ccs when ccs/magma libraries are > >> out > >> of sync, but this case sounds different. > > I tried -v, and the cluster is quorate. Only the fencing is not starting up correctly. Here are the logs with ccsd -v: Added -v flag to ccsd, producing this log entry on fouroften: Dec 10 15:50:00 fouroften kernel: Lock_Harness <CVS> (built Dec 10 2004 09:14:45) installed Dec 10 15:50:00 fouroften kernel: GFS <CVS> (built Dec 10 2004 09:14:04) installed Dec 10 15:50:00 fouroften kernel: CMAN <CVS> (built Dec 10 2004 09:51:59) installed Dec 10 15:50:00 fouroften kernel: NET: Registered protocol family 30 Dec 10 15:50:00 fouroften kernel: DLM <CVS> (built Dec 10 2004 09:52:25) installed Dec 10 15:50:00 fouroften kernel: Lock_DLM (built Dec 10 2004 09:14:25) installed Dec 10 15:50:00 fouroften ccsd[3379]: Starting ccsd DEVEL.1102700899: Dec 10 15:50:01 fouroften ccsd[3379]: Built: Dec 10 2004 10:50:57 Dec 10 15:50:01 fouroften ccsd[3379]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Dec 10 15:50:01 fouroften ccsd[3379]: Verbose Flag:: SET Dec 10 15:50:01 fouroften ccsd[3379]: cluster.conf (cluster name = CSMTEST, version = 6) found. Dec 10 15:50:01 fouroften kernel: CMAN: Waiting to join or form a Linux-cluster Dec 10 15:50:01 fouroften crond(pam_unix)[3388]: session opened for user root by (uid=0) Dec 10 15:50:01 fouroften crond(pam_unix)[3387]: session opened for user root by (uid=0) Dec 10 15:50:02 fouroften ccsd[3379]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1 Dec 10 15:50:02 fouroften ccsd[3379]: Initial status:: Inquorate Dec 10 15:50:02 fouroften crond(pam_unix)[3388]: session closed for user root Dec 10 15:50:03 fouroften crond(pam_unix)[3387]: session closed for user root Dec 10 15:50:33 fouroften kernel: CMAN: forming a new cluster Dec 10 15:50:33 fouroften kernel: CMAN: quorum regained, resuming activity Dec 10 15:50:33 fouroften kernel: CMAN: got node fiveoften Logs on fiveoften: Dec 10 15:50:07 fiveoften kernel: Lock_Harness <CVS> (built Dec 10 2004 09:14:45) installed Dec 10 15:50:07 fiveoften kernel: GFS <CVS> (built Dec 10 2004 09:14:04) installed Dec 10 15:50:07 fiveoften crond(pam_unix)[3354]: session closed for user root Dec 10 15:50:07 fiveoften kernel: CMAN <CVS> (built Dec 10 2004 09:51:59) installed Dec 10 15:50:07 fiveoften kernel: NET: Registered protocol family 30 Dec 10 15:50:07 fiveoften kernel: DLM <CVS> (built Dec 10 2004 09:52:25) installed Dec 10 15:50:07 fiveoften kernel: Lock_DLM (built Dec 10 2004 09:14:25) installed Dec 10 15:50:08 fiveoften ccsd[3381]: Starting ccsd DEVEL.1102700899: Dec 10 15:50:08 fiveoften ccsd[3381]: Built: Dec 10 2004 10:50:57 Dec 10 15:50:08 fiveoften ccsd[3381]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Dec 10 15:50:08 fiveoften ccsd[3381]: Verbose Flag:: SET Dec 10 15:50:08 fiveoften ccsd[3381]: cluster.conf (cluster name = CSMTEST, version = 6) found. Dec 10 15:50:08 fiveoften kernel: CMAN: Waiting to join or form a Linux-cluster Dec 10 15:50:35 fiveoften kernel: CMAN: sending membership request Dec 10 15:50:35 fiveoften kernel: CMAN: got node fouroften Dec 10 15:50:35 fiveoften kernel: CMAN: quorum regained, resuming activity Dec 10 15:50:35 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:35 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:36 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:36 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:37 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:37 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:38 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:38 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:39 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:39 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:40 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:40 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:41 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:41 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:42 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:42 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:43 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:43 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:50:44 fiveoften ccsd[3381]: Cluster is not quorate. Refusing connection. Dec 10 15:50:44 fiveoften ccsd[3381]: Error while processing connect: Connection refused Dec 10 15:53:26 fiveoften ccsd[3381]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1 Dec 10 15:53:26 fiveoften ccsd[3381]: Initial status:: Quorate Matt mbrookov@xxxxxxxxx