sorry for the long e-mail in advance.
trying to do on a test environment what in subject and I think it could be useful for others too, both in RH EL and in CentOS.
I have configured two ip+fs services and HA-LVM
Starting point is CentOS 5.3 updated at these components:
cman-2.0.98-1.el5_3.1
openais-0.80.3-22.el5_3.4
rgmanager-2.0.46-1.el5.centos.3
luci-0.12.1-7.3.el5.centos.1
ricci-0.12.1-7.3.el5.centos.1
lvm2-2.02.40-6.el5
device-mapper-multipath-0.4.7-23.el5_3.4
Target would be:
cman-2.0.115-1.el5_4.3
openais-0.80.6-8.el5_4.1
rgmanager-2.0.52-1.el5.centos.2
luci-0.12.2-6.el5.centos
ricci-0.12.2-6.el5.centos
lvm2-2.02.46-8.el5_4.1
device-mapper-multipath-0.4.7-30.el5_4.2
they are guests in Qemu-KVM environment and I have a backup of the starting situation, so that I can reply and change eventually order of operations.
node1 is mork, node2 is mindy
Attempt of approach:
- services are on node2 (mindy)
- shutdown ad restart node1 in single user mode
- activate network and update node1 with:
yum clean all
yum update glibc\*
yum update yum\* rpm\* python\*
yum clean all
yum update
shutdown -r now and start in single user mode to check correct start and so on
- init 3 for node1 and join to cluster
QUESTION1: are there any incompatibilities in this first join of the cluster, based on the different components' versions?
Would it be better in your opinion to make a shutdown of node2 and then have node1 start alone and take the services and then upgrade node2 and have the first contemporary two-nodes join with aligned versions of clusterware software?
Now, following my approach, after the init 3 on node1 all was ok with cluster join, but I forgot to do a touch of the initrd file of the updated kernel,
due to de-optimized check in HA-LVM service comparing timestamp of initrd of running kernel and lvm.conf
So clurgmgrd complains having
-rw-r--r-- 1 root root 16433 Nov 2 12:28 /etc/lvm/lvm.conf
newer than initrd that is dated end of September..... (see below)
Nov 2 12:41:00 mork kernel: DLM (built Sep 30 2009 12:53:28) installed
Nov 2 12:41:00 mork kernel: GFS2 (built Sep 30 2009 12:54:10) installed
Nov 2 12:41:00 mork kernel: Lock_DLM (built Sep 30 2009 12:54:16) installed
Nov 2 12:41:00 mork ccsd[2290]: Starting ccsd 2.0.115:
Nov 2 12:41:00 mork ccsd[2290]: Built: Oct 26 2009 22:01:34
Nov 2 12:41:00 mork ccsd[2290]: Copyright (C) Red Hat, Inc. 2004 All rights reserved.
Nov 2 12:41:00 mork ccsd[2290]: cluster.conf (cluster name = clumm, version = 5) found.
Nov 2 12:41:00 mork ccsd[2290]: Remote copy of cluster.conf is from quorate node.
Nov 2 12:41:00 mork ccsd[2290]: Local version # : 5
Nov 2 12:41:00 mork ccsd[2290]: Remote version #: 5
Nov 2 12:41:00 mork ccsd[2290]: Remote copy of cluster.conf is from quorate node.
Nov 2 12:41:00 mork ccsd[2290]: Local version # : 5
Nov 2 12:41:00 mork ccsd[2290]: Remote version #: 5
Nov 2 12:41:00 mork ccsd[2290]: Remote copy of cluster.conf is from quorate node.
Nov 2 12:41:00 mork ccsd[2290]: Local version # : 5
Nov 2 12:41:00 mork ccsd[2290]: Remote version #: 5
Nov 2 12:41:00 mork ccsd[2290]: Remote copy of cluster.conf is from quorate node.
Nov 2 12:41:00 mork ccsd[2290]: Local version # : 5
Nov 2 12:41:00 mork ccsd[2290]: Remote version #: 5
Nov 2 12:41:00 mork openais[2302]: [MAIN ] AIS Executive Service RELEASE 'subrev 1887 version 0.80.6'
Nov 2 12:41:00 mork openais[2302]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors.
Nov 2 12:41:00 mork openais[2302]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Nov 2 12:41:00 mork openais[2302]: [MAIN ] AIS Executive Service: started and ready to provide service.
Nov 2 12:41:00 mork openais[2302]: [MAIN ] Using default multicast address of 239.192.12.183
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Token Timeout (162000 ms) retransmit timeout (8019 ms)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] token hold (6405 ms) retransmits before loss (20 retrans)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500
s)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] send threads (0 threads)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] RRP token expired timeout (8019 ms)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] RRP token problem counter (2000 ms)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] RRP threshold (10 problem count)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] RRP mode set to none.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] heartbeat_failures_allowed (0)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] max_network_delay (50 ms)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes).
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Nov 2 12:41:00 mork openais[2302]: [TOTEM] The network interface [172.16.0.11] is now up.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Created or loaded sequence id 336.172.16.0.11 for this ring.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering GATHER state from 15.
Nov 2 12:41:00 mork openais[2302]: [CMAN ] CMAN 2.0.115 (built Oct 26 2009 22:01:42) started
Nov 2 12:41:00 mork openais[2302]: [MAIN ] Service initialized 'openais CMAN membership service 2.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais extended virtual synchrony service'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais cluster membership service B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais availability management framework B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais checkpoint service B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais event service B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais distributed locking service B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais message service B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais configuration service'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais cluster closed process group service v1.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais cluster config database access v1.01'
Nov 2 12:41:00 mork openais[2302]: [SYNC ] Not using a virtual synchrony filter.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Creating commit token because I am the rep.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Saving state aru 0 high seq received 0
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Storing new sequence id for ring 154
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering COMMIT state.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering RECOVERY state.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] position [0] member 172.16.0.11:
Nov 2 12:41:00 mork openais[2302]: [TOTEM] previous ring seq 336 rep 172.16.0.11
Nov 2 12:41:00 mork openais[2302]: [TOTEM] aru 0 high delivered 0 received flag 1
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Did not need to originate any messages in recovery.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Sending initial ORF token
Nov 2 12:41:00 mork openais[2302]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:41:00 mork openais[2302]: [CLM ] New Configuration:
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Left:
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Joined:
Nov 2 12:41:00 mork openais[2302]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:41:00 mork openais[2302]: [CLM ] New Configuration:
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Left:
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Joined:
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:41:00 mork openais[2302]: [SYNC ] This node is within the primary component and will provide service.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering OPERATIONAL state.
Nov 2 12:41:00 mork openais[2302]: [CLM ] got nodejoin message 172.16.0.11
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering GATHER state from 11.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Creating commit token because I am the rep.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Saving state aru a high seq received a
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Storing new sequence id for ring 158
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering COMMIT state.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering RECOVERY state.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] position [0] member 172.16.0.11:
Nov 2 12:41:00 mork openais[2302]: [TOTEM] previous ring seq 340 rep 172.16.0.11
Nov 2 12:41:00 mork openais[2302]: [TOTEM] aru a high delivered a received flag 1
Nov 2 12:41:00 mork openais[2302]: [TOTEM] position [1] member 172.16.0.12:
Nov 2 12:41:00 mork openais[2302]: [TOTEM] previous ring seq 340 rep 172.16.0.12
Nov 2 12:41:00 mork openais[2302]: [TOTEM] aru d high delivered d received flag 1
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Did not need to originate any messages in recovery.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Sending initial ORF token
Nov 2 12:41:00 mork openais[2302]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:41:00 mork openais[2302]: [CLM ] New Configuration:
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Left:
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Joined:
Nov 2 12:41:00 mork openais[2302]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:41:00 mork openais[2302]: [CLM ] New Configuration:
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Left:
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Joined:
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:41:00 mork openais[2302]: [SYNC ] This node is within the primary component and will provide service.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering OPERATIONAL state.
Nov 2 12:41:00 mork openais[2302]: [CMAN ] quorum regained, resuming activity
Nov 2 12:41:00 mork openais[2302]: [CLM ] got nodejoin message 172.16.0.11
Nov 2 12:41:00 mork openais[2302]: [CLM ] got nodejoin message 172.16.0.12
Nov 2 12:41:00 mork openais[2302]: [CPG ] got joinlist message from node 2
Nov 2 12:41:01 mork ccsd[2290]: Initial status:: Quorate
uorum
Nov 2 12:41:01 mork qdiskd[2331]: <info> Quorum Daemon Initializing
Nov 2 12:41:02 mork qdiskd[2331]: <info> Heuristic: 'ping -c1 -w1 192.168.122.1' UP
Nov 2 12:41:12 mork modclusterd: startup succeeded
Nov 2 12:41:12 mork kernel: dlm: Using TCP for communications
Nov 2 12:41:12 mork kernel: dlm: connecting to 2
Nov 2 12:41:12 mork kernel: dlm: got connection from 2
Nov 2 12:41:12 mork clurgmgrd[2886]: <notice> Resource Group Manager Starting
Nov 2 12:41:13 mork oddjobd: oddjobd startup succeeded
Nov 2 12:41:13 mork saslauthd[3338]: detach_tty : master pid is: 3338
Nov 2 12:41:13 mork saslauthd[3338]: ipc_init : listening on socket: /var/run/saslauthd/mux
Nov 2 12:41:14 mork ricci: startup succeeded
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> HA LVM: Improper setup detected
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> HA LVM: Improper setup detected
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> - initrd image needs to be newer than lvm.conf
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> - initrd image needs to be newer than lvm.conf
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> WARNING: An improper setup can cause data corruption!
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> WARNING: An improper setup can cause data corruption!
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> node2 owns vg_cl1/lv_cl1 unable to stop
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> node2 owns vg_cl2/lv_cl2 unable to stop
Nov 2 12:41:14 mork clurgmgrd[2886]: <notice> stop on lvm "CL2" returned 1 (generic error)
Nov 2 12:41:14 mork clurgmgrd[2886]: <notice> stop on lvm "CL1" returned 1 (generic error)
Nov 2 12:41:31 mork qdiskd[2331]: <info> Node 2 is the master
Nov 2 12:42:21 mork qdiskd[2331]: <info> Initial score 1/1
Nov 2 12:42:21 mork qdiskd[2331]: <info> Initialization complete
Nov 2 12:42:21 mork openais[2302]: [CMAN ] quorum device registered
Nov 2 12:42:21 mork qdiskd[2331]: <notice> Score sufficient for master operation (1/1; required=1); upgrading
Note that a clustat of both nodes gives correct results (in the sens of nodes taking part in the cluster and rgmanager active on both and quorum disk).
At this point, after touching initrd file, I think to do a shutdown -r of mork again and see if all goes well.
It seems so, as I get again:
...
Nov 2 12:46:23 mork openais[2278]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:46:23 mork openais[2278]: [CLM ] New Configuration:
Nov 2 12:46:23 mork openais[2278]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:46:23 mork openais[2278]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:46:23 mork openais[2278]: [CLM ] Members Left:
Nov 2 12:46:23 mork openais[2278]: [CLM ] Members Joined:
Nov 2 12:46:23 mork openais[2278]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:46:23 mork openais[2278]: [SYNC ] This node is within the primary component and will provide service.
Nov 2 12:46:23 mork openais[2278]: [TOTEM] entering OPERATIONAL state.
Nov 2 12:46:23 mork openais[2278]: [CMAN ] quorum regained, resuming activity
Nov 2 12:46:23 mork openais[2278]: [CLM ] got nodejoin message 172.16.0.11
Nov 2 12:46:23 mork openais[2278]: [CLM ] got nodejoin message 172.16.0.12
Nov 2 12:46:23 mork openais[2278]: [CPG ] got joinlist message from node 2
Nov 2 12:46:24 mork ccsd[2267]: Initial status:: Quorate
uorum
Nov 2 12:46:25 mork qdiskd[2310]: <info> Quorum Daemon Initializing
Nov 2 12:46:26 mork qdiskd[2310]: <info> Heuristic: 'ping -c1 -w1 192.168.122.1' UP
...
Nov 2 12:46:35 mork modclusterd: startup succeeded
Nov 2 12:46:35 mork kernel: dlm: Using TCP for communications
Nov 2 12:46:35 mork kernel: dlm: connecting to 2
Nov 2 12:46:36 mork oddjobd: oddjobd startup succeeded
Nov 2 12:46:36 mork saslauthd[2990]: detach_tty : master pid is: 2990
Nov 2 12:46:36 mork saslauthd[2990]: ipc_init : listening on socket: /var/run/saslauthd/mux
Nov 2 12:46:36 mork ricci: startup succeeded
Nov 2 12:46:55 mork qdiskd[2310]: <info> Node 2 is the master
Nov 2 12:47:45 mork qdiskd[2310]: <info> Initial score 1/1
Nov 2 12:47:45 mork qdiskd[2310]: <info> Initialization complete
Nov 2 12:47:45 mork openais[2278]: [CMAN ] quorum device registered
Nov 2 12:47:45 mork qdiskd[2310]: <notice> Score sufficient for master operation (1/1; required=1); upgrading
but instead, on mindy I get this error and the node goes out of memory and I have to power off it....
Nov 2 12:47:54 mindy kernel: dlm: connect from non cluster node
Donna if the problem with cluster is the cause or the effect of the problem....
In particular, these are messages on mindy, during the first join of the cluster and the reboot of mork:
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] entering GATHER state from 11.
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] Saving state aru d high seq received d
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] Storing new sequence id for ring 158
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] entering COMMIT state.
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] entering RECOVERY state.
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] position [0] member 172.16.0.11:
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] previous ring seq 340 rep 172.16.0.11
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] aru a high delivered a received flag 1
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] position [1] member 172.16.0.12:
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] previous ring seq 340 rep 172.16.0.12
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] aru d high delivered d received flag 1
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] Did not need to originate any messages in recovery.
Nov 2 12:42:20 mindy openais[2465]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:42:20 mindy openais[2465]: [CLM ] New Configuration:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:42:20 mindy openais[2465]: [CLM ] Members Left:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] Members Joined:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:42:20 mindy openais[2465]: [CLM ] New Configuration:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:42:20 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:42:20 mindy openais[2465]: [CLM ] Members Left:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] Members Joined:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:42:20 mindy openais[2465]: [SYNC ] This node is within the primary component and will provide service.
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] entering OPERATIONAL state.
Nov 2 12:42:20 mindy openais[2465]: [CLM ] got nodejoin message 172.16.0.11
Nov 2 12:42:20 mindy openais[2465]: [CLM ] got nodejoin message 172.16.0.12
Nov 2 12:42:20 mindy openais[2465]: [CPG ] got joinlist message from node 2
Nov 2 12:42:32 mindy kernel: dlm: connecting to 1
Nov 2 12:42:32 mindy kernel: dlm: got connection from 1
Nov 2 12:46:16 mindy clurgmgrd[3101]: <notice> Member 1 shutting down
Nov 2 12:46:26 mindy qdiskd[2508]: <info> Node 1 shutdown
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] entering GATHER state from 12.
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] Saving state aru 3e high seq received 3e
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] Storing new sequence id for ring 160
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] entering COMMIT state.
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] entering RECOVERY state.
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] position [0] member 172.16.0.11:
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] previous ring seq 348 rep 172.16.0.11
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] aru a high delivered a received flag 1
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] position [1] member 172.16.0.12:
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] previous ring seq 344 rep 172.16.0.11
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] aru 3e high delivered 3e received flag 1
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] Did not need to originate any messages in recovery.
Nov 2 12:47:43 mindy openais[2465]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:47:43 mindy openais[2465]: [CLM ] New Configuration:
Nov 2 12:47:43 mindy kernel: dlm: closing connection to node 1
Nov 2 12:47:43 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:47:43 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:47:43 mindy openais[2465]: [CLM ] Members Left:
Nov 2 12:47:43 mindy openais[2465]: [CLM ] Members Joined:
Nov 2 12:47:43 mindy openais[2465]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:47:43 mindy openais[2465]: [CLM ] New Configuration:
Nov 2 12:47:43 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:47:43 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:47:43 mindy openais[2465]: [CLM ] Members Left:
Nov 2 12:47:43 mindy openais[2465]: [CLM ] Members Joined:
Nov 2 12:47:43 mindy openais[2465]: [SYNC ] This node is within the primary component and will provide service.
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] entering OPERATIONAL state.
Nov 2 12:47:43 mindy openais[2465]: [CLM ] got nodejoin message 172.16.0.11
Nov 2 12:47:43 mindy openais[2465]: [CLM ] got nodejoin message 172.16.0.12
Nov 2 12:47:43 mindy openais[2465]: [CPG ] got joinlist message from node 2
Nov 2 12:47:54 mindy kernel: dlm: connect from non cluster node
Nov 2 12:59:48 mindy kernel: dlm_send invoked oom-killer: gfp_mask=0xd0, order=1, oomkilladj=0
Nov 2 12:59:48 mindy kernel:
Nov 2 12:59:48 mindy kernel: Call Trace:
Nov 2 12:59:48 mindy kernel: [<ffffffff800c3a6a>] out_of_memory+0x8e/0x2f5
Nov 2 12:59:48 mindy kernel: [<ffffffff8009dba4>] autoremove_wake_function+0x0/0x2e
Nov 2 12:59:48 mindy kernel: [<ffffffff8000f2eb>] __alloc_pages+0x245/0x2ce
Nov 2 12:59:48 mindy kernel: [<ffffffff8000f10b>] __alloc_pages+0x65/0x2ce
Nov 2 12:59:48 mindy kernel: [<ffffffff80017493>] cache_grow+0x137/0x395
Nov 2 12:59:48 mindy kernel: [<ffffffff8005bbf7>] cache_alloc_refill+0x136/0x186
Nov 2 12:59:48 mindy kernel: [<ffffffff8000a96e>] kmem_cache_alloc+0x6c/0x76
Nov 2 12:59:48 mindy kernel: [<ffffffff80043ae3>] sk_alloc+0x2e/0xf3
Nov 2 12:59:48 mindy kernel: [<ffffffff80059676>] inet_create+0x137/0x267
Nov 2 12:59:49 mindy kernel: [<ffffffff8004c9af>] __sock_create+0x170/0x27c
Nov 2 12:59:49 mindy kernel: [<ffffffff8839086e>] :dlm:process_send_sockets+0x0/0x179
Nov 2 12:59:49 mindy kernel: [<ffffffff883902f4>] :dlm:tcp_connect_to_sock+0x70/0x1de
Nov 2 12:59:49 mindy kernel: [<ffffffff80063097>] thread_return+0x62/0xfe
Nov 2 12:59:49 mindy kernel: [<ffffffff8839088e>] :dlm:process_send_sockets+0x20/0x179
Nov 2 12:59:49 mindy kernel: [<ffffffff8839086e>] :dlm:process_send_sockets+0x0/0x179
Nov 2 12:59:49 mindy kernel: [<ffffffff8004d159>] run_workqueue+0x94/0xe4
Nov 2 12:59:49 mindy kernel: [<ffffffff800499da>] worker_thread+0x0/0x122
Nov 2 12:59:49 mindy kernel: [<ffffffff8009d98c>] keventd_create_kthread+0x0/0xc4
Nov 2 12:59:49 mindy kernel: [<ffffffff80049aca>] worker_thread+0xf0/0x122
Nov 2 12:59:49 mindy kernel: [<ffffffff8008a4b3>] default_wake_function+0x0/0xe
Nov 2 12:59:49 mindy kernel: [<ffffffff8009d98c>] keventd_create_kthread+0x0/0xc4
Nov 2 12:59:49 mindy kernel: [<ffffffff8009d98c>] keventd_create_kthread+0x0/0xc4
Nov 2 12:59:49 mindy kernel: [<ffffffff80032380>] kthread+0xfe/0x132
Nov 2 12:59:49 mindy kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
Nov 2 12:59:49 mindy kernel: [<ffffffff8009d98c>] keventd_create_kthread+0x0/0xc4
Nov 2 12:59:49 mindy kernel: [<ffffffff8804e024>] :ext3:ext3_journal_dirty_data+0x0/0x34
Nov 2 12:59:49 mindy kernel: [<ffffffff80032282>] kthread+0x0/0x132
Nov 2 12:59:49 mindy kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
Nov 2 12:59:49 mindy kernel:
Both nodes are Qemu-KVM x86_64 guests, each one assigned 1Gb of ram and 2 cpus
I can send copy of cluster.conf eventually
Thanks in advance for your comments.
Gianluca
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
