AMD64 dual CPU/duo core HP DL385, 8GB RAM, dual hba (PowerPath)
# uname -r
2.6.9-42.0.10.ELsmp
ccs-1.0.7-0
cman-1.0.11-0
dlm-1.0.1-1
fence-1.32.25-1
GFS-6.1.6-1
magma-1.0.6-0
magma-plugins-1.0.9-0
rgmanager-1.9.54-1
My central syslog server showed that all nodes registered the membership change, yet the service continued to hang.
Mar 20 11:06:18 app1 shutdown: shutting down for system reboot
Mar 20 11:06:18 app1 init: Switching to runlevel: 6
Mar 20 11:06:19 app1 rgmanager: [1873]: <notice> Shutting down Cluster Service Manager...
Mar 20 11:06:20 app1 clurgmgrd[11220]: <notice> Shutting down
Mar 20 11:06:20 net2 clurgmgrd[30893]: <info> State change: app1 DOWN
Mar 20 11:06:20 app3 clurgmgrd[11092]: <info> State change: app1 DOWN
Mar 20 11:06:20 db1 clurgmgrd[8351]: <info> State change: app1 DOWN
Mar 20 11:06:20 db3 clurgmgrd[8279]: <info> State change: app1 DOWN
Mar 20 11:06:20 db2 clurgmgrd[10875]: <info> State change: app1 DOWN
Mar 20 11:06:20 app6 clurgmgrd[10959]: <info> State change: app1 DOWN
Mar 20 11:06:20 app4 clurgmgrd[11146]: <info> State change: app1 DOWN
Mar 20 11:06:20 app2 clurgmgrd[10835]: <info> State change: app1 DOWN
Mar 20 11:06:20 app5 clurgmgrd[11198]: <info> State change: app1 DOWN
Mar 20 11:06:20 net1 clurgmgrd[12689]: <info> State change: app1 DOWN
Mar 20 11:12:26 net2 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats
Mar 20 11:12:26 db2 kernel: CMAN: removing node app1 from the cluster : Missed too many heartbeats
Mar 20 11:12:26 db3 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats
Mar 20 11:12:26 app4 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats
Mar 20 11:12:26 app5 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats
Mar 20 11:12:26 app6 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats
Mar 20 11:12:26 net1 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats
Mar 20 11:12:26 app3 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats
Mar 20 11:12:26 db1 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats
Mar 20 11:12:26 app2 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats
Mar 20 11:12:32 net1 fenced[10510]: app1 not a cluster member after 0 sec post_fail_delay
Mar 20 11:12:32 net1 fenced[10510]: fencing node "app1"
Mar 20 11:13:42 net1 fenced[10510]: fence "app1" success
I issued a 'power reset' on its HP ILO management port to hardware reboot the server around 11:12. That is when the net1 server attempted to fence app1, after it was missing. Here's net1's syslog entries on that event:
Mar 20 11:06:20 net1 clurgmgrd[12689]: <info> Magma Event: Membership Change
Mar 20 11:06:20 net1 clurgmgrd[12689]: <info> State change: app1 DOWN
Mar 20 11:12:26 net1 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats
Mar 20 11:12:32 net1 fenced[10510]: app1 not a cluster member after 0 sec post_fail_delay
Mar 20 11:12:32 net1 fenced[10510]: fencing node "app1"
Mar 20 11:13:42 net1 fenced[10510]: fence "app1" success
Mar 20 11:15:45 net1 kernel: CMAN: node app1 rejoining
Mar 20 11:18:05 net1 clurgmgrd[12689]: <info> Magma Event: Membership Change
Mar 20 11:18:05 net1 clurgmgrd[12689]: <info> State change: app1 UP
|
Robert Hurst, Sr. Caché Administrator Beth Israel Deaconess Medical Center 1135 Tremont Street, REN-7 Boston, Massachusetts 02120-2140 617-754-8754 ∙ Fax: 617-754-8730 ∙ Cell: 401-787-3154 Any technology distinguishable from magic is insufficiently advanced. |
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
