On 02/22/2014 11:10 AM, cluster lab wrote: > hi, > > At the middle of cluster activity i received this messages: (cluster > is 3 node with SAN ... GFS2 filesystem) OS? version of the packages? cluster.conf > > log messages on USBAck-prox2: > > Feb 21 13:06:41 USBack-prox2 corosync[3911]: [QUORUM] Members[2]: 2 3 > Feb 21 13:06:41 USBack-prox2 corosync[3911]: [TOTEM ] A processor > joined or left the membership and a new membership was formed. > Feb 21 13:06:41 USBack-prox2 rgmanager[4130]: State change: USBack-prox1 DOWN > Feb 21 13:06:41 USBack-prox2 kernel: dlm: closing connection to node 1 > Feb 21 13:06:41 USBack-prox2 corosync[3911]: [CPG ] downlist received > left_list: 1 > Feb 21 13:06:41 USBack-prox2 corosync[3911]: [CPG ] downlist received > left_list: 1 > Feb 21 13:06:41 USBack-prox2 corosync[3911]: [CPG ] chosen downlist > from node r(0) ip(--.--.--.22) > Feb 21 13:06:41 USBack-prox2 corosync[3911]: [MAIN ] Completed service > synchronization, ready to provide service. > Feb 21 13:06:41 USBack-prox2 kernel: GFS2: > fsid=USBackCluster:VMStorage1.0: jid=1: Trying to acquire journal > lock... > Feb 21 13:06:41 USBack-prox2 kernel: GFS2: > fsid=USBackCluster:VMStorage2.0: jid=1: Trying to acquire journal > lock... > Feb 21 13:06:51 USBack-prox2 fenced[3957]: fencing node USBack-prox1 > Feb 21 13:06:52 USBack-prox2 fenced[3957]: fence USBack-prox1 dev 0.0 > agent fence_ipmilan result: error from agent > Feb 21 13:06:52 USBack-prox2 fenced[3957]: fence USBack-prox1 failed > Feb 21 13:06:54 USBack-prox2 kernel: dlm: connect from non cluster node > Feb 21 13:06:54 USBack-prox2 kernel: dlm: connect from non cluster node ^^^ good hint here. something is off. Fabio > Feb 21 13:06:55 USBack-prox2 corosync[3911]: [TOTEM ] A processor > joined or left the membership and a new membership was formed. > Feb 21 13:06:55 USBack-prox2 corosync[3911]: [QUORUM] Members[3]: 1 2 3 > Feb 21 13:06:55 USBack-prox2 corosync[3911]: [QUORUM] Members[3]: 1 2 3 > Feb 21 13:06:55 USBack-prox2 rgmanager[4130]: State change: USBack-prox1 UP > Feb 21 13:06:55 USBack-prox2 corosync[3911]: [CPG ] downlist received > left_list: 2 > Feb 21 13:06:55 USBack-prox2 corosync[3911]: [CPG ] downlist received > left_list: 0 > Feb 21 13:06:55 USBack-prox2 corosync[3911]: [CPG ] downlist received > left_list: 0 > Feb 21 13:06:55 USBack-prox2 corosync[3911]: [CPG ] chosen downlist > from node r(0) ip(--.--.--.21) > Feb 21 13:06:55 USBack-prox2 corosync[3911]: [MAIN ] Completed service > synchronization, ready to provide service. > Feb 21 13:06:55 USBack-prox2 gfs_controld[4029]: cpg_mcast_joined > error 12 handle 3a95f87400000000 protocol > Feb 21 13:06:55 USBack-prox2 gfs_controld[4029]: cpg_mcast_joined > error 12 handle 1e7ff52100000001 start > Feb 21 13:06:55 USBack-prox2 gfs_controld[4029]: cpg_mcast_joined > error 12 handle 22221a7000000002 start > Feb 21 13:06:55 USBack-prox2 gfs_controld[4029]: cpg_mcast_joined > error 12 handle 419ac24100000003 start > Feb 21 13:06:55 USBack-prox2 gfs_controld[4029]: cpg_mcast_joined > error 12 handle 3804823e00000004 start > > > ------------------------------------------------- > Then GFS2 generates error logs (Activities blocked). > > Logs of cisco switch (Time is UTC): > > Feb 21 09:37:02.375: %LINEPROTO-5-UPDOWN: Line protocol on Interface > GigabitEthernet0/11, changed state to down > Feb 21 09:37:02.459: %LINEPROTO-5-UPDOWN: Line protocol on Interface > GigabitEthernet0/4, changed state to down > Feb 21 09:37:03.382: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, > changed state to down > Feb 21 09:37:03.541: %LINK-3-UPDOWN: Interface GigabitEthernet0/4, > changed state to down > Feb 21 09:37:07.283: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, > changed state to up > Feb 21 09:37:07.350: %LINK-3-UPDOWN: Interface GigabitEthernet0/4, > changed state to up > Feb 21 09:37:08.289: %LINEPROTO-5-UPDOWN: Line protocol on Interface > GigabitEthernet0/11, changed state to up > Feb 21 09:37:09.472: %LINEPROTO-5-UPDOWN: Line protocol on Interface > GigabitEthernet0/4, changed state to up > Feb 21 09:40:20.045: %LINEPROTO-5-UPDOWN: Line protocol on Interface > GigabitEthernet0/11, changed state to down > Feb 21 09:40:21.043: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, > changed state to down > Feb 21 09:40:23.401: %LINK-3-UPDOWN: Interface GigabitEthernet0/11, > changed state to up > _______________________________________________ > discuss mailing list > discuss@xxxxxxxxxxxx > http://lists.corosync.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss@xxxxxxxxxxxx http://lists.corosync.org/mailman/listinfo/discuss
