On Wed, Jun 21, 2006 at 03:41:58PM -0300, German Staltari wrote: > Jun 21 14:59:17 qmail-be-04 kernel: CMAN: removing node qmail-be-02 from > the cluster : Missed too many heartbeats > Jun 21 14:59:23 qmail-be-04 kernel: CMAN: removing node qmail-be-01 from > the cluster : No response to messages > Jun 21 14:59:29 qmail-be-04 kernel: CMAN: removing node qmail-be-06 from > the cluster : No response to messages > Jun 21 14:59:39 qmail-be-04 kernel: CMAN: removing node qmail-be-03 from > the cluster : No response to messages > Jun 21 14:59:46 qmail-be-04 kernel: CMAN: removing node qmail-be-05 from > the cluster : No response to messages > Jun 21 14:59:52 qmail-be-04 kernel: CMAN: quorum lost, blocking activity > Jun 21 14:59:52 qmail-be-04 kernel: CMAN: node qmail-be-04 has been > removed from the cluster : No response to messages > Jun 21 14:59:52 qmail-be-04 kernel: CMAN: killed by NODEDOWN message > Jun 21 14:59:52 qmail-be-04 kernel: CMAN: we are leaving the cluster. No > response to messages This is what led to the gfs panic, the cluster shut down when it lost contact with all the other nodes. Dave > Jun 21 14:59:52 qmail-be-04 kernel: WARNING: dlm_emergency_shutdown > Jun 21 14:59:52 qmail-be-04 fenced[17897]: process_events: service get > event failed > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 1000041 from 3 req 1 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003b from 3 req 3 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003b from 3 req 9 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003b from 3 req 9 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003b from 3 req 3 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 3 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 9 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 3 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 3 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 9 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 3 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 9 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 3 > Jun 21 14:59:53 qmail-be-04 last message repeated 7 times > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003d from 3 req 3 > Jun 21 14:59:53 qmail-be-04 last message repeated 6 times > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 1000041 from 3 req 3 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003b from 3 req 3 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003b from 3 req 3 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 1 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 1000041 from 3 req 9 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 9 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 1000041 from 3 req 1 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 1 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003b from 3 req 1 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003b from 3 req 5 > Jun 21 14:59:53 qmail-be-04 last message repeated 5 times > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003d from 3 req 5 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003f from 3 req 5 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003d from 3 req 9 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003d from 3 req 5 > Jun 21 14:59:53 qmail-be-04 last message repeated 3 times > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003d from 3 req 9 > Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid > lockspace 100003d from 3 req 5 > Jun 21 14:59:54 qmail-be-04 last message repeated 20 times > Jun 21 14:59:54 qmail-be-04 kernel: dlm: dlm_unlock: lkid 3b013d > lockspace not found > Jun 21 14:59:54 qmail-be-04 kernel: store004-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore002-004 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore001-001 add_to_requestq cmd 3 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore001-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore004-004 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore002-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore003-004 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore002-002 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore002-004 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore002-002 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore001-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore002-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore003-002 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore004-004 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3 > Jun 21 14:59:54 qmail-be-04 kernel: type 2 event 282 flags 21a > Jun 21 14:59:54 qmail-be-04 kernel: 28975 pr_start 282 done 1 > Jun 21 14:59:54 qmail-be-04 kernel: 28975 pr_finish flags 1a > Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start last_stop 273 > last_start 283 last_finish 273 > Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start count 5 type 2 event > 283 flags 21a > Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start 283 done 1 > Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_finish flags 1a > Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start last_stop 283 > last_start 285 last_finish 283 > Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start count 6 type 2 event > 285 flags 21a > Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start 285 done 1 > Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_finish flags 1a > Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start last_stop 116 > last_start 287 last_finish 116 > Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start count 4 type 2 event > 287 flags 21a > Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start 287 done 1 > Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_finish flags 1a > Jun 21 14:59:55 qmail-be-04 kernel: 28975 rereq 2,1cec36 id e029f 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start last_stop 282 > last_start 289 last_finish 282 > Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start count 5 type 2 event > 289 flags 21a > Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start 289 done 1 > Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_finish flags 1a > Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start last_stop 287 > last_start 291 last_finish 287 > Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start count 5 type 2 event > 291 flags 21a > Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start 291 done 1 > Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_finish flags 1a > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,2fe4b id a001e 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,3fd13 id 7007a 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,6faaf id 90009 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,2fdd6 id c0135 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,4fc8f id c023b 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,34a id 8011d 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,2fe4b id b03c3 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,6faaf id b001e 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,34a id 11000e 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,6faa8 id f0016 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,6faa8 id 1001a8 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,5fbd9 id f00e9 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,5fb4d id 802ac 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,5fbd9 id f0026 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,3fd13 id c009b 3,0 > Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,2fdd6 id 8001d 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28974 rereq 2,4fc8f id c0367 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28974 rereq 2,2fe40 id a01fd 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start last_stop 289 > last_start 293 last_finish 289 > Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start count 6 type 2 event > 293 flags 21a > Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,7fa97 id 1502b3 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,3fcea id 702f3 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,5fc1e id 6015f 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,5fc1e id c01c1 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,2fdfa id c0362 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,2fdfa id c02ad 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,5fb4d id f01ce 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,7fa97 id d0293 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,3fcea id d02ad 3,0 > Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start 293 done 1 > Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_finish flags 1a > Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start last_stop 118 > last_start 295 last_finish 118 > Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start count 4 type 2 event > 295 flags 21a > Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start 295 done 1 > Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_finish flags 1a > Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start last_stop 291 > last_start 297 last_finish 291 > Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start count 6 type 2 event > 297 flags 21a > Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start 297 done 1 > Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_finish flags 1a > Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start last_stop 295 > last_start 299 last_finish 295 > Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_start count 5 type 2 event > 299 flags 21a > Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_start 299 done 1 > Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_finish flags 1a > Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start last_stop 299 > last_start 301 last_finish 299 > Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start count 6 type 2 event > 301 flags 21a > Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start 301 done 1 > Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_finish flags 1a > Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start last_stop 120 > last_start 303 last_finish 120 > Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start count 4 type 2 event > 303 flags 21a > Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start 303 done 1 > Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a > Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start last_stop 122 > last_start 305 last_finish 122 > Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start count 4 type 2 event > 305 flags 21a > Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start 305 done 1 > Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_finish flags 1a > Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start last_stop 124 > last_start 308 last_finish 124 > Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start count 4 type 2 event > 308 flags 21a > Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start 308 done 1 > Jun 21 14:59:57 qmail-be-04 kernel: 29457 pr_finish flags 1a > Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start last_stop 303 > last_start 309 last_finish 303 > Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start count 5 type 2 event > 309 flags 21a > Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start 309 done 1 > Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a > Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start last_stop 305 > last_start 311 last_finish 305 > Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start count 5 type 2 event > 311 flags 21a > Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start 311 done 1 > Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_finish flags 1a > Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start last_stop 309 > last_start 313 last_finish 309 > Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start count 6 type 2 event > 313 flags 21a > Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start 313 done 1 > Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a > Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start last_stop 308 > last_start 315 last_finish 308 > Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start count 5 type 2 event > 315 flags 21a > Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start 315 done 1 > Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_finish flags 1a > Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start last_stop 311 > last_start 317 last_finish 311 > Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start count 6 type 2 event > 317 flags 21a > Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start 317 done 1 > Jun 21 14:59:58 qmail-be-04 kernel: 29408 pr_finish flags 1a > Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start last_stop 315 > last_start 319 last_finish 315 > Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start count 6 type 2 event > 319 flags 21a > Jun 21 14:59:58 qmail-be-04 kernel: 29457 rereq 2,2bd7b5 id 801e5 3,0 > Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start 319 done 1 > Jun 21 14:59:58 qmail-be-04 kernel: 29457 pr_finish flags 1a > Jun 21 14:59:58 qmail-be-04 kernel: > Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm: Assertion failed on line > 357 of file /soft/kernel/cluster-1.02.00/gfs-kernel/src/dlm/lock.c > Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm: assertion: "!error" > Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm: time = 2512697 > Jun 21 14:59:58 qmail-be-04 kernel: mstore008-002: error=-22 > num=2,7798a8 lkf=10001 flags=84 > > Is the second panic today :( > Thanks again > German -- Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
