David Teigland wrote:
Any messages before this? The best you could hope for with a SAN failure
is that all the cluster nodes withdraw gfs, allowing you to reboot them
without the panic. So, the end result wouldn't be all that different than
the panics.
This is the log just before the panic:
Jun 21 14:59:17 qmail-be-04 kernel: CMAN: removing node qmail-be-02 from
the cluster : Missed too many heartbeats
Jun 21 14:59:23 qmail-be-04 kernel: CMAN: removing node qmail-be-01 from
the cluster : No response to messages
Jun 21 14:59:29 qmail-be-04 kernel: CMAN: removing node qmail-be-06 from
the cluster : No response to messages
Jun 21 14:59:39 qmail-be-04 kernel: CMAN: removing node qmail-be-03 from
the cluster : No response to messages
Jun 21 14:59:46 qmail-be-04 kernel: CMAN: removing node qmail-be-05 from
the cluster : No response to messages
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: quorum lost, blocking activity
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: node qmail-be-04 has been
removed from the cluster : No response to messages
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: killed by NODEDOWN message
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: we are leaving the cluster. No
response to messages
Jun 21 14:59:52 qmail-be-04 kernel: WARNING: dlm_emergency_shutdown
Jun 21 14:59:52 qmail-be-04 fenced[17897]: process_events: service get
event failed
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 1000041 from 3 req 1
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003b from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003b from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003b from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003b from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 last message repeated 7 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003d from 3 req 3
Jun 21 14:59:53 qmail-be-04 last message repeated 6 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 1000041 from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003b from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003b from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 1
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 1000041 from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 1000041 from 3 req 1
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 1
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003b from 3 req 1
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003b from 3 req 5
Jun 21 14:59:53 qmail-be-04 last message repeated 5 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003d from 3 req 5
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003f from 3 req 5
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003d from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003d from 3 req 5
Jun 21 14:59:53 qmail-be-04 last message repeated 3 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003d from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid
lockspace 100003d from 3 req 5
Jun 21 14:59:54 qmail-be-04 last message repeated 20 times
Jun 21 14:59:54 qmail-be-04 kernel: dlm: dlm_unlock: lkid 3b013d
lockspace not found
Jun 21 14:59:54 qmail-be-04 kernel: store004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore001-001 add_to_requestq cmd 3 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore001-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-002 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-002 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore001-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-002 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: type 2 event 282 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28975 pr_start 282 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28975 pr_finish flags 1a
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start last_stop 273
last_start 283 last_finish 273
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start count 5 type 2 event
283 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start 283 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_finish flags 1a
Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start last_stop 283
last_start 285 last_finish 283
Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start count 6 type 2 event
285 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start 285 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_finish flags 1a
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start last_stop 116
last_start 287 last_finish 116
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start count 4 type 2 event
287 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start 287 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_finish flags 1a
Jun 21 14:59:55 qmail-be-04 kernel: 28975 rereq 2,1cec36 id e029f 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start last_stop 282
last_start 289 last_finish 282
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start count 5 type 2 event
289 flags 21a
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start 289 done 1
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_finish flags 1a
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start last_stop 287
last_start 291 last_finish 287
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start count 5 type 2 event
291 flags 21a
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start 291 done 1
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_finish flags 1a
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,2fe4b id a001e 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,3fd13 id 7007a 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,6faaf id 90009 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,2fdd6 id c0135 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,4fc8f id c023b 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,34a id 8011d 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,2fe4b id b03c3 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,6faaf id b001e 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,34a id 11000e 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,6faa8 id f0016 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,6faa8 id 1001a8 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,5fbd9 id f00e9 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,5fb4d id 802ac 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,5fbd9 id f0026 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,3fd13 id c009b 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,2fdd6 id 8001d 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 rereq 2,4fc8f id c0367 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 rereq 2,2fe40 id a01fd 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start last_stop 289
last_start 293 last_finish 289
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start count 6 type 2 event
293 flags 21a
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,7fa97 id 1502b3 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,3fcea id 702f3 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,5fc1e id 6015f 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,5fc1e id c01c1 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,2fdfa id c0362 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,2fdfa id c02ad 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,5fb4d id f01ce 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,7fa97 id d0293 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,3fcea id d02ad 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start 293 done 1
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_finish flags 1a
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start last_stop 118
last_start 295 last_finish 118
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start count 4 type 2 event
295 flags 21a
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start 295 done 1
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_finish flags 1a
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start last_stop 291
last_start 297 last_finish 291
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start count 6 type 2 event
297 flags 21a
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start 297 done 1
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_finish flags 1a
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start last_stop 295
last_start 299 last_finish 295
Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_start count 5 type 2 event
299 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_start 299 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start last_stop 299
last_start 301 last_finish 299
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start count 6 type 2 event
301 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start 301 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start last_stop 120
last_start 303 last_finish 120
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start count 4 type 2 event
303 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start 303 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start last_stop 122
last_start 305 last_finish 122
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start count 4 type 2 event
305 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start 305 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start last_stop 124
last_start 308 last_finish 124
Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start count 4 type 2 event
308 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start 308 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29457 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start last_stop 303
last_start 309 last_finish 303
Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start count 5 type 2 event
309 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start 309 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start last_stop 305
last_start 311 last_finish 305
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start count 5 type 2 event
311 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start 311 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start last_stop 309
last_start 313 last_finish 309
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start count 6 type 2 event
313 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start 313 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start last_stop 308
last_start 315 last_finish 308
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start count 5 type 2 event
315 flags 21a
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start 315 done 1
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start last_stop 311
last_start 317 last_finish 311
Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start count 6 type 2 event
317 flags 21a
Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start 317 done 1
Jun 21 14:59:58 qmail-be-04 kernel: 29408 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start last_stop 315
last_start 319 last_finish 315
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start count 6 type 2 event
319 flags 21a
Jun 21 14:59:58 qmail-be-04 kernel: 29457 rereq 2,2bd7b5 id 801e5 3,0
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start 319 done 1
Jun 21 14:59:58 qmail-be-04 kernel: 29457 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel:
Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm: Assertion failed on line
357 of file /soft/kernel/cluster-1.02.00/gfs-kernel/src/dlm/lock.c
Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm: assertion: "!error"
Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm: time = 2512697
Jun 21 14:59:58 qmail-be-04 kernel: mstore008-002: error=-22
num=2,7798a8 lkf=10001 flags=84
Is the second panic today :(
Thanks again
German
--
Linux-cluster@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/linux-cluster
