Hi, We have 16 nodes cluster. Recently facing issues with the nodes. The problem is, occassionaly find one of the nodes is not accessable through ssh. The node is up and running, the zen guests on the nodes are also pingable . But the node, and the guests on the nodes are not able to accessable. Very recently it happened to one of the node again. The nodes are of rhel5.5, kernel 2.6.18-194.3.1.el5xen #1 SMP Sun May 2 04:26:43 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux when it happens that node becomes detached from the cluster. If anybody can give some hints that will be really appreciated. Not sure it is the kernel but or not...Here is the few line of the log file, when it happened last time. Thanks in advance.. ____________________________________________________________ Jul 1 17:11:03 server crond[11715]: (root) CMD (python /usr/share/rhn/virtualization/poller.py) Jul 1 17:11:03 server crond[11716]: (root) CMD (python /usr/share/rhn/virtualization/poller.py) Jul 1 17:11:01 server crond[11685]: (root) error: Job execution of per-minute job scheduled for 17:10 delayed into subsequent minute 17:11. Skipping job run. Jul 1 17:11:03 server crond[11685]: CRON (root) ERROR: cannot set security context Jul 1 17:17:13 server xinetd[6778]: START: pblocald pid=11896 from=xxx.xx.222.4 Jul 1 17:21:01 server crond[11852]: (root) error: Job execution of per-minute job scheduled for 17:15 delayed into subsequent minute 17:21. Skipping job run. Jul 1 17:21:01 server crond[11852]: CRON (root) ERROR: cannot set security context Jul 1 17:21:05 server crond[12031]: (root) CMD (python /usr/share/rhn/virtualization/poller.py) Jul 1 17:23:34 server INFO: task cmahealthd:7492 blocked for more than 120 seconds. Jul 1 17:23:37 server "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 1 17:23:37 server cmahealthd D 0000000000000180 0 7492 1 7507 7430 (NOTLB) Jul 1 17:23:37 server ffff880065f29b18 0000000000000282 0000000000000000 0000000000000000 Jul 1 17:23:37 server 0000000000000009 ffff880065c35040 ffff88007f6720c0 000000000001d982 Jul 1 17:23:37 server ffff880065c35228 ffff88007e16b400 Jul 1 17:23:37 server Call Trace: Jul 1 17:23:37 server [<ffffffff80287795>] __wake_up_common+0x3e/0x68 Jul 1 17:23:37 server [<ffffffff802d81f3>] base_probe+0x0/0x36 Jul 1 17:23:37 server [<ffffffff80262fb3>] wait_for_completion+0x7d/0xaa Jul 1 17:23:52 server [<ffffffff80288f86>] default_wake_function+0x0/0xe Jul 1 17:23:52 server [<ffffffff80298e84>] call_usermodehelper_keys+0xe3/0xf8 Jul 1 17:23:52 server [<ffffffff80298e99>] __call_usermodehelper+0x0/0x4f Jul 1 17:23:52 server [<ffffffff802071b2>] find_get_page+0x4d/0x55 Jul 1 17:23:52 server [<ffffffff80299275>] request_module+0x139/0x14d Jul 1 17:23:52 server [<ffffffff8022cf67>] mntput_no_expire+0x19/0x89 Jul 1 17:23:52 server [<ffffffff8020edda>] link_path_walk+0xa6/0xb2 Jul 1 17:23:52 server [<ffffffff80263914>] mutex_lock+0xd/0x1d Jul 1 17:23:52 server [<ffffffff802d8211>] base_probe+0x1e/0x36 Jul 1 17:23:52 server [<ffffffff803af5c9>] kobj_lookup+0x132/0x19b Jul 1 17:31:24 server xinetd[6778]: START: pblocald pid=12151 from=xxx.xx.222.4 Jul 1 17:28:16 server openais[6172]: [TOTEM] entering GATHER state from 12. Jul 1 17:28:41 server openais[6172]: [TOTEM] Creating commit token because I am the rep. Jul 1 17:28:41 server openais[6172]: [TOTEM] Saving state aru 20a high seq received 20a Jul 1 17:28:41 server openais[6172]: [TOTEM] Storing new sequence id for ring 2e90 Jul 1 17:28:49 server openais[6172]: [TOTEM] entering COMMIT state. Jul 1 17:31:30 server openais[6172]: [TOTEM] Creating commit token because I am the rep. Jul 1 17:31:30 server openais[6172]: [TOTEM] Storing new sequence id for ring 2e94 Jul 1 17:31:30 server openais[6172]: [TOTEM] entering COMMIT state. Jul 1 17:31:30 server openais[6172]: [TOTEM] entering GATHER state from 13. Jul 1 17:31:30 server openais[6172]: [TOTEM] Creating commit token because I am the rep. Jul 1 17:33:30 server [<ffffffff8024b204>] chrdev_open+0x53/0x183 Jul 1 17:33:30 server [<ffffffff8024b1b1>] chrdev_open+0x0/0x183 Jul 1 17:33:30 server [<ffffffff8021edc8>] __dentry_open+0xd9/0x1dc Jul 1 17:33:30 server [<ffffffff80227bca>] do_filp_open+0x2a/0x38 Jul 1 17:33:30 server [<ffffffff8021a270>] do_sys_open+0x44/0xbe Jul 1 17:33:30 server [<ffffffff8026168d>] ia32_sysret+0x0/0x5 Jul 1 17:33:30 server Jul 1 17:31:30 server openais[6172]: [TOTEM] Storing new sequence id for ring 2e98 Jul 1 17:31:30 server openais[6172]: [TOTEM] entering COMMIT state. Jul 1 17:31:30 server openais[6172]: [TOTEM] entering RECOVERY state. Jul 1 17:31:30 server openais[6172]: [TOTEM] position [0] member 192.168.xxx.9: Jul 1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11916 rep 192.168.xxx.9 Jul 1 17:31:30 server openais[6172]: [TOTEM] aru 20a high delivered 20a received flag 1 Jul 1 17:31:30 server openais[6172]: [TOTEM] position [1] member 192.168.xxx.10: Jul 1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11924 rep 192.168.xxx.10 Jul 1 17:31:30 server openais[6172]: [TOTEM] aru e2 high delivered e2 received flag 1 Jul 1 17:31:30 server openais[6172]: [TOTEM] position [2] member 192.168.xxx.11: Jul 1 17:33:30 server INFO: task cmahealthd:7492 blocked for more than 120 seconds. Jul 1 17:33:30 server "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 1 17:33:30 server cmahealthd D 0000000000000180 0 7492 1 7507 7430 (NOTLB) Jul 1 17:33:30 server ffff880065f29b18 0000000000000282 0000000000000000 0000000000000000 Jul 1 17:33:30 server 0000000000000009 ffff880065c35040 ffff88007f6720c0 000000000001d982 Jul 1 17:33:30 server ffff880065c35228 ffff88007e16b400 Jul 1 17:33:30 server Call Trace: Jul 1 17:33:30 server [<ffffffff80287795>] __wake_up_common+0x3e/0x68 Jul 1 17:33:30 server [<ffffffff802d81f3>] base_probe+0x0/0x36 Jul 1 17:33:30 server [<ffffffff80262fb3>] wait_for_completion+0x7d/0xaa Jul 1 17:33:29 server dlm_controld[6272]: cluster is down, exiting Jul 1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11924 rep 192.168.xxx.10 Jul 1 17:31:30 server openais[6172]: [TOTEM] aru e2 high delivered e2 received flag 1 Jul 1 17:31:30 server openais[6172]: [TOTEM] position [3] member 192.168.xxx.12: Jul 1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11924 rep 192.168.xxx.10 Jul 1 17:31:30 server openais[6172]: [TOTEM] aru e2 high delivered e2 received flag 1 Jul 1 17:31:30 server openais[6172]: [TOTEM] position [4] member 192.168.xxx.13: Jul 1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11924 rep 192.168.xxx.10 Jul 1 17:31:30 server openais[6172]: [TOTEM] aru e2 high delivered e2 received flag 1 Jul 1 17:31:30 server openais[6172]: [TOTEM] position [5] member 192.168.xxx.14: Jul 1 17:31:30 server openais[6172]: [TOTEM] previous ring seq 11924 rep 192.168.xxx.10 Jul 1 17:33:30 server [<ffffffff80288f86>] default_wake_function+0x0/0xe Jul 1 17:33:30 server [<ffffffff80298e84>] call_usermodehelper_keys+0xe3/0xf8 Jul 1 17:33:30 server [<ffffffff80298e99>] __call_usermodehelper+0x0/0x4f Jul 1 17:33:30 server [<ffffffff802071b2>] find_get_page+0x4d/0x55 Jul 1 17:33:30 server [<ffffffff80299275>] request_module+0x139/0x14d Jul 1 17:33:30 server [<ffffffff8022cf67>] mntput_no_expire+0x19/0x89 Jul 1 17:33:30 server [<ffffffff8020edda>] link_path_walk+0xa6/0xb2 Jul 1 17:33:30 server [<ffffffff80263914>] mutex_lock+0xd/0x1d Jul 1 17:33:30 server [<ffffffff802d8211>] base_probe+0x1e/0x36 Jul 1 17:33:30 server [<ffffffff803af5c9>] kobj_lookup+0x132/0x19b Jul 1 17:33:30 server gfs_controld[6280]: cluster is down, exiting -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
