Re: GlusterFS hangs/fails: Transport endpoint is not connected

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

crawling through all /var/log/messages, I found on one of the failing nodes (node68)

Nov 25 04:04:12 node68 kernel: INFO: task pw.x:20052 blocked for more than 120 seconds. Nov 25 04:04:12 node68 kernel: "echo 0 > /proc/sys/kernel/ hung_task_timeout_secs" disables this message. Nov 25 04:04:12 node68 kernel: pw.x D ffff81027c3d5d68 0 20052 1 Nov 25 04:04:12 node68 kernel: ffff81027c3d5d48 0000000000000086 ffff81021c0e7460 0000000000000000 Nov 25 04:04:12 node68 kernel: ffff81041f14e800 000000038022a7ae ffff81041f314238 ffff81041f314000 Nov 25 04:04:12 node68 kernel: 0000000000000000 0000000000000001 0000000000000246 0000000000000003
Nov 25 04:04:12 node68 kernel: Call Trace:
Nov 25 04:04:12 node68 kernel: [<ffffffff882ae9c7>] :fuse:request_send +0x2c8/0x2f0 Nov 25 04:04:12 node68 kernel: [<ffffffff80242ab3>] autoremove_wake_function+0x0/0x2e Nov 25 04:04:12 node68 kernel: [<ffffffff80242ab3>] autoremove_wake_function+0x0/0x2e Nov 25 04:04:12 node68 kernel: [<ffffffff882ae037>] :fuse:fuse_request_init+0x2f/0x38 Nov 25 04:04:12 node68 kernel: [<ffffffff882b1761>] :fuse:fuse_open_common+0xef/0x15e Nov 25 04:04:12 node68 kernel: [<ffffffff882b188e>] :fuse:fuse_open +0x0/0x7 Nov 25 04:04:12 node68 kernel: [<ffffffff80286e30>] __dentry_open +0xe6/0x1ba Nov 25 04:04:12 node68 kernel: [<ffffffff80286f2a>] nameidata_to_filp +0x26/0x35 Nov 25 04:04:12 node68 kernel: [<ffffffff80286f66>] do_filp_open+0x2d/ 0x3d Nov 25 04:04:12 node68 kernel: [<ffffffff80287180>] get_unused_fd_flags+0x104/0x113 Nov 25 04:04:12 node68 kernel: [<ffffffff802872a3>] do_sys_open +0x46/0xc3 Nov 25 04:04:12 node68 kernel: [<ffffffff8020b08b>] system_call_after_swapgs+0x7b/0x80
Nov 25 04:04:12 node68 kernel:
Nov 25 04:04:12 node68 kernel: INFO: task pw.x:20053 blocked for more than 120 seconds. Nov 25 04:04:12 node68 kernel: "echo 0 > /proc/sys/kernel/ hung_task_timeout_secs" disables this message. Nov 25 04:04:12 node68 kernel: pw.x D ffff8101c5083d68 0 20053 1 Nov 25 04:04:12 node68 kernel: ffff8101c5083d48 0000000000000086 ffff81021c0e7460 0000000000000000 Nov 25 04:04:12 node68 kernel: ffff81041f14a800 000000008022a7ae ffff81021d8b9238 ffff81021d8b9000 Nov 25 04:04:12 node68 kernel: 0000000000000000 0000000000000001 0000000000000246 0000000000000003
Nov 25 04:04:12 node68 kernel: Call Trace:
Nov 25 04:04:12 node68 kernel: [<ffffffff882ae9c7>] :fuse:request_send +0x2c8/0x2f0 Nov 25 04:04:12 node68 kernel: [<ffffffff80242ab3>] autoremove_wake_function+0x0/0x2e Nov 25 04:04:12 node68 kernel: [<ffffffff80242ab3>] autoremove_wake_function+0x0/0x2e Nov 25 04:04:12 node68 kernel: [<ffffffff882ae037>] :fuse:fuse_request_init+0x2f/0x38 Nov 25 04:04:12 node68 kernel: [<ffffffff882b1761>] :fuse:fuse_open_common+0xef/0x15e Nov 25 04:04:12 node68 kernel: [<ffffffff882b188e>] :fuse:fuse_open +0x0/0x7 Nov 25 04:04:12 node68 kernel: [<ffffffff80286e30>] __dentry_open +0xe6/0x1ba Nov 25 04:04:12 node68 kernel: [<ffffffff80286f2a>] nameidata_to_filp +0x26/0x35 Nov 25 04:04:12 node68 kernel: [<ffffffff80286f66>] do_filp_open+0x2d/ 0x3d Nov 25 04:04:12 node68 kernel: [<ffffffff80287180>] get_unused_fd_flags+0x104/0x113 Nov 25 04:04:12 node68 kernel: [<ffffffff802872a3>] do_sys_open +0x46/0xc3 Nov 25 04:04:12 node68 kernel: [<ffffffff8020b08b>] system_call_after_swapgs+0x7b/0x80
Nov 25 04:04:12 node68 kernel:

The other two failing nodes had nothing related in the logs. Note that pw.x:20052 and pw.x:20053 are the two parallel jobs running on this node.

A similar error was logged during the crash two days ago on node22:

Nov 23 14:16:43 node22 kernel: INFO: task pw.x:32355 blocked for more than 120 seconds. Nov 23 14:16:43 node22 kernel: "echo 0 > /proc/sys/kernel/ hung_task_timeout_secs" disables this message. Nov 23 14:16:43 node22 kernel: pw.x D ffff8102049c1d68 0 32355 1 Nov 23 14:16:43 node22 kernel: ffff8102049c1d48 0000000000000082 ffff81013e0e1c60 0000000000000000 Nov 23 14:16:43 node22 kernel: ffff81021e4ea000 000000038022a7ae ffff81021f004a38 ffff81021f004800 Nov 23 14:16:43 node22 kernel: 0000000000000000 0000000000000001 0000000000000246 0000000000000003
Nov 23 14:16:43 node22 kernel: Call Trace:
Nov 23 14:16:43 node22 kernel: [<ffffffff882ae9c7>] :fuse:request_send +0x2c8/0x2f0 Nov 23 14:16:43 node22 kernel: [<ffffffff80242ab3>] autoremove_wake_function+0x0/0x2e Nov 23 14:16:43 node22 kernel: [<ffffffff80242ab3>] autoremove_wake_function+0x0/0x2e Nov 23 14:16:43 node22 kernel: [<ffffffff882ae037>] :fuse:fuse_request_init+0x2f/0x38 Nov 23 14:16:43 node22 kernel: [<ffffffff882b1761>] :fuse:fuse_open_common+0xef/0x15e Nov 23 14:16:43 node22 kernel: [<ffffffff882b188e>] :fuse:fuse_open +0x0/0x7 Nov 23 14:16:43 node22 kernel: [<ffffffff80286e30>] __dentry_open +0xe6/0x1ba Nov 23 14:16:43 node22 kernel: [<ffffffff80286f2a>] nameidata_to_filp +0x26/0x35 Nov 23 14:16:43 node22 kernel: [<ffffffff80286f66>] do_filp_open+0x2d/ 0x3d Nov 23 14:16:43 node22 kernel: [<ffffffff80287180>] get_unused_fd_flags+0x104/0x113 Nov 23 14:16:43 node22 kernel: [<ffffffff802872a3>] do_sys_open +0x46/0xc3 Nov 23 14:16:43 node22 kernel: [<ffffffff8020b08b>] system_call_after_swapgs+0x7b/0x80
Nov 23 14:16:43 node22 kernel:

That's all in /var/log/messages. Remember that the program "pw.x" runs without problems via NFS as that this is the only program used for testing presently.

Fred

On 25.11.2008, at 13:42, Joe Landman wrote:

Fred Hucht wrote:
Hi!
The glusterfsd.log on all nodes are virtually empty, the only entry on 2008-11-25 reads 2008-11-25 03:13:48 E [io-threads.c:273:iot_flush] sc1-ioth: fd context is NULL, returning EBADFD
on all nodes. I don't think that this is related to our problems.
Regards,
    Fred

Hi Fred

Could you post complete /var/log/messages file on pastebin? I have seen something like this before when fuse crashes. Fuse crashing could be due to a bug in fuse, the kernel, etc. Also could be hardware that is failing.

 Does an unmount/remount fix the problem?

Joe


--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman@xxxxxxxxxxxxxxxxxxxxxxx
web  : http://www.scalableinformatics.com
      http://jackrabbit.scalableinformatics.com
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

Dr. Fred Hucht <fred@xxxxxxxxxxxxxx>
Institute for Theoretical Physics
University of Duisburg-Essen, 47048 Duisburg, Germany





[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux