Hi,
We have a web server installation that is using glusterFS. There are 3
GlusterFS servers and 4 GlusterFS clients acessing one share. We're
having a problem that randomly, the httpd processes begin to hung
while waiting for I/O. This begins to happen with the following kernel
call trace:
kernel: INFO: task httpd.worker:18974 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
kernel: httpd.worker D ffffffff80150462 0 18974 4818
18975 18972 (NOTLB)
kernel: ffff8108c0ee7c78 0000000000000086 ffff8101f3b2b000 ffff810116d0f200
kernel: ffff81037392c6d8 000000000000000a ffff8106b73d57a0 ffff810116dd4100
kernel: 00004f0baea8a21e 00000000000026f5 ffff8106b73d5988 0000000300000000
kernel: Call Trace:
kernel: [<ffffffff88443cda>] :fuse:fuse_dentry_revalidate+0x13e/0x1c1
kernel: [<ffffffff80063c6f>] __mutex_lock_slowpath+0x60/0x9b
kernel: [<ffffffff80063cb9>] .text.lock.mutex+0xf/0x14
kernel: [<ffffffff8000cec2>] do_lookup+0x90/0x1e6
kernel: [<ffffffff80009bb2>] __link_path_walk+0x3a6/0xf42
kernel: [<ffffffff8000e9e2>] link_path_walk+0x42/0xb2
kernel: [<ffffffff8000ccb2>] do_path_lookup+0x275/0x2f1
kernel: [<ffffffff8001280e>] getname+0x15b/0x1c2
kernel: [<ffffffff80023876>] __user_walk_fd+0x37/0x4c
kernel: [<ffffffff80028846>] vfs_stat_fd+0x1b/0x4a
kernel: [<ffffffff8000c5fe>] _atomic_dec_and_lock+0x39/0x57
kernel: [<ffffffff800235a8>] sys_newstat+0x19/0x31
kernel: [<ffffffff80012b54>] __fput+0x191/0x1bd
kernel: [<ffffffff8002c9e4>] mntput_no_expire+0x19/0x89
kernel: [<ffffffff80023b84>] filp_close+0x5c/0x64
kernel: [<ffffffff8001dfa6>] sys_close+0x88/0xbd
kernel: [<ffffffff8005d116>] system_call+0x7e/0x83
Looking at the glusterfs client logs, the following entries begin to
show up before this call trace begins:
[fuse-bridge.c:585:fuse_lookup] glusterfs-fuse: 89023783: LOOKUP
46912563008752/2010 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024196: GETATTR
46912563008752 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024203: GETATTR
46912563008752 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024207: GETATTR
46912563008752 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024210: GETATTR
46912563008752 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024213: GETATTR
46912563008752 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024218: GETATTR
46912563008752 (fuse_loc_fill() failed)
The same message repeats 1116 times. Talking with a friend, we noticed
that the 4 web servers are reading *and* writing very concurrently and
because of this, there are many lookups, reads and unlinks returning
"no such file or directory" on the log, like these:
[fuse-bridge.c:859:fuse_fd_cbk] glusterfs-fuse: 88969079: OPEN()
/cache/70ea5f0d4df01648be4f70d9813f24b7.meta => -1 (No such file or
directory)
[fuse-bridge.c:1215:fuse_unlink_cbk] glusterfs-fuse: 88969080:
UNLINK() /cache/70ea5f0d4df01648be4f70d9813f24b7.meta => -1 (No such
file or directory)
[fuse-bridge.c:491:fuse_entry_cbk] glusterfs-fuse:
LOOKUP(/cache/index.html) inode (ptr=0x2aaab313f9f0, ino=1528908,
gen=5480759143103581058) found conflict (ptr=0x2aaadebc8690,
ino=1528908, gen=5480759143103581058)
After the kernel call trace appears, the fuse filesystem becomes very
unresponsive and unstable. Who tries to read the mounted directory,
randomly gets its process locked (with status waiting for I/O) and
hangs forever. We know that maybe glusterfs is not made for high
concurrency for both writing and reading, so we're working on getting
an alternate solution for this behavior (it's a disk cache, we're
transfering for something like memcached). But maybe this error helps
to find something unexpected going on... :)
Here's the client configuration:
volume vol01
type protocol/client
option transport-type tcp
option remote-host server01.domain
option remote-subvolume vol
end-volume
volume vol02
type protocol/client
option transport-type tcp
option remote-host server02.domain
option remote-subvolume vol
end-volume
volume vol03
type protocol/client
option transport-type tcp
option remote-host server03.domain
option remote-subvolume vol
end-volume
volume vol-mirror
type cluster/replicate
subvolumes vol01 vol02 vol03
end-volume
volume vol-iocache
type performance/io-cache
option cache-size 128MB
option cache-timeout 2
subvolumes vol-mirror
end-volume
volume vol-quickread
type performance/quick-read
option cache-timeout 1
option max-file-size 64kB
subvolumes vol-iocache
end-volume
volume vol-writebehind
type performance/write-behind
option cache-size 4MB
subvolumes vol-quickread
end-volume
volume vol-statprefetch
type performance/stat-prefetch
subvolumes vol-writebehind
end-volume
Thanks,
--
[]'s
Hugo
www.devin.com.br
