Hi All,
We're running a 10 node cluster in raid1 configuration with each node
exporting 4 disks and are seeing several issues right now. We recently
upgraded from 2.0.7 to 3.03.
First, we are seeing major performance degradation but without a clear
bottleneck of either network i/o, cpu, or disk i/o. We see a number of
processes trying to access the mount stuck in uninterruptible sleep. We
also see the directory structure being replicated to the new nodes whenever
a stat call is made to an existing directory; this seems to be the source of
the slowness, but again we don't see network/cpu/disk as a bottleneck. We
have 8 clients accessing the cluster with a write-heavy load.
Second, we are receiving many "Stale NFS file handle" errors in the logs.
These errors are essentially causing reads and writes to fail.
[2010-04-12 22:36:56] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse:
2880: LOOKUP() / => -1 (Stale NFS file handle)
[2010-04-12 22:36:56] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse:
2881: LOOKUP() / => -1 (Stale NFS file handle)
[2010-04-12 22:36:56] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse:
2882: LOOKUP() / => -1 (Stale NFS file handle)
[2010-04-12 22:36:56] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse:
2883: LOOKUP() / => -1 (Stale NFS file handle)
[2010-04-12 22:36:56] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse:
2886: LOOKUP() / => -1 (Stale NFS file handle)
Does anyone have any thoughts on how we can fix these problems? The new
stale file handle errors have started since upgrading to 3.0.3, but the
performance degradation was happening on 2.0.
Any help is greatly appreciated.
Our server config is as follows:
#####################################
### GlusterFS Server Volume File ##
#####################################
volume posix1
type storage/posix
option directory /srv/disk1
end-volume
volume posix2
type storage/posix
option directory /srv/disk2
end-volume
volume posix3
type storage/posix
option directory /srv/disk3
end-volume
volume posix4
type storage/posix
option directory /srv/disk4
end-volume
volume locks1
type features/locks
subvolumes posix1
end-volume
volume locks2
type features/locks
subvolumes posix2
end-volume
volume locks3
type features/locks
subvolumes posix3
end-volume
volume locks4
type features/locks
subvolumes posix4
end-volume
volume threads1
type performance/io-threads
option thread-count 64
subvolumes locks1
end-volume
volume threads2
type performance/io-threads
option thread-count 64
subvolumes locks2
end-volume
volume threads3
type performance/io-threads
option thread-count 64
subvolumes locks3
end-volume
volume threads4
type performance/io-threads
option thread-count 64
subvolumes locks4
end-volume
volume server
type protocol/server
option transport-type tcp
option transport.socket.nodelay on
subvolumes threads1 threads2 threads3 threads4
option auth.addr.threads1.allow *
option auth.addr.threads2.allow *
option auth.addr.threads3.allow *
option auth.addr.threads4.allow *
end-volume
Our client config is as follows:
#####################################################################
## Remote Servers
## remote 4 (192.168.150.184)
volume remote4-1
type protocol/client
option transport-type tcp
option remote-host 192.168.150.184
option remote-subvolume threads1
end-volume
volume remote4-2
type protocol/client
option transport-type tcp
option remote-host 192.168.150.184
option remote-subvolume threads2
end-volume
volume remote4-3
type protocol/client
option transport-type tcp
option remote-host 192.168.150.184
option remote-subvolume threads3
end-volume
volume remote4-4
type protocol/client
option transport-type tcp
option remote-host 192.168.150.184
option remote-subvolume threads4
end-volume
## end remote 4
## remote 5 (192.168.150.185)
volume remote5-1
type protocol/client
option transport-type tcp
option remote-host 192.168.150.185
option remote-subvolume threads1
end-volume
volume remote5-2
type protocol/client
option transport-type tcp
option remote-host 192.168.150.185
option remote-subvolume threads2
end-volume
volume remote5-3
type protocol/client
option transport-type tcp
option remote-host 192.168.150.185
option remote-subvolume threads3
end-volume
volume remote5-4
type protocol/client
option transport-type tcp
option remote-host 192.168.150.185
option remote-subvolume threads4
end-volume
## end remote 5
## remote 6 (192.168.150.186)
volume remote6-1
type protocol/client
option transport-type tcp
option remote-host 192.168.150.186
option remote-subvolume threads1
end-volume
volume remote6-2
type protocol/client
option transport-type tcp
option remote-host 192.168.150.186
option remote-subvolume threads2
end-volume
volume remote6-3
type protocol/client
option transport-type tcp
option remote-host 192.168.150.186
option remote-subvolume threads3
end-volume
volume remote6-4
type protocol/client
option transport-type tcp
option remote-host 192.168.150.186
option remote-subvolume threads4
end-volume
## end remote 6
## remote 7 (192.168.150.187)
volume remote7-1
type protocol/client
option transport-type tcp
option remote-host 192.168.150.187
option remote-subvolume threads1
end-volume
volume remote7-2
type protocol/client
option transport-type tcp
option remote-host 192.168.150.187
option remote-subvolume threads2
end-volume
volume remote7-3
type protocol/client
option transport-type tcp
option remote-host 192.168.150.187
option remote-subvolume threads3
end-volume
volume remote7-4
type protocol/client
option transport-type tcp
option remote-host 192.168.150.187
option remote-subvolume threads4
end-volume
## end remote 7
## remote 8 (192.168.150.188)
volume remote8-1
type protocol/client
option transport-type tcp
option remote-host 192.168.150.188
option remote-subvolume threads1
end-volume
volume remote8-2
type protocol/client
option transport-type tcp
option remote-host 192.168.150.188
option remote-subvolume threads2
end-volume
volume remote8-3
type protocol/client
option transport-type tcp
option remote-host 192.168.150.188
option remote-subvolume threads3
end-volume
volume remote8-4
type protocol/client
option transport-type tcp
option remote-host 192.168.150.188
option remote-subvolume threads4
end-volume
## end remote 8
## remote 9 (192.168.150.189)
volume remote9-1
type protocol/client
option transport-type tcp
option remote-host 192.168.150.189
option remote-subvolume threads1
end-volume
volume remote9-2
type protocol/client
option transport-type tcp
option remote-host 192.168.150.189
option remote-subvolume threads2
end-volume
volume remote9-3
type protocol/client
option transport-type tcp
option remote-host 192.168.150.189
option remote-subvolume threads3
end-volume
volume remote9-4
type protocol/client
option transport-type tcp
option remote-host 192.168.150.189
option remote-subvolume threads4
end-volume
## end remote 9
## remote 10 (192.168.150.190)
volume remote10-1
type protocol/client
option transport-type tcp
option remote-host 192.168.150.190
option remote-subvolume threads1
end-volume
volume remote10-2
type protocol/client
option transport-type tcp
option remote-host 192.168.150.190
option remote-subvolume threads2
end-volume
volume remote10-3
type protocol/client
option transport-type tcp
option remote-host 192.168.150.190
option remote-subvolume threads3
end-volume
volume remote10-4
type protocol/client
option transport-type tcp
option remote-host 192.168.150.190
option remote-subvolume threads4
end-volume
## end remote 10
## remote 11 (192.168.150.191)
volume remote11-1
type protocol/client
option transport-type tcp
option remote-host 192.168.150.191
option remote-subvolume threads1
end-volume
volume remote11-2
type protocol/client
option transport-type tcp
option remote-host 192.168.150.191
option remote-subvolume threads2
end-volume
volume remote11-3
type protocol/client
option transport-type tcp
option remote-host 192.168.150.191
option remote-subvolume threads3
end-volume
volume remote11-4
type protocol/client
option transport-type tcp
option remote-host 192.168.150.191
option remote-subvolume threads4
end-volume
## end remote 11
## remote 12 (192.168.150.192)
volume remote12-1
type protocol/client
option transport-type tcp
option remote-host 192.168.150.192
option remote-subvolume threads1
end-volume
volume remote12-2
type protocol/client
option transport-type tcp
option remote-host 192.168.150.192
option remote-subvolume threads2
end-volume
volume remote12-3
type protocol/client
option transport-type tcp
option remote-host 192.168.150.192
option remote-subvolume threads3
end-volume
volume remote12-4
type protocol/client
option transport-type tcp
option remote-host 192.168.150.192
option remote-subvolume threads4
end-volume
## end remote 12
## remote 13 (192.168.150.193)
volume remote13-1
type protocol/client
option transport-type tcp
option remote-host 192.168.150.193
option remote-subvolume threads1
end-volume
volume remote13-2
type protocol/client
option transport-type tcp
option remote-host 192.168.150.193
option remote-subvolume threads2
end-volume
volume remote13-3
type protocol/client
option transport-type tcp
option remote-host 192.168.150.193
option remote-subvolume threads3
end-volume
volume remote13-4
type protocol/client
option transport-type tcp
option remote-host 192.168.150.193
option remote-subvolume threads4
end-volume
## end remote 13
#####################################################################
## Replication pairs
## remote 4+5
volume pair4-5-1
type cluster/replicate
subvolumes remote4-1 remote5-1
end-volume
volume pair4-5-2
type cluster/replicate
subvolumes remote4-2 remote5-2
end-volume
volume pair4-5-3
type cluster/replicate
subvolumes remote4-3 remote5-3
end-volume
volume pair4-5-4
type cluster/replicate
subvolumes remote4-4 remote5-4
end-volume
## End remote 4+5
## remote 6+7
volume pair6-7-1
type cluster/replicate
subvolumes remote6-1 remote7-1
end-volume
volume pair6-7-2
type cluster/replicate
subvolumes remote6-2 remote7-2
end-volume
volume pair6-7-3
type cluster/replicate
subvolumes remote6-3 remote7-3
end-volume
volume pair6-7-4
type cluster/replicate
subvolumes remote6-4 remote7-4
end-volume
## End remote 6+7
## remote 8+9
volume pair8-9-1
type cluster/replicate
subvolumes remote8-1 remote9-1
end-volume
volume pair8-9-2
type cluster/replicate
subvolumes remote8-2 remote9-2
end-volume
volume pair8-9-3
type cluster/replicate
subvolumes remote8-3 remote9-3
end-volume
volume pair8-9-4
type cluster/replicate
subvolumes remote8-4 remote9-4
end-volume
## End remote 8+9
## remote 10+11
volume pair10-11-1
type cluster/replicate
subvolumes remote10-1 remote11-1
end-volume
volume pair10-11-2
type cluster/replicate
subvolumes remote10-2 remote11-2
end-volume
volume pair10-11-3
type cluster/replicate
subvolumes remote10-3 remote11-3
end-volume
volume pair10-11-4
type cluster/replicate
subvolumes remote10-4 remote11-4
end-volume
## End remote 10+11
## remote 12+13
volume pair12-13-1
type cluster/replicate
subvolumes remote12-1 remote13-1
end-volume
volume pair12-13-2
type cluster/replicate
subvolumes remote12-2 remote13-2
end-volume
volume pair12-13-3
type cluster/replicate
subvolumes remote12-3 remote13-3
end-volume
volume pair12-13-4
type cluster/replicate
subvolumes remote12-4 remote13-4
end-volume
## End remote 12+13
#####################################################################
## Unified filesystem
volume distribute
type cluster/distribute
option lookup-unhashed yes
option min-free-disk 2%
subvolumes pair4-5-1 pair4-5-2 pair4-5-3 pair4-5-4 pair6-7-1 pair6-7-2
pair6-7-3 pair6-7-4 pair8-9-1 pair8-9-2 pair8-9-3 pair8-9-4 pair10-11-1
pair10-11-2 pair10-11-3 pair10-11-4 pair12-13-1 pair12-13-2 pair12-13-3
pair12-13-4
end-volume
volume writebehind
type performance/write-behind
option cache-size 3MB
option flush-behind on
subvolumes distribute
end-volume
volume io-cache
type performance/io-cache
option cache-size 128MB
subvolumes writebehind
end-volume
volume client
type performance/io-threads
option thread-count 64
subvolumes io-cache
end-volume
All nodes are running Ubuntu 9.04.
root at userdata12:/etc/glusterfs/glusterfs-upgrade# uname -a
Linux userdata12 2.6.28-18-server #59-Ubuntu SMP Thu Jan 28 02:25:03 UTC
2010 x86_64 GNU/Linux
Thanks,
James
