Does client logs contain log messages like "connection timed out"? It might be that statfs call is taking long time to complete on server and the client connection is timing out. Can you attach the client-logfile? Can you also try with larger transport-timeout in protocol/client?
regards,
2009/5/13 "José M. Martín" <jmartin@xxxxxxxxxxxxxx>
Hello,
I'm having problems when I set option export-statfs-size to "no" in
brick-posix. Clients can't connect to the servers.
If I comment that option, all works. Is there any bug or am I doing
something wrong?
I'm using glusterfs-2.0.0.
Thank you.
LOG SERVER
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]
ns-posix: allowed = "192.168.*", received addr = "192.168.3.56"
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:7502:mop_setvolume] server: accepted client from
192.168.3.56:1023
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]
ns-posix: allowed = "192.168.*", received addr = "192.168.3.56"
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:7502:mop_setvolume] server: accepted client from
192.168.3.56:1022
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]
brick: allowed = "192.168.*", received addr = "192.168.3.56"
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:7502:mop_setvolume] server: accepted client from
192.168.3.56:1019
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]
brick: allowed = "192.168.*", received addr = "192.168.3.56"
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:7502:mop_setvolume] server: accepted client from
192.168.3.56:1018
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]
stripe: allowed = "192.168.*", received addr = "192.168.3.56"
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:7502:mop_setvolume] server: accepted client from
192.168.3.56:1009
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]
stripe: allowed = "192.168.*", received addr = "192.168.3.56"
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:7502:mop_setvolume] server: accepted client from
192.168.3.56:1008
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D
[server-protocol.c:5235:server_statfs] brick: 3: STATFS '/ (0)'
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D
[server-protocol.c:5235:server_statfs] brick: 3: STATFS '/ (0)'
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D
[socket.c:90:__socket_rwv] server: EOF from peer 192.168.3.56:1023
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D
[socket.c:562:__socket_proto_state_machine] server: read (Transport
endpoint is not connected) in state 1 (192.168.3.56:1023)
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:8272:notify] server: 192.168.3.56:1023 disconnected
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]
server: transport 0x7f50bc000910 destroyed
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D
[socket.c:90:__socket_rwv] server: EOF from peer 192.168.3.56:1022
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D
[socket.c:562:__socket_proto_state_machine] server: read (Transport
endpoint is not connected) in state 1 (192.168.3.56:1022)
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:8272:notify] server: 192.168.3.56:1022 disconnected
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-helpers.c:723:server_connection_destroy] server: destroyed
connection of calypso56-14634-2009/05/13-10:42:50:315134-ns-remote
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]
server: transport 0x7f50bc000c30 destroyed
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 E
[socket.c:102:__socket_rwv] server: readv failed (Connection reset by peer)
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 E
[socket.c:562:__socket_proto_state_machine] server: read (Connection
reset by peer) in state 1 (192.168.3.56:1019)
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:8272:notify] server: 192.168.3.56:1019 disconnected
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]
server: transport 0x7f50bc000f50 destroyed
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 E
[socket.c:102:__socket_rwv] server: readv failed (Connection reset by peer)
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 E
[socket.c:562:__socket_proto_state_machine] server: read (Connection
reset by peer) in state 1 (192.168.3.56:1018)
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:8272:notify] server: 192.168.3.56:1018 disconnected
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-helpers.c:723:server_connection_destroy] server: destroyed
connection of calypso56-14634-2009/05/13-10:42:50:315134-brick2
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]
server: transport 0x7f50bc001270 destroyed
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D
[socket.c:90:__socket_rwv] server: EOF from peer 192.168.3.56:1009
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D
[socket.c:562:__socket_proto_state_machine] server: read (Transport
endpoint is not connected) in state 1 (192.168.3.56:1009)
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:8272:notify] server: 192.168.3.56:1009 disconnected
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]
server: transport 0x7f50bc001590 destroyed
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D
[socket.c:90:__socket_rwv] server: EOF from peer 192.168.3.56:1008
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D
[socket.c:562:__socket_proto_state_machine] server: read (Transport
endpoint is not connected) in state 1 (192.168.3.56:1008)
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-protocol.c:8272:notify] server: 192.168.3.56:1008 disconnected
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N
[server-helpers.c:723:server_connection_destroy] server: destroyed
connection of calypso56-14634-2009/05/13-10:42:50:315134-stripe2
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]
server: transport 0x7f50bc0018b0 destroyed
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N
[server-protocol.c:7502:mop_setvolume] server: accepted client from
192.168.3.56:1013
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N
[server-protocol.c:7502:mop_setvolume] server: accepted client from
192.168.3.56:1012
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N
[server-protocol.c:7502:mop_setvolume] server: accepted client from
192.168.3.56:1003
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N
[server-protocol.c:7502:mop_setvolume] server: accepted client from
192.168.3.56:1002
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 E
[socket.c:102:__socket_rwv] server: readv failed (Connection reset by peer)
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 E
[socket.c:562:__socket_proto_state_machine] server: read (Connection
reset by peer) in state 1 (192.168.3.56:1013)
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N
[server-protocol.c:8272:notify] server: 192.168.3.56:1013 disconnected
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 E
[socket.c:102:__socket_rwv] server: readv failed (Connection reset by peer)
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 E
[socket.c:562:__socket_proto_state_machine] server: read (Connection
reset by peer) in state 1 (192.168.3.56:1012)
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N
[server-protocol.c:8272:notify] server: 192.168.3.56:1012 disconnected
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N
[server-helpers.c:723:server_connection_destroy] server: destroyed
connection of calypso56-14634-2009/05/13-10:42:50:315134-brick5
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N
[server-protocol.c:8272:notify] server: 192.168.3.56:1003 disconnected
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N
[server-protocol.c:8272:notify] server: 192.168.3.56:1002 disconnected
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N
[server-helpers.c:723:server_connection_destroy] server: destroyed
connection of calypso56-14634-2009/05/13-10:42:50:315134-stripe5
SERVER CONFIG
### Export volume "brick" with the contents of
"/mnt/raid0/glusterfs/export" directory.
volume brick-posix
type storage/posix # POSIX FS translator
option directory /mnt/raid0/glusterfs/exports # Export this
directory
#When more than one directory is exported from a single mountpoint in
backend
#(ie, there is another volume exporting directory from this mount point), it
#is possible that 'df' on the client shows more diskspace than that of
it actually has.
#In that case, one can say 'no' for exporting statfs-size from redundant
exports.
option export-statfs-size no
end-volume
volume stripe-posix
type storage/posix
option directory /mnt/raid0/glusterfs/stripe
# option export-statfs-size no
# it was observed that when a very big file (file size > 100GB) is
getting unlinked,
# and the backend storage is of ext3 or some other journaling fs,
glusterfs mountpoint
# becomes un-usable till unlink() call is returned. This can be solved
by enabling this
# option in posix layer. Advised to use it when the file sizes are
serveral GBs.
option background-unlink yes
end-volume
volume ns-posix
type storage/posix
option directory /mnt/raid0/glusterfs/ns
# option export-statfs-size no
end-volume
# lock
volume brick-locks
type features/locks
subvolumes brick-posix
end-volume
volume stripe
type features/locks
subvolumes stripe-posix
end-volume
# io-cache
volume brick-cache
type performance/io-cache
option cache-size 1GB
subvolumes brick-locks
end-volume
# iothreads
volume brick
type performance/io-threads
option thread-count 16
subvolumes brick-cache
end-volume
### Add network serving capability to above brick.
volume server
type protocol/server
option transport-type tcp
option transport.socket.listen-port 6993 # Default is 6996
subvolumes brick stripe ns-posix
# NOTE: Access to any volume through protocol/server is denied by
# default. You need to explicitly grant access through # "auth"
# option.
option auth.addr.brick.allow 192.168.* # Allow access to "brick" volume
option auth.addr.stripe.allow 192.168.*
option auth.addr.ns-posix.allow 192.168.*
end-volume
CLIENT CONFIG
volume brick1
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume brick # name of the remote volume
option remote-host nix01
option remote-port 6993
end-volume
volume brick2
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume brick # name of the remote volume
option remote-host nix02
option remote-port 6993
end-volume
volume brick3
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume brick # name of the remote volume
option remote-host nix03
option remote-port 6993
end-volume
volume brick4
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume brick # name of the remote volume
option remote-host nix04
option remote-port 6993
end-volume
volume brick5
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume brick # name of the remote volume
option remote-host nix05
option remote-port 6993
end-volume
############################################################################
# DHT
#
volume dht
type cluster/distribute
# option lookup-unhashed yes
# option min-free-disk 10%
subvolumes brick1 brick2 brick3 brick4 brick5
end-volume
############################################################################
# CLIENTE STRIPE
#
volume stripe1
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume stripe # name of the remote volume
option remote-host nix01
option remote-port 6993
end-volume
volume stripe2
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume stripe # name of the remote volume
option remote-host nix02
option remote-port 6993
end-volume
volume stripe3
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume stripe # name of the remote volume
option remote-host nix03
option remote-port 6993
end-volume
volume stripe4
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume stripe # name of the remote volume
option remote-host nix04
option remote-port 6993
end-volume
volume stripe5
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume stripe # name of the remote volume
option remote-host nix05
option remote-port 6993
end-volume
############################################################################
# STRIPE
#
volume stripe
type cluster/stripe
option block-size 1MB
subvolumes stripe1 stripe2 stripe3 stripe4 stripe5
end-volume
############################################################################
# NAMESPACE
#
volume ns-remote
type protocol/client
option transport-type tcp # for TCP/IP transport
option remote-subvolume ns-posix # name of the remote volume
option remote-host nix02
option remote-port 6993
end-volume
############################################################################
# UNIFY SWITCH
#
volume bricks
type cluster/unify
option scheduler switch
option namespace ns-remote
option self-heal off
option switch.case *.big*:stripe;*:dht
subvolumes dht stripe
end-volume
volume writebehind
type performance/write-behind
option aggregate-size 128KB # default is 0bytes
option cache-size 1MB # default is equal to aggregate-size
option flush-behind on # default is 'off'
subvolumes bricks
end-volume
volume readahead
type performance/read-ahead
option page-size 1024kB # 256KB is the default option
option page-count 32 # 2 is default option
option force-atime-update off # default is off
subvolumes writebehind
end-volume
volume iothreads
type performance/io-threads
option thread-count 8 # deault is 1
subvolumes readahead
end-volume
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
--
Raghavendra G
