Re: Mounting with file servers is failing very frequently (in every day)

[Joseph Job <joseph@xxxxxxxxxxxxxxx>]

Hi Raghavendra,

I would to get some tips and tricks regarding in which direction I 
need to explore.
Let me brief by setup.

I have two web servers as glusterfs client and both of them are 
accessing the Glusterfs server for file storage.

volume afrbricks
 type cluster/afr
 subvolumes  client_214  client_213
 option replicate *:2
 option self-heal on
end-volume

The glusterfs servers and clients are identical in hardware side.

The servers are using Ultra SCSI HDD and the storage capacity is around 540GB

What are the things which I need to consider for performance improvement.

See the time difference in accessing the files...

root@w3-cok /storage/rvdoc# time ls -l | wc -l (From gluster client)
96514

real    11m51.644s
user    0m1.994s
sys     0m3.858s


root@fs4-cok /storage/rvdoc# time ls -l | wc -l (From gluster server)
96514

real    0m30.812s
user    0m1.542s
sys     0m2.442s


If you want any more info from my side, please inform...

Thanks,
Joseph




At 08:38 PM 10/30/2007, Joseph Job wrote:
> Hi Raghavendra,
>
> Thanks for your help, I would like to know the default value, how 
> much seconds it is ?
>
> Thanks,
> Joseph
>
> At 05:07 PM 10/30/2007, Raghavendra G wrote:
>
> > Hi Joseph,
> > transport-timeout in clients is very less in your configuration (4 
> > seconds). Use a higher timeout or just comment out the option to 
> > use the default one, which should be sufficient.
> >
> > regards,
> >
> > On Oct 30, 2007 3:16 PM, Joseph Job 
> > <<mailto:joseph@xxxxxxxxxxxxxxx>joseph@xxxxxxxxxxxxxxx> wrote:
> > My Setup details..
> > Operating System : Trustix Secure Linux release 3.0.5 (Mirch Masala)
> > Kernel Version : 2.6.19.7-1
> > Gluster version :
> > glusterfs-server-1.3.6-1
> > glusterfs-client-1.3.6-1
> > glusterfs-common-1.3.6-1
> > glusterfs-devel-1.3.6-1
> > Working Mode.
> > Two web servers are there and both web servers are using Gluster file
> > server in TCP network for file access.
> > see my server side configuration...
> >
> >
> > Server 1
> > ## Define the stroage
> > volume fs3-storage
> >  type storage/posix                   # POSIX FS translator
> >  option directory /storage            # Export this directory
> > end-volume
> > volume iothreads                      #iothreads can give 
> > performance a boost
> >   type performance/io-threads
> >   option thread-count 16
> >   subvolumes fs3-storage
> > end-volume
> > ## Add network serving capability to above brick.
> > volume server
> >  type protocol/server
> >  option transport-type tcp/server     # For TCP/IP transport
> >  option listen-port 6996              # Default is 6996
> >  option client-volume-filename /var/log/glusterfs/client.vol
> >  subvolumes iothreads
> >  option auth.ip.iothreads.allow * # Allow access to "brick" volume
> > end-volume
> > Server 2
> > ## Define the stroage
> > volume fs4-storage
> >  type storage/posix                   # POSIX FS translator
> >  option directory /storage            # Export this directory
> > end-volume
> > volume iothreads                      #iothreads can give 
> > performance a boost
> >   type performance/io-threads
> >   option thread-count 16
> >   subvolumes fs4-storage
> > end-volume
> > ## Add network serving capability to above brick.
> > volume server
> >  type protocol/server
> >  option transport-type tcp/server     # For TCP/IP transport
> >  option listen-port 6996              # Default is 6996
> >  option client-volume-filename /var/log/glusterfs/client.vol
> >  subvolumes iothreads
> >  option auth.ip.iothreads.allow * # Allow access to "brick" volume
> > end-volume
> >
> > Client side configuration...
> > Client 1
> > ### Add client feature and attach to remote subvolume
> > volume client_214
> >  type protocol/client
> >  option transport-type tcp/client     # for TCP/IP transport
> >  option remote-host <http://10.10.0.214>10.10.0.214     # IP 
> > address of the remote brick
> >  option remote-port 6996              # default server port is 6996
> >  option remote-subvolume iothreads        # name of the remote volume
> >  option transport-timeout 4
> > end-volume
> > ### Add client feature and attach to remote subvolume
> > volume client_213
> >  type protocol/client
> >  option transport-type tcp/client     # for TCP/IP transport
> >  option remote-host <http://10.10.0.213>10.10.0.213     # IP 
> > address of the remote brick
> >  option remote-port 6996              # default server port is 6996
> >  option remote-subvolume iothreads        # name of the remote volume
> >  option transport-timeout 4
> > end-volume
> > volume afrbricks
> >  type cluster/afr
> >  subvolumes  client_214  client_213
> >  option replicate *:2
> >  option self-heal on
> > end-volume
> > volume iothreads    #iothreads can give performance a boost
> >   type performance/io-threads
> >   option thread-count 8
> >   subvolumes afrbricks
> > end-volume
> > ##########################
> > Client 2
> > Client 1
> > ### Add client feature and attach to remote subvolume
> > volume client_214
> >  type protocol/client
> >  option transport-type tcp/client     # for TCP/IP transport
> >  option remote-host <http://10.10.0.214>10.10.0.214     # IP 
> > address of the remote brick
> >  option remote-port 6996              # default server port is 6996
> >  option remote-subvolume iothreads        # name of the remote volume
> >  option transport-timeout 4
> > end-volume
> > ### Add client feature and attach to remote subvolume
> > volume client_213
> >  type protocol/client
> >  option transport-type tcp/client     # for TCP/IP transport
> >  option remote-host <http://10.10.0.213>10.10.0.213     # IP 
> > address of the remote brick
> >  option remote-port 6996              # default server port is 6996
> >  option remote-subvolume iothreads        # name of the remote volume
> >  option transport-timeout 4
> > end-volume
> > volume afrbricks
> >  type cluster/afr
> >  subvolumes  client_214  client_213
> >  option replicate *:2
> >  option self-heal on
> > end-volume
> > volume iothreads    #iothreads can give performance a boost
> >   type performance/io-threads
> >   option thread-count 8
> >   subvolumes afrbricks
> > end-volume
> > ##########################
> >
> > I am mounting the server to client with glusterfs -f
> > /etc/glusterfs/glusterfs- client.vol /storage/
> > I could able to mount, files are getting replicated to both file
> > servers. But the problem, the mounting is braking very frequently...
> > I am getting the error in glusterfs.log
> > 2007-10-30 00:46:10 C [ tcp.c:81:tcp_disconnect] client_213:
> > connection disconnected
> > 2007-10-30 00:46:18 C [client-protocol.c:218:call_bail] client_214:
> > bailing transport
> > 2007-10-30 00:46:18 C [client-protocol.c:218:call_bail] client_213:
> > bailing transport
> > 2007-10-30 00:46:18 C [tcp.c:81:tcp_disconnect] client_214:
> > connection disconnected
> > 2007-10-30 00:46:18 C [tcp.c:81:tcp_disconnect] client_213:
> > connection disconnected
> > But the physical connection is still there...I can ping from client
> > to servers...
> > The server is using gigabit networking bonding.
> > I am using kernel with Fuse supported
> > root@w3-cok ~# lsmod
> > Module                  Size  Used by
> > fuse                   39444  2
> > ipv6                  221344  24
> > tg3                   105860  0
> > bonding                79224  0
> > jfs                   163564  2
> > usbhid                 35936  0
> > ohci_hcd               18564  0
> > usbcore               112772  3 usbhid,ohci_hcd
> > parport_pc             21956  0
> > parport                20032  1 parport_pc
> > shpchp                 32416  0
> > serverworks             8840  0 [permanent]
> > cciss                  54020  8
> > dm_mod                 49432  0
> > sd_mod                 17024  0
> > piix                    9604  0 [permanent]
> > ide_disk               14336  0
> > ide_generic             2048  0 [permanent]
> > ide_core              106444  4 serverworks,piix,ide_disk,ide_generic
> >
> > Also see the glusterfsd.log in server side
> > 2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
> > transport_writev failed
> > 2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
> > error: Transport endpoint is not connected
> > 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:03:43 E [protocol.c:253:gf_block_unserialize_transport]
> > server: EOF from peer ( <http://10.10.0.203:1018>10.10.0.203:1018)
> > 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
> > transport_writev failed
> > 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
> > error: Transport endpoint is not connected
> > 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
> > transport_writev failed
> > 2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
> > error: Transport endpoint is not connected
> > 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:03:43 E [ server-protocol.c:197:generic_reply] server:
> > transport_writev failed
> > 2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
> > error: Transport endpoint is not connected
> > 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
> > transport_writev failed
> > 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
> > error: Transport endpoint is not connected
> > 2007-10-30 00:48:52 E [server-protocol.c:197:generic_reply] server:
> > transport_writev failed
> > 2007-10-30 00:48:52 E [tcp.c:118:tcp_except] server: shutdown () -
> > error: Transport endpoint is not connected
> > 2007-10-30 00:48:52 C [ tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:48:52 E [server-protocol.c:197:generic_reply] server:
> > transport_writev failed
> > 2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:48:52 E [tcp.c:118:tcp_except] server: shutdown () -
> > error: Transport endpoint is not connected
> > 2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:48:52 E [ server-protocol.c:197:generic_reply] server:
> > transport_writev failed
> > 2007-10-30 00:48:52 E [tcp.c:118:tcp_except] server: shutdown () -
> > error: Transport endpoint is not connected
> > 2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> > 2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection 
> > disconnected
> >
> >
> > JOSEPH JOB
> > Spectrum Softtech Solutions(P)Ltd.
> > MahaKavi G Road,
> > Karikkamuri Cross Road
> > Kochi-682011
> > 0484-4082000
> > <mailto:joseph@xxxxxxxxxxxxxxx>joseph@xxxxxxxxxxxxxxx
> > Visit at <http://www.spectrum.net.in>www.spectrum.net.in
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > <mailto:Gluster-devel@xxxxxxxxxx>Gluster-devel@xxxxxxxxxx
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> >
> >
> >
> > --
> > Raghavendra G
> >
> > A centipede was happy quite, until a toad in fun,
> > Said, "Prey, which leg comes after which?",
> > This raised his doubts to such a pitch,
> > He fell flat into the ditch,
> > Not knowing how to run.
> > -Anonymous
>
> JOSEPH JOB
> Spectrum Softtech Solutions(P)Ltd.
> MahaKavi G Road,
> Karikkamuri Cross Road
> Kochi-682011
> 0484-4082000
> joseph@xxxxxxxxxxxxxxx
> Visit at www.spectrum.net.in

JOSEPH JOB
Spectrum Softtech Solutions(P)Ltd.
MahaKavi G Road,
Karikkamuri Cross Road
Kochi-682011
0484-4082000
joseph@xxxxxxxxxxxxxxx
Visit at www.spectrum.net.in