Daniel, There were fixes that went into HA recently. Can you check if the bug is still there? Krishna On Wed, Jan 14, 2009 at 11:22 PM, Daniel Maher <dma+gluster at witbe.net> wrote: > Hi all, > > In testing the HA translator under 2.0.0rc1, i've managed to create a > simple and reproducible scenario in which Gluster fails to maintain > communication between the client and the server(s). > > Server01 and Server02 are AFR'ing each other, with Client01 connected > via the HA translator. As a simple test, i launch a script that echoes > an increasing counter to a text file in the Gluster mount on Client01. > Client01 is communicating with Server01 in this instance. > > I cleanly stop glusterfsd on Server01, and after a momentary hiccup > (noted in the log excerpt below), things continue to function as > expected - Client01 commences communication with Server02. So far so good. > > 2009-01-15 15:54:19 E [socket.c:708:socket_connect_finish] export01: > connection failed (Connection refused) > > I re-start glusterfsd on Server01, then, i cleanly stop glusterfsd on > Server02 (which, of course, Client01 is now communicating with). > Client01 freaks out (see log excerpt below), does /not/ attempt to > contact Server01 again, and leaves me with the dreaded "transport > endpoint not connected" situation. > > 2009-01-15 16:06:02 E [ha-helpers.c:266:_ha_next_active_child_for_ctx] > export-ha: none of the children are connected other than export02 > 2009-01-15 16:06:02 E [ha.c:2715:ha_fstat_cbk] export-ha: no active > subvolume > 2009-01-15 16:06:02 E [fuse-bridge.c:533:fuse_attr_cbk] glusterfs-fuse: > 2932: FSTAT() /counter.txt => -1 (Transport endpoint is not connected) > > Client01 sometimes recovers from this, and sometimes it does not. When > it does not recover from this situation, the only solution is manual > intervention (unmount / remount). That's not the worst of it, though : > when it /does/ recover, re-starting glusterfsd on Server02 (!) causes > even more of the errors (see below), and /always/ results in a total > failure on Client01 within a second or two (transport endpoint not > connected). Client01 never recovers from this. > > 2009-01-15 19:04:56 E [ha-helpers.c:266:_ha_next_active_child_for_ctx] > export-ha: none of the children are connected other than export01 > 2009-01-15 19:04:56 E [ha.c:2515:ha_flush_cbk] export-ha: no active > subvolume > 2009-01-15 19:04:56 E [fuse-bridge.c:911:fuse_err_cbk] glusterfs-fuse: > 3058: FLUSH() ERR => -1 (Transport endpoint is not connected) > > > I strongly suspect this is not the expected behaviour of the High > Availability translator. :) > > > Servers are running FC9 i386, Client is FC10 i386. > > # glusterfs --version > glusterfs 2.0.0rc1 built on Jan 14 2009 13:19:06 > Repository revision: glusterfs--mainline--3.0--patch-844 > > # rpm -qa | grep fuse > fuse-2.7.3glfs10-1.i386 > fuse-devel-2.7.3glfs10-1.i386 > fuse-libs-2.7.3glfs10-1.i386 > > > Server config : > > # cat /etc/glusterfs/glusterfs-server.vol > # dataspace > volume test-ds > type storage/posix > option directory /opt/datadir > end-volume > > # posix locks for test-ds > volume test-ds-locks > type features/locks > option mandatory-locks on > subvolumes test-ds > end-volume > > # dataspace of test-ds on Server01 > volume test-01-ds > type protocol/client > option transport-type tcp/client > option remote-host 192.168.0.183 > option remote-subvolume test-ds-locks > option transport-timeout 10 > end-volume > > # automatic file replication translator for test dataspace > volume test-ds-afr > type cluster/afr > subvolumes test-ds-locks test-01-ds > end-volume > > # the actual export > volume export > type performance/io-threads > option thread-count 8 > subvolumes test-ds-afr > end-volume > > # server declaration > volume server > type protocol/server > option transport-type tcp/server > subvolumes export > option auth.addr.export.allow > 192.168.0.73,192.168.0.183,192.168.0.166,127.0.0.1 > option auth.addr.test-ds-locks.allow > 192.168.0.73,192.168.0.183,192.168.0.166,127.0.0.1 > end-volume > > > > client config : > # cat /etc/glusterfs/glusterfs-client.vol > > # export on Server01 > volume export01 > type protocol/client > option transport-type tcp/client > option remote-host 192.168.0.183 > option remote-subvolume export # exported volume > end-volume > > # export on Server02 > volume export02 > type protocol/client > option transport-type tcp/client > option remote-host 192.168.0.166 > option remote-subvolume export # exported volume > end-volume > > # exports clustered via HA > volume export-ha > type cluster/ha > subvolumes export01 export02 > end-volume > > > > -- > Daniel Maher <dma+gluster AT witbe DOT net> > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >
