Hello,
I'm running some tests with with GlusterFS and so far I like what I'm
seeing. I've got a test 4 node system set up with AFR-Unify. node1
and node3 are replicated and node2 and node4 are replicated. These
are then unified together into a single brick. afr and unify are
done on the client side. All of the servers are running ubuntu +
glusterfs 1.3.12 with an underlaying ext3 filesystem.
Durning one of my tests i took down a server during an update/addition
of a few thousand files. After the update was complete, i brought up
the downed node. I was able to see all the new files after i did a
directory listing on the client, but they all had a size of 0 and the
updated files still had the old contents. When I opened these files
on the client, the correct contents were returned and the once down
node was then corrected for that file.
From searching through the email archives, this seems like the
intended way it supposed to work. However, in the state that the
filesystem is in now, my redundancy is lost for those changes until i
open every file and directory on the client. In my configuration I
intend to have many million files. Am I supposed to open every single
one of them after a node goes down to get the replication back in
sync? There will often be times where servers are brought down for
routine maintenance for 10-15 minutes at a time and during that time
only a few hundred files might change. What is the proper
procedure for resynchronizing? How are other people handling this?
I've seen a few comments about fsck in the mail archive referencing a
path that doesn't exist in my GlusterFS distribution (possibly it's
the 1.4 branch)
Also the log file is very verbose about the downed server. There are
lots of messages like:
2008-12-09 11:18:08 E [tcp-client.c:190:tcp_connect] brick2: non-
blocking connect() returned: 111 (Connection refused)
2008-12-09 11:18:08 W [client-protocol.c:332:client_protocol_xfer]
brick2: not connected at the moment to submit frame type(1) op(34)
2008-12-09 11:18:08 E [client-protocol.c:4430:client_lookup_cbk]
brick2: no proper reply from server, returning ENOTCONN
2008-12-09 11:18:08 E [tcp-client.c:190:tcp_connect] brick2: non-
blocking connect() returned: 111 (Connection refused)
2008-12-09 11:18:08 W [client-protocol.c:332:client_protocol_xfer]
brick2: not connected at the moment to submit frame type(1) op(9)
2008-12-09 11:18:08 E [client-protocol.c:2787:client_chmod_cbk]
brick2: no proper reply from server, returning ENOTCONN
In some of my tests I'm seeing several hundred a second logged. Is
there some way to make this a bit less verbose?
I'm sorry if these are FAQ, but I've so far been unable to find
anything on the wiki or mailing lists.
Thanks in advance for you help and this great project.
Brian Hirt
