Brian Hirt wrote:
Hello,
I'm running some tests with with GlusterFS and so far I like what I'm
seeing. I've got a test 4 node system set up with AFR-Unify. node1
and node3 are replicated and node2 and node4 are replicated. These are
then unified together into a single brick. afr and unify are done on
the client side. All of the servers are running ubuntu + glusterfs
1.3.12 with an underlaying ext3 filesystem.
Durning one of my tests i took down a server during an update/addition
of a few thousand files. After the update was complete, i brought up
the downed node. I was able to see all the new files after i did a
directory listing on the client, but they all had a size of 0 and the
updated files still had the old contents. When I opened these files on
the client, the correct contents were returned and the once down node
was then corrected for that file.
From searching through the email archives, this seems like the intended
way it supposed to work. However, in the state that the filesystem is
in now, my redundancy is lost for those changes until i open every file
and directory on the client. In my configuration I intend to have many
million files. Am I supposed to open every single one of them after a
node goes down to get the replication back in sync? There will often be
times where servers are brought down for routine maintenance for 10-15
minutes at a time and during that time only a few hundred files might
change. What is the proper procedure for resynchronizing? How are
other people handling this? I've seen a few comments about fsck in the
mail archive referencing a path that doesn't exist in my GlusterFS
distribution (possibly it's the 1.4 branch)
My understanding is that the accepted way to force synchronization
between servers is to open and read a small amount (a single byte will
do, healing is triggered on an actual open system call) of every file,
or the files you want to ensure are synced. Something like the following:
find /mnt/glusterfs -type f -exec head -c 1 {} \; >/dev/null
If one system has been down for a known amount of time, you can use that
to your advantage:
find /mnt/glusterfs -type f -mtime -1 exec head -c 1 {} \; >/dev/null
will only read from files which have a modified time of less than 24
hours from now.
You could probably speed up the find by passing multiple file names to
head at a time using xargs, to reduce program (head) initialization
overhead:
find /mnt/glusterfs -type f -mtime -1 | xargs -n100 head -c1 >/dev/null
which will run 100 files through head at a time.
Looks like this and more is covered in
http://www.gluster.org/docs/index.php/Understanding_AFR_Translator,
which is probably more complete than anything I've mentioned.
Also the log file is very verbose about the downed server. There are
lots of messages like:
2008-12-09 11:18:08 E [tcp-client.c:190:tcp_connect] brick2:
non-blocking connect() returned: 111 (Connection refused)
2008-12-09 11:18:08 W [client-protocol.c:332:client_protocol_xfer]
brick2: not connected at the moment to submit frame type(1) op(34)
2008-12-09 11:18:08 E [client-protocol.c:4430:client_lookup_cbk] brick2:
no proper reply from server, returning ENOTCONN
2008-12-09 11:18:08 E [tcp-client.c:190:tcp_connect] brick2:
non-blocking connect() returned: 111 (Connection refused)
2008-12-09 11:18:08 W [client-protocol.c:332:client_protocol_xfer]
brick2: not connected at the moment to submit frame type(1) op(9)
2008-12-09 11:18:08 E [client-protocol.c:2787:client_chmod_cbk] brick2:
no proper reply from server, returning ENOTCONN
In some of my tests I'm seeing several hundred a second logged. Is
there some way to make this a bit less verbose?
I'm sorry if these are FAQ, but I've so far been unable to find anything
on the wiki or mailing lists.
Thanks in advance for you help and this great project.
IMO better whiney until taken care of than quiet and missed...
--
-Kevan Benson
-A-1 Networks
