How to re-sync

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



That seems correct with 1 change, not only do I get the old file in step 5, that old file overwrites the newer file on the node that did not go down.

 > 1) What versions are you using ?
glusterfs 3.0.2 built on Feb  7 2010 00:15:44
Repository revision: v3.0.2

 > 2) Can you share your volume files ? Are they generated using volgen ?
I did generate them via volgen, but then modified them because I have 3 shares, but only to rename things.
(vol files at end of e-mail)

 > 3) Did you notice any patterns for the files where the wrong copy was picked ? like
 > were they open when the node was brought down ?
I was not monitoring this.

 > 4) Any other way to reproduce the problem ?
See my nfs issue below, although I don't think they are related.

 > 5) Any other patterns you observed when you see the problem ?
See my nfs issue below, although I don't think they are related.

 > 6) Would you have listings of problem file(s) from the replica nodes ?
No.

Also I did something today that works on nfs but does not work in gluster.
I have a share mounted on /cs_data.
I have directories in that share /cs_data/web and /cs_data/home
I move the /cs_data/web into /cs_data/home (so I get: /cs_data/home/web) then symlink /cs_data/web to /cs_data/home/web, like this:
cd /cs_data;
mv web home;
ln -s home/web

On all the clients /cs_data/web does not work anymore.
If I unmount and remount it works again.
Unfortunately for the unmount/mount to work I have to kill things like httpd.
So to do a simple dir move (because I had it in the wrong place) on a read-only dir, I have to kill my service.

I have done exactly this with an nfs mount and it did not fail at all, I did not have to kill httpd and I did not have to unmount/remount the share.

------------------
--- server.vol ---
------------------
# $ /usr/bin/glusterfs-volgen -n tcb_data -p 50001 -r 1 -c /etc/glusterfs 10.0.0.24:/mnt/tcb_data 10.0.0.25:/mnt/tcb_data

######################################
# Start tcb share
######################################
volume tcb_posix
   type storage/posix
   option directory /mnt/tcb_data
end-volume

volume tcb_locks
     type features/locks
     subvolumes tcb_posix
end-volume

volume tcb_brick
     type performance/io-threads
     option thread-count 8
     subvolumes tcb_locks
end-volume

volume tcb_server
     type protocol/server
     option transport-type tcp
     option auth.addr.tcb_brick.allow *
     option transport.socket.listen-port 50001
     option transport.socket.nodelay on
     subvolumes tcb_brick
end-volume

------------------
--- tcb client.vol ---
------------------
volume tcb_remote_glust1
         type protocol/client
         option transport-type tcp
         option ping-timeout 5
         option remote-host 10.0.0.24
         option transport.socket.nodelay on
         option transport.remote-port 50001
         option remote-subvolume tcb_brick
end-volume

volume tcb_remote_glust2
         type protocol/client
         option transport-type tcp
         option ping-timeout 5
         option remote-host 10.0.0.25
         option transport.socket.nodelay on
         option transport.remote-port 50001
         option remote-subvolume tcb_brick
end-volume

volume tcb_mirror
         type cluster/replicate
         subvolumes tcb_remote_glust1 tcb_remote_glust2
end-volume

volume tcb_writebehind
         type performance/write-behind
         option cache-size 4MB
         subvolumes tcb_mirror
end-volume

volume tcb_readahead
         type performance/read-ahead
         option page-count 4
         subvolumes tcb_writebehind
end-volume

volume tcb_iocache
         type performance/io-cache
         option cache-size `grep 'MemTotal' /proc/meminfo  | awk '{print $2 * 0.2 / 1024}' | cut -f1 -d.`MB
         option cache-timeout 1
         subvolumes tcb_readahead
end-volume

volume tcb_quickread
         type performance/quick-read
         option cache-timeout 1
         option max-file-size 64kB
         subvolumes tcb_iocache
end-volume

volume tcb_statprefetch
         type performance/stat-prefetch
         subvolumes tcb_quickread
end-volume

^C



Tejas N. Bhise wrote:
> Chad, Stephan - thank you for your feedback.
> 
> Just to clarify on what wrote, do you mean to say that -
> 
> 1) The setup is a replicate setup with the file being written to multiple nodes.
> 2) One of these nodes is brought down.
> 3) A replicated file with a copy on the node brought down is written to.
> 4) The other copies are updates as writes  happen while this node is still down.
> 5) After this node is brought up, the client sometimes sees the old file on the node brought up
> instead of picking the file from a node that has the latest copy.
> 
> If the above is correct, quick questions -
> 
> 1) What versions are you using ?
> 2) Can you share your volume files ? Are they generated using volgen ? 
> 3) Did you notice any patterns for the files where the wrong copy was picked ? like 
> were they open when the node was brought down ?
> 4) Any other way to reproduce the problem ?
> 5) Any other patterns you observed when you see the problem ?
> 6) Would you have listings of problem file(s) from the replica nodes ?
> 
> If however my understanding was not  correct, then please let me know with some
> examples.
> 
> Regards,
> Tejas.
> 
> ----- Original Message -----
> From: "Chad" <ccolumbu at hotmail.com>
> To: "Stephan von Krawczynski" <skraw at ithnet.com>
> Cc: gluster-users at gluster.org
> Sent: Sunday, March 7, 2010 9:32:27 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi
> Subject: Re: How to re-sync
> 
> I actually do prefer top post.
> 
> Well this "overwritten" behavior is what I saw as well and that is a REALLY REALLY bad thing.
> Which is why I asked my question in the first place.
> 
> Is there a gluster developer out there working on this problem specifically?
> Could we add some kind of "sync done" command that has to be run manually and until it is the failed node is not used?
> The bottom line for me is that I would much rather run on a performance degraded array until a sysadmin intervenes, than loose any data.
> 
> ^C
> 
> 
> 
> Stephan von Krawczynski wrote:
>> I love top-post ;-)
>>
>> Generally, you are right. But in real-life you cannot trust on this
>> "smartness". We tried exactly this point and had to find out that the clients
>> do not always select the correct file version (i.e. the latest) automatically.
>> Our idea in the testcase was to bring down a node, update its kernel an revive
>> it - just as you would like to do it in real world for a kernel update.
>> We found out that some files were taken from the downed node afterwards and
>> the new contents on the other node got in fact overwritten.
>> This does not happen generally, of course. But it does happen. We could only
>> stop this behaviour by setting "favorite-child". But that does not really help
>> a lot, since we want to take down all nodes some other day.
>> This is in fact one of our show-stoppers.
>>
>>
>> On Sun, 7 Mar 2010 01:33:14 -0800
>> Liam Slusser <lslusser at gmail.com> wrote:
>>
>>> Assuming you used raid1 (distribute), you DO bring up the new machine
>>> and start gluster.  On one of your gluster mounts you run a ls -alR
>>> and it will resync the new node.  The gluster clients are smart enough
>>> to get the files from the first node.
>>>
>>> liam
>>>
>>> On Sat, Mar 6, 2010 at 11:48 PM, Chad <ccolumbu at hotmail.com> wrote:
>>>> Ok, so assuming you have N glusterfsd servers (say 2 cause it does not
>>>> really matter).
>>>> Now one of the servers dies.
>>>> You repair the machine and bring it back up.
>>>>
>>>> I think 2 things:
>>>> 1. You should not start glusterfsd on boot (you need to sync the HD first)
>>>> 2. When it is up how do you re-sync it?
>>>>
>>>> Do you rsync the underlying mount points?
>>>> If it is a busy gluster cluster it will be getting new files all the time.
>>>> So how do you sync and bring it back up safely so that clients don't connect
>>>> to an incomplete server?
>>>>
>>>> ^C
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> 
> 


[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux