>Hmm, I'm confused, surely in both these cases there is
>not much more network overhead than when the 20G file
>was supposed to have been written in the first place
>is there?
?!
>If you have two nodes and the 20 GB file
>only got written to node A while node B was down and
>node B comes up the whole 20 GB is resynced to node B;
>is that more network usage than if the 20 GB file were
>written immediately to node A & node B.
Ah. Let's say you have both nodes running with a 20Gb file synced.
Then you have to restart one glusterfs on one of the nodes.
While it's down, let's say the other node appends 1 byte to the file.
When it comes back up and looks a the file, the other node will see it's out of date and re-copy the entire 20Gb.
>Perhaps the issue is really that the cost comes at an
>unexpected time, on node startup instead of when the
>file was originally written? Would a startup
>throttling mechanism help here on resyncs?
Yes, unfortunately you can't open a file while it's syncing .. so when you reboot your gluster server, downtime is the length of time it takes to restart glusterfs (or the machine, either way..) PLUS the amount of time it takes to recopy every file that was written to while one node was down ...
Take a Xen server for example serving disk images off a gluster partition.
10 Images at 10G each gives you a 100G copy to do.
Wait, it get's better .. it will only re-sync the file on opening, so you actually have to close all the files, then try to re-open them , then wait while it re-syncs the data (during this time your cluster is effectively down), then the file open completes and you are back up again.
Yet there is a claim in the FAQ that there is no single point of failure .. yet to upgrade gluster for example you effectively need to shut down the entire cluster in order to get all files to re-sync ...
!!!
Effectively storing anything like a large file on AFR is pretty unworkable and makes split-brian issues pale into insignificance ... or at least that's my experience of trying to use it...
Gareth.
--
Managing Director, Encryptec Limited
Tel: 0845 5082719, Mob: 0785 3305393
Email: gareth@xxxxxxxxxxxxx
Statements made are at all times subject to Encryptec's Terms and Conditions of Business, which are available upon request.
----- Original Message -----
From: "Martin Fick" <mogulguy@xxxxxxxxx>
To: gordan@xxxxxxxxxx, gluster-devel@xxxxxxxxxx
Sent: Friday, April 25, 2008 4:44:36 PM GMT +00:00 GMT Britain, Ireland, Portugal
Subject: Re: Re; Load balancing ...
--- gordan@xxxxxxxxxx wrote:
> On Fri, 25 Apr 2008, Gareth Bult wrote:
...
> > Moving to the other end of the scale, AFR can't
> > cope with large files either .. handling of sparse
> > files doesn't work properly and self-heal
> > has no concept of repairing part of a file .. so
> > sticking a 20Gb file on a GlusterFS is just asking
> > for trouble as every time you restart a
> > gluster server (or every time one crashes) it'll
> > crucify your network.
>
> I thought about this, and there isn't really a way
> to do anything about this, unless you relax the
> constraints. You could to a rsync-type rolling
> checksum block-sync, but this would both take up
> more CPU time and result in theoretical scope for
the
> file to not be the same on both ends. Whether
> this minute possibility of corruption that the
> hashing algorithm doedn't pick up is a reasonable
> trade-off, I don't know. Perhaps if such a thing
> were implemented it should be made optional.
Hmm, I'm confused, surely in both these cases there is
not much more network overhead than when the 20G file
was supposed to have been written in the first place
is there? If you have two nodes and the 20 GB file
only got written to node A while node B was down and
node B comes up the whole 20 GB is resynced to node B;
is that more network usage than if the 20 GB file were
written immediately to node A & node B.
Perhaps the issue is really that the cost comes at an
unexpected time, on node startup instead of when the
file was originally written? Would a startup
throttling mechanism help here on resyncs?
-Martin
____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
