We have a work around for this so that clients don't see a "blocked"
situation. we have a 6 node cluster and whenever a server node comes up
it lists all the files on it's "slice" of the gluster mount and then
does a find on those files on the main mounted volume. In other words
each server is also a client, and does the heal process right after
starting up.
-Mic
Gareth Bult wrote:
Ok, I'm afraid I'm not agreeing with some of the things you're saying, but that doesn't really get us anywhere.
I think the bottom line is, having read or write operations block for any length of time in order to merge a node back into the cluster following an outage simply isn't "real world". "any length of time" will vary for different operators, for me it's seconds, for anyone running real-time systems or VoIP for example, it might be milliseconds.
My feeling is that self-heal "on open" is "nice", easy to code and maintain, neat, and I can see "how it came to be".
It's now however practical for a commercial solution, healing needs to be a background process, not a foreground process.
The only question I really have is, what sort of timescale are we looking at before GlusterFS has the kind of capabilities (in context) that one would expect of a production clustering filesystem ?
(I think we all know hash/sync would be an interim solution)
(I know it's "different", but just as a point of reference, DRBD syncs in the background, as does Linux software raid .. can you imagine anyone using either if they tried to make people wait while they did a foreground sync ??)
Gareth.
----- Original Message -----
From: gordan@xxxxxxxxxx
To: gluster-devel@xxxxxxxxxx
Sent: Wednesday, April 30, 2008 1:56:12 PM GMT +00:00 GMT Britain, Ireland, Portugal
Subject: Re: Re; Load balancing ...
On Wed, 30 Apr 2008, Gareth Bult wrote:
It would certainly ber beneficial in the cases when the network speed
is slow (e.g. WAN replication).
So long as it's server side AFR and not client-side ... ?
Sure.
I'm guessing there would need to be some server side logic to ensure
that local servers generated their own hashes and only exchanged the
hashes over the network rather than the data ?
Indeed - same as rsync does.
Journal per se wouldn't work, because that implies fixed size and write-ahead logging.
What would be required here is more like the snapshot style undo logging.
A journal wouldn't work ?!
You mean it's effectiveness would be governed by it's size?
Among other things. A "journal" just isn't suitable for this sort of
thing.
1) Categorically establish whether each server is connected and up to date
for the file being checked, and only log if the server has disconnected.
This involves overhead.
Surely you would log anyway, as there could easily be latency between an
actual "down" and one's ability to detect it .. in which case detecting
whether a server has disconnected it a moot point.
Not really. A connected client/server will have a live/working TCP
connection open. Read-locks don't matter as they can be served locally,
but when a write occurs, the file gets locked. If a remote machine doesn't
ack the lock, and/or it's TCP connection resets, then it's safe to assume
that it's not connected.
In terms of the
overhead of logging, I guess this would be a decision for the sysadmin
concerned, whether the overhead of logging to a journal was worthwhile
.vs. the potential issues involved in recovering from an outage?
That complicates things further, then. You'd essentially have asynchronous
logging/replication. At that point you pretty much have to log all writes
all the time. That means potentially huge space and speed overheads.
From my point of view, if journaling halved my write performance (which
it wouldn't) I wouldn't even have to think about it.
Actually, saving an undo-log a-la snapshots, which is what would be
required, _WOULD_ halve your write performance on all surviving servers if
one server was out. If multiple servers were out, you could probably work
around some of this with merging/splitting the undo logs for various
machines, so your write performance would generally be around 1/2 of
standard, but wouldn't end up degrading to 1/n+1 where n is the number of
failed servers for which the logging needs to be done.
The problem that arises then is that the fast(er) resyncs on small changes
come at the cost of massive slowdown in operation when you have multiple
downed servers. As the number of servers grows, this rapidly stops being a
workable solution.
Ok, I don't know about anyone else, but my setups all rely on
consistency rather than peaks and troughs. I'd far rather run a journal
at half potential speed, and have everything run at that speed all the
time .. than occasionally have to stop the entire setup while the system
recovers, or essentially wait for 5-10 minutes while the system re-syncs
after a node is reloaded.
There may be a way to address the issue of halting the rest of the cluster
during the sync, though. Read lock on a syncing file shouldn't stop other
read locks. Of course, it will block writes while the file syncs and the
reading app finishes the operation.
Gordan
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
--
