Ok, I'm afraid I'm not agreeing with some of the things you're saying, but that doesn't really get us anywhere. I think the bottom line is, having read or write operations block for any length of time in order to merge a node back into the cluster following an outage simply isn't "real world". "any length of time" will vary for different operators, for me it's seconds, for anyone running real-time systems or VoIP for example, it might be milliseconds. My feeling is that self-heal "on open" is "nice", easy to code and maintain, neat, and I can see "how it came to be". It's now however practical for a commercial solution, healing needs to be a background process, not a foreground process. The only question I really have is, what sort of timescale are we looking at before GlusterFS has the kind of capabilities (in context) that one would expect of a production clustering filesystem ? (I think we all know hash/sync would be an interim solution) (I know it's "different", but just as a point of reference, DRBD syncs in the background, as does Linux software raid .. can you imagine anyone using either if they tried to make people wait while they did a foreground sync ??) Gareth. ----- Original Message ----- From: gordan@xxxxxxxxxx To: gluster-devel@xxxxxxxxxx Sent: Wednesday, April 30, 2008 1:56:12 PM GMT +00:00 GMT Britain, Ireland, Portugal Subject: Re: Re; Load balancing ... On Wed, 30 Apr 2008, Gareth Bult wrote: >> It would certainly ber beneficial in the cases when the network speed >> is slow (e.g. WAN replication). > > So long as it's server side AFR and not client-side ... ? Sure. > I'm guessing there would need to be some server side logic to ensure > that local servers generated their own hashes and only exchanged the > hashes over the network rather than the data ? Indeed - same as rsync does. >> Journal per se wouldn't work, because that implies fixed size and write-ahead logging. >> What would be required here is more like the snapshot style undo logging. > > A journal wouldn't work ?! > You mean it's effectiveness would be governed by it's size? Among other things. A "journal" just isn't suitable for this sort of thing. >> 1) Categorically establish whether each server is connected and up to date >> for the file being checked, and only log if the server has disconnected. >> This involves overhead. > > Surely you would log anyway, as there could easily be latency between an > actual "down" and one's ability to detect it .. in which case detecting > whether a server has disconnected it a moot point. Not really. A connected client/server will have a live/working TCP connection open. Read-locks don't matter as they can be served locally, but when a write occurs, the file gets locked. If a remote machine doesn't ack the lock, and/or it's TCP connection resets, then it's safe to assume that it's not connected. > In terms of the > overhead of logging, I guess this would be a decision for the sysadmin > concerned, whether the overhead of logging to a journal was worthwhile > .vs. the potential issues involved in recovering from an outage? That complicates things further, then. You'd essentially have asynchronous logging/replication. At that point you pretty much have to log all writes all the time. That means potentially huge space and speed overheads. > From my point of view, if journaling halved my write performance (which > it wouldn't) I wouldn't even have to think about it. Actually, saving an undo-log a-la snapshots, which is what would be required, _WOULD_ halve your write performance on all surviving servers if one server was out. If multiple servers were out, you could probably work around some of this with merging/splitting the undo logs for various machines, so your write performance would generally be around 1/2 of standard, but wouldn't end up degrading to 1/n+1 where n is the number of failed servers for which the logging needs to be done. >> The problem that arises then is that the fast(er) resyncs on small changes >> come at the cost of massive slowdown in operation when you have multiple >> downed servers. As the number of servers grows, this rapidly stops being a >> workable solution. > > Ok, I don't know about anyone else, but my setups all rely on > consistency rather than peaks and troughs. I'd far rather run a journal > at half potential speed, and have everything run at that speed all the > time .. than occasionally have to stop the entire setup while the system > recovers, or essentially wait for 5-10 minutes while the system re-syncs > after a node is reloaded. There may be a way to address the issue of halting the rest of the cluster during the sync, though. Read lock on a syncing file shouldn't stop other read locks. Of course, it will block writes while the file syncs and the reading app finishes the operation. Gordan _______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxx http://lists.nongnu.org/mailman/listinfo/gluster-devel
