Re: Re; Load balancing ...

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



We have a work around for this so that clients don't see a "blocked" situation. we have a 6 node cluster and whenever a server node comes up it lists all the files on it's "slice" of the gluster mount and then does a find on those files on the main mounted volume. In other words each server is also a client, and does the heal process right after starting up.

-Mic

Gareth Bult wrote:
Ok, I'm afraid I'm not agreeing with some of the things you're saying, but that doesn't really get us anywhere.

I think the bottom line is, having read or write operations block for any length of time in order to merge a node back into the cluster following an outage simply isn't "real world". "any length of time" will vary for different operators, for me it's seconds, for anyone running real-time systems or VoIP for example, it might be milliseconds.

My feeling is that self-heal "on open" is "nice", easy to code and maintain, neat, and I can see "how it came to be".

It's now however practical for a commercial solution, healing needs to be a background process, not a foreground process.

The only question I really have is, what sort of timescale are we looking at before GlusterFS has the kind of capabilities (in context) that one would expect of a production clustering filesystem ? (I think we all know hash/sync would be an interim solution)

(I know it's "different", but just as a point of reference, DRBD syncs in the background, as does Linux software raid .. can you imagine anyone using either if they tried to make people wait while they did a foreground sync ??)

Gareth.


----- Original Message -----
From: gordan@xxxxxxxxxx
To: gluster-devel@xxxxxxxxxx
Sent: Wednesday, April 30, 2008 1:56:12 PM GMT +00:00 GMT Britain, Ireland, Portugal
Subject: Re: Re; Load balancing ...



On Wed, 30 Apr 2008, Gareth Bult wrote:

It would certainly ber beneficial in the cases when the network speed is slow (e.g. WAN replication).
So long as it's server side AFR and not client-side ... ?

Sure.

I'm guessing there would need to be some server side logic to ensure that local servers generated their own hashes and only exchanged the hashes over the network rather than the data ?

Indeed - same as rsync does.

Journal per se wouldn't work, because that implies fixed size and write-ahead logging.
What would be required here is more like the snapshot style undo logging.
A journal wouldn't work ?!
You mean it's effectiveness would be governed by it's size?

Among other things. A "journal" just isn't suitable for this sort of thing.

1) Categorically establish whether each server is connected and up to date
for the file being checked, and only log if the server has disconnected.
This involves overhead.
Surely you would log anyway, as there could easily be latency between an actual "down" and one's ability to detect it .. in which case detecting whether a server has disconnected it a moot point.

Not really. A connected client/server will have a live/working TCP connection open. Read-locks don't matter as they can be served locally, but when a write occurs, the file gets locked. If a remote machine doesn't ack the lock, and/or it's TCP connection resets, then it's safe to assume that it's not connected.

In terms of the overhead of logging, I guess this would be a decision for the sysadmin concerned, whether the overhead of logging to a journal was worthwhile .vs. the potential issues involved in recovering from an outage?

That complicates things further, then. You'd essentially have asynchronous logging/replication. At that point you pretty much have to log all writes all the time. That means potentially huge space and speed overheads.

From my point of view, if journaling halved my write performance (which it wouldn't) I wouldn't even have to think about it.

Actually, saving an undo-log a-la snapshots, which is what would be required, _WOULD_ halve your write performance on all surviving servers if one server was out. If multiple servers were out, you could probably work around some of this with merging/splitting the undo logs for various machines, so your write performance would generally be around 1/2 of standard, but wouldn't end up degrading to 1/n+1 where n is the number of failed servers for which the logging needs to be done.

The problem that arises then is that the fast(er) resyncs on small changes
come at the cost of massive slowdown in operation when you have multiple
downed servers. As the number of servers grows, this rapidly stops being a
workable solution.
Ok, I don't know about anyone else, but my setups all rely on consistency rather than peaks and troughs. I'd far rather run a journal at half potential speed, and have everything run at that speed all the time .. than occasionally have to stop the entire setup while the system recovers, or essentially wait for 5-10 minutes while the system re-syncs after a node is reloaded.

There may be a way to address the issue of halting the rest of the cluster during the sync, though. Read lock on a syncing file shouldn't stop other read locks. Of course, it will block writes while the file syncs and the reading app finishes the operation.

Gordan


_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel


_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel


--




[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux