Martin Fick wrote:
Why is that the correct way? There's nothing
wrong with having "bonding" at the glusterfs
protocol level, is there?
The problem is that it only covers a very narrow edge case
that isn't all that likely. A bonded NIC over separate
switches all the way to both servers is a much more sensible
option. Or else what failure are you trying to protect
yourself against? It's a bit like fitting a big padlock
on the door when there's a wall missing.
I think you need to be more specific then using
analogies. My only guess from your assertions is
that you have a very narrow specific use case /
setup / terminology in mind that does not
necessarily mesh with my narrow use case ... :)
LOL! That is a distinct possibility. :)
So, the HA translator supports talking to two
different servers with two different transport
mechanism and two different IPs. Bonding does
not support anything like this is far as I can
tell.
True. Bonding is more transparent. You make two NICs into one virtual
NIC and round-robin packets down them. If one NIC/path fails, all the
traffic will fail over to the other NIC/path.
So, it seems like you are assuming a
different back end use case, one where the
servers employ the same IP perhaps using round
robin or perhaps in an active passive way.
No, not at all. Multiple servers, 1 floating IP per server. Floating as
in it can be migrated to the other server if one fails. You balance the
load by assigning half of your clients to one floating IP, and the other
half of the clients to the other floating IP. So, when both servers are
up, each handles half the load. If one server fails, it's IP gets
migrated to the other server, and all clients thereafter talk to the
surviving server since it has both IPs (until the other server comes
back up and asks for it's IP address back).
Both
of these are very different beasts and I would
need to know which you are talking about to
understand what you are getting at. But the HA
translator setup is closer to the round robin
(active/active) setup and I am guessing you
are taking about an active / passive setup.
In general, there are relatively few things that you cannot make
active/active, so I always mean active/active + failover unless I
explicitly state it.
That is somewhat what the HA translator is, except
that it is supposed to take care of some additional
failures. It is supposed to retransmit "in
progress" operations that have not succeeded because of
comm failures (I have yet to figure out where in the code
this happens though).
This is a reinvention of a wheel. NFS already handles this
gracefully for the use-case you are describing.
I am lost, what does NFS have to do with it?
It already handles the "server has gone away" situation gracefully. What
I'm saying is that you can use GlusterFS underneath for mirroring the
data (AFR) and re-export with NFS to the clients. If you want to avoid
client-side AFR and still have graceful failover with lightweight
transport, NFS is not a bad choice.
Why re-invent the wheel when the tools to deal
with these
failure modes already exist?
Are you referring to bonding here? If so, see above
why HA may be better (or additional benefit).
My original point is that it doesn't add anything new
that you couldn't achieve with tools that are already
available.
Well, I was trying to explain to you that it
does, but then the NFS thing, I am confused.
How do current tools achieve the following
setup? Client A talks to Server A and
submits a read request. The read request
is received on Server A (TCP acked to the
client), and then Server A dies. How will
the following request be completed without
glusterfs returning an "endpoint not
connected" error?
You make client <-> server comms NFS.
You make server <-> server comms GlusterFS.
If the NFS server goes away, the client will keep retrying until the
server returns. In this case, that would mean it'll keep retrying until
the other server fails the IP address over to itself.
This achieves:
1) server side AFR with GlusterFS for redundancy
2) client connects to a single server via NFS so there's no
double-bandwidth used by the client
3) servers can fail over relatively transparently to the client
No, I have not confirmed that this actually
works with the HA translator, but I was told
that the following would happen if it were
used. Client A talks to Server A and
submits a read request. The read request
is received on Server A (TCP acked to the
client), and then Server A dies. Client A
will then in theory retry the read request
on Server B. Bonding cannot do anything
like this (since the read was tcp ACKed)?
Agreed, if a server fails, bonding won't help. Cluster fail-over
server-side, however, will, provided the network file system protocol
can deal with it reasonably well.
Neither can heartbeat/failover
of an active/passive backend since on the
first failure the client will get a
connection error and the glusterfs client
protocol does not retransmit).
This is where I clearly failed to clarify what I meant. I was talking
about using NFS for the client<->server part of the communication. NFS
will typically block until the server starts responding again (note: it
doesn't have to be the same server, just one like it).
I think that this is quite different from
any bonding solution. Not better, different,
If I were to use this it would not preclude
me from also using bonding, but it solves a
somewhat different problem. It is not a
complete solution, it is a piece, but not
a duplicated piece. If you don't like it,
or it doesn't fit your backend use case,
don't use it! :)
If it can handle the described failure more gracefully than what I'm
proposing, then I'm all for it. I'm just not sure there is that much
scope for it being better since the last write may not have made it to
the mirror server anyway, so even if the protocol can re-try, it would
need to have some kind of journaling, roll back the journal and replay
the operation.
This, however, is a much more complex approach (very similar to what GFS
does), and there is a high price to pay in terms of performance when the
nodes aren't on the same LAN.
Yes, if a server goes down you are fine (aside from the
scenario where the other server then goes down followed
by the first one coming back up). But, if you are using
the HA translator above and the communication goes down
between the two servers you may still get split brain
(thus the need for heartbeat/fencing).
And therein lies the problem - unless you are proposing
adding a complete fencing infrastructure into glusterfs,
too.
No. I am proposing adding a complete transactional
model to AFR so that if a write fails on one node,
some policy can decide whether the same write
should be committed of rolled back on the other
nodes. Today, the policy is to simply apply it to
the other nodes regardless. This is a recipe for
split brain.
OK, I get what you mean. It's basically the same problem I described
above when I mentioned that you'd need some kind of a journal to
roll-back the operation that hasn't been fully committed.
In the case of network segregation some policy
should decide to allow writes to be applied
to one side of the segregation and denied on the
other. This does not require fencing (but it
would be better with it), it could be a simple
policy like: "apply writes if a majority of nodes
can be reached", if not fail (or block would be
even better).
Hmm... This could lead to an elastic shifting quorum. I'm not sure how
you'd handle resyncing if nodes are constantly leaving/joining. It seems
a bit non-deterministic.
AFR needs to be able write all or nothing to all
servers until some external policy machine (such as
heartbeat) decides that it is safe (because of fencing or
other mechanism) to proceed writing to only a portion of the
subvolumes (servers). Without this I don't see how you
can prevent split brain?
With server-side AFR, splitbrain cannot really occur (OK,
there's a tiny window of opportunity for it if the
server isn't really totally dead since there's no
total FS lock-out until fencing is completed like on GFS,
but it's probably close enough). If the server's
can't heartbeat to each other, they can't AFR to
each other, either. So either the write gets propagated, or
it doesn't. The machine that remained operational will
have more up to date files and as necessary those will get
synced back. It's not quite as tight as GFS in terms of
ensuring data consistency like a DRBD+GFS solution would be,
but it is probably close enough for most use-cases.
I guess what you call tiny, I call huge. Even if
you have your heartbeat fencing occur in under a
tenth of a second, that is time enough to split
brain a major portion of a filesystem. I would
never trust it.
In GlusterFS that problem exists anyway, but it is largely mitigated by
the fact that it works on file level rather than block device level. In
the case of GFS, RHCS will block all access to the file system until the
note is successfully fenced and confirmed fenced before rolling back
it's journals and resuming operation.
To borrow your analogy, adding heartbeat to the
current AFR: "It's a bit like fitting a big
padlock on the door when there's a wall missing."
:)
Every single write needs to ensure that it will
not cause split brain for me to trust it.
Sounds like GlusterFS isn't necessarily the solution for you, then. :(
If not, why would I bother with gluserfs over
AFR instead of glusterfs over DRBD? Oh right,
because I cannot get glusterfs to failover without
incurring connection errors on the client! ;)
(not your beef, I know, from another thread)
Precisely - which is why I originally suggested not using GlusterFS for
client-server communication. :)
This is one reason I was hoping that the HA
translator would address this, but the HA
translator is useless in an active/passive
backend setup, it only works in active/active.
If you try using it in an active/passive setup,
during failover it will retry too quickly on
the second server causing connection errors
on the client!!! This is the primary reason
that I am suggesting that the HA translator
block until the connection is restored, it
would allow for failovers to occur.
And this is exactly why I suggested using NFS for the client<->server
connection. NFS blocks until the server becomes contactable again.
But, to be clear, I am not disagreeing with you
that the HA translator does not solve the split
brain problem at all. Perhaps this is what is
really "upsetting" you, not that it is
"duplicated" functionality, but rather that it
does not help AFR solve it's split brain
personality disorders, it only helps make them
more available, thus making split brain even
more likely!! ;(
I'm not sure it makes it any worse WRT split-brain, it just seems that
you are looking for GlusterFS+HA to provide you with exactly the same
set of features that NFS+(server fail-over) already provides. Of course,
there could be advantages in GlusterFS behaving the same way as NFS when
the server goes away if it's a single-server setup - it would be easier
to set up and a bit more elegant. But it wouldn's add any functionality
that couldn't be re-created using the sort of a setup I described.
Gordan
