Non-traditional Failover Query

Linux Advanced Routing and Traffic Control

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

Partly because I never like straightforward solutions,
I am looking to implement a non-standard failover
system that owes its origins to mixing RAID 5 with
some beer.

The idea is to have machines A, B and C, configured as
follows:

1) Any given process is running on TWO machines at the
same time. If a process or machine fails, then a new
backup is started on the third machine. There is thus
a rotation around the nodes.

2) Packets destined for any of the machines should be
received by ALL of the machines.

3) If the primary process is on A, then replies from B
and C should still be generated, but be transparently
dropped. Likewise, if the primary process is on B or
C.

Let's say you have an Apache process running on A and
B. B is shadowing A on everything. It's at the same
point, has the same connections established, etc.
Failover becomes merely "ungagging" B.

How is this a LARTC problem?

Uhhh... because this approach is seriously abusing the
entire networking stack. We essentially have all three
machines running with an identical MAC and IP address
visible to the outside. The "distinct" address is
purely internal.

What this requires is a way of tricking all three
machines into believing that they are the sole
recipients. This keeps the stacks in a uniform state,
which means we can fail-over the connections without
having to either checkpoint or copy stateful
information, both of which get ugly when you start
talking about lots of information.

This leads to the second network-related problem. If
you have two identical machines starting from
identical states, and processing identical streams,
then they should end up in identical states - ie:
crashed.

This is easily fixed. If A is the machine you are
starting all the processes on, and B is your "mirror",
then C needs to take up the excess load.

In other words, A+C is a cluster, and B+C is a second
cluster. Processes migrated to C from A or B aren't
mirrored. (This is akin to RAID 5's partial backup.)

So, the three machines need to be seen from four
distinct views:

A) From outside, a single machine is visible.

B) From the HA perspective, there is one primary
machine, one mirror machine and one spare

C) From the load-balance perspective, there are two
overlapping clusters.

D) From the LAN perspective, there are three distinct,
uniquely-addressable machines.

Using Linux' advanced networking, BPF and netfilter
layers, connection mirroring for HA/load-balancing
purposes should be straight-forward. Much more so than
using the "wedge" concept proposed by MIT. Because you
need send nothing more than flags between the
machines, it should also be less expensive on the LAN
and processor.

So... how would you go about doing this?


__________________________________
Do you Yahoo!?
Yahoo! Mail SpamGuard - Read only the mail you want.
http://antispam.yahoo.com/tools
_______________________________________________
LARTC mailing list / LARTC@xxxxxxxxxxxxxxx
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

[Index of Archives]     [LARTC Home Page]     [Netfilter]     [Netfilter Development]     [Network Development]     [Bugtraq]     [GCC Help]     [Yosemite News]     [Linux Kernel]     [Fedora Users]
  Powered by Linux