Re: Gluster Recovery

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi all,

To come back to the issue of "self-healing" in an AFR situation. Consider the rather complex situation below where A=AFR, S=stripe and U=unify:

/---- Server1---\ Client3 S---- Server2 ----S / /-------U---< >---U--------\ / S---- Server3 ----S \ / \---- Server4 ---/ \ Client1 ------A--< >---A------Client2 \ /---- Server5---\ / \ S---- Server6 ----S / \-------U---< >---U-------/
                                     /         S---- Server7 ----S
                              Client4       \---- Server8 ---/


Client1 and 2: AFR of two separate unions of two separate stripes
Client3 and 4: Union of two separate stripes

I think this is quite a complex arrangement and could probably account for 80% of large installation cases. The obvious question here is what would the method of healing be for a server failure. Some thoughts:

1) As mentioned later on in this thread, the flexibility of gluster is great but it is somewhat rediculouos to imagine that this flexibility frees one from using good cluster design. For instance, the following configuration is probably of little use, the clients must have a useful configuration, possible like the larger one above:
                      /---Server1---\
Client1---S---<                     >---U---Client2
                      \---Server2---/

2) If a server is replaced, healing must take place from any or all clients, otherwise the distributed nature of the system is lost.

3) No client should exist below a striping such as:
              Client2
                        \    /---- Server1...
                          U---- Server2...
...Client1---S---< U---- Server3....
                           \---- Server4....
Correct me if I'm wrong, but trying to read striped data as the above drawing shows for client2 would not be very useful to client2.

4) A suggestion here is to have each AFR client with a self-heal filter/translator. ONLY AFR clients should have self-healing for replication. Other clients such as the union clients can have self-healing filters but for different filesystem health checks. When a server fails and is replaced, all AFR clients get stuck in and attempt to reconstruct the data. Thus in this situation, Clients 1 and 2 will heal the system. Clients 3 and 4 cannot because they don't have a full set of data from which to work. 5) Who is the dominant reconstruction client? A simple possible solution is to have a "pre-healing" lock for each file to be reconstructed. For instance, Client1 finds "hello.c" in bad shape because of the failure. Client1 placed a lock file in the directory identifying itself with a timestamp. Client2 also notices that "hello.c" is in bad shape and moves to fix, but notices a lock file with a timestamp on it, and so will move on to another file/folder. If Client2 notices that the timestamp has not been updated in 20s or something reasonable, that means that Client1 has crashed or failed in some manner and is no longer healing "hello.c". Therefore Client2 will continue to heal "hello.c". Obviously, during healing, nothing else should access the file for fear of further corruption. Comments on that may run far, but so be it. 6) What it all comes down to is: 1) do not make the system's distributed nature worthless; let all clients get stuck in as if they were all trying to make breakfast. If someone is making the eggs, don't make eggs, go make the toast. If the eggs start burning because the cook went to the toilet, take over and finish the eggs. Soon enough, with clever co-operation, the breakfast will be done.

Comments?

Regards,
Danson Joseph




Anand Avati wrote:
The concern here is the following though:

Two separate clients are identically configured to use AFR to two identical server configurations as follows:
             Server1
           /         \
Client1 ---           ---Client2
           \         /
             Server2

Client1 puts "hello.c" onto both Server1 and Server2 via AFR.  Client2 then changes hello.c in some way.
Server1 goes down; data lost, no chance of recovery and is replaced by Server3, a brand new server with fresh disks.
In this case, how does the data get reconstructed from the client's side because you mentioned that the automatic recovery was going to be on the glusterfs side.  Client1 believes hello.c is something different to what Client2 believes.  Which client will responsibly reconstruct the data?  Will the journaling of the remaining servers be used to reconstruct the data on the new server?

'changes' are done in sync on both server1 and server2 always
(writes()s are sent to all child nodes). when server3 comes in place
of server1, the self-heal should detect that hello.c is missing on
server3 and sync it from server2.



regards,
avati





[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux