Hi All, There has been lot of mails in the recent times on AFR and HA, I guess the reason was lack of documentation on self-heal which has been pending for long. In case any user thinks that his mail was not responded to or if there was an incomplete response, please follow it up in the mailing-list or IRC. There is a change in AFR's functionality regarding the "option replicate" feature. We realised that design wise it is not good to have this inside AFR and that it is better to have it outside AFR as a separate translator. This will not affect the users who have been using "option replicate *:n" where n is the number of subvols. For people making use of this option the inconvenience caused is regretted. The pattern matching translator will be available in 1.4 release. So the over all functionality is not being compromised. The reasons for taking "option replicate" feature out of AFR: * It does not belong there. You can guess by the fact that the "option replicate" option has to be exactly same across all the AFRs, if it is same then we have to be able to specify it at a place common to all AFRs instead of specifying the same option in each AFR. * "option replicate" was making the working of selfheal more complicated. We had to come up with workarounds to make it work. In the long run workarounds are not good. Here is the document which will be put up in the wiki. Any feedback regarding what should be added or any form of suggestions will be appreciated. ========= AFR provides RAID-1 like functionality. AFR replicates files and directories across the subvolumes. Hence if AFR has four subvolumes, there will be four copies of all files and directories. AFR provides HA, i.e in case one of the subvolumes go down (ex. server crash, network disconnection) AFR will still service the requests from the redundant copies. AFR also provides self-heal functionality, i.e in case the crashed servers comeup, the outdated files and directories will be updated with the latest versions. AFR uses extended attributes of the backend file system to track the versioning of files and directories to provide the self-heal feature. * Note that previously supported "option replicate *html:2,*txt:1" pattern matching feature is moved out of AFR. It will be provided as a separate translator in 1.4 volume afr-example type cluster/afr subvolumes brick1 brick2 brick3 end-volume This sample configuration will replicate all directories and files on brick1, brick2 and brick3. The subvolumes can be another translator (storage/posix or protocol/client) All the read() operations happen from the first alive child. If all the three subvols are up, read() will be done on brick1, if brick1 is down read() will be done on brick2. In case read() was being done on brick1 and it goes down, we fallback to brick2 which will be completely transparent to the user applications. In 1.4 we will have: * a feature where user can specify the subvol from which AFR has to do read() operations (this will help users who have one of the subvols as local storage/posix) * feature to allow scheduling of read() operations amongst the subvols in round-robin fashion. The order of the subvolumes list should be same across all the AFRs as they will be used as lock servers. TODO: details on working of locking. Self-Heal AFR has self-heal feature, which updates the outdated file and directory copies by the most recent versions. For example consider the following config: volume afr-example type cluster/afr subvolumes brick1 brick2 end-volume File self-heal Now if we create a file foo.txt on afr-example, the file will be created on brick1 and brick2. The file will have two extended attributes associated with it in the backend filesystem. One is trusted.afr.createtime and the other is trusted.afr.version. The trusted.afr.createtime xattr has the create time (in terms of seconds since epoch) and trusted.afr.version is a number that is incremented each time a file is modified. This increment happens during close (incase any write was done before close). If brick1 goes down, we edit foo.txt the version gets incremented. Now the brick1 comes back up, when we open() on foo.txt AFR will check if their versions are same. If they are not same, the outdated copy is replaced by the latest copy and its version is updated. After the sync the open() proceeds in the usual manner and the application calling open() can continue on its access to the file. If brick1 goes down, we delete foo.txt and create a file with the same name again i.e foo.txt. Now brick1 comes back up, clearly there is a chance that the version on brick1 being more than the version on brick2, this is where createtime extended attribute helps in deciding which the outdated copy is. Hence we need to consider both createtime and version to decide on the latest copy. The version attribute is incremented during the close() call. Version will not be incremented in case there was no write() done. In case the fd that the close() gets was got by create() call, we also create the createtime extended attribute. Directory self-heal Suppose brick1 goes down, we delete foo.txt, brick1 comes back up, now we should not create foo.txt on brick2 but we should delete foo.txt on brick1. We handle this situation by having the createtime and version attribute on the directory similar to the file. when lookup() is done on the directory, we compare the createtime/version attributes of the copies and see which files needs to be deleted and delete those files and update the extended attributes of the outdated directory copy. Each time a directory is modified (a file or a subdirectory is created or deleted inside the directory) and one of the subvols is down, we increment the directory's version. lookup() is a call initiated by the kernel on a file or directory just before any access to that file or directory. In glusterfs, by default, lookup() will not be called in case it was called in the past one second on that particular file or directory. The extended attributes can be seen in the backend filesystem using the getfattr command. (getfattr -n trusted.afr.version <file>) ========
