Hello and again.. after some time .. rsync stop working in storage.log I see a lot of message like: [2013-08-16 02:56:08.025068] I [afr-self-heal-common.c:1213:afr_sh_missing_entry_call_impunge_recreate] 0-storage-replicate-0: no missing files - /home/bizexperts/data/www/bizexpert.kg/libraries/joomla/database/databaseexception.php. proceeding to metadata check [2013-08-16 02:56:08.033602] I [afr-self-heal-common.c:1213:afr_sh_missing_entry_call_impunge_recreate] 0-storage-replicate-0: no missing files - /home/bizexperts/data/www/bizexpert.kg/libraries/joomla/database/exception.php. proceeding to metadata check I checked this file on both nodes (peers) on first: root at mirror:~# ls -al /home/bizon1/data/www/bmx.on.kg/engine/editor/static.php -rw-r--r-- 2 bizon1 bizon1 12331 Sep 25 2012 /home/bizon1/data/www/bmx.on.kg/engine/editor/static.php root at mirror:~# on second: root at nas2-backup:/# ls -al /home/bizon1/data/www/bmx.on.kg/engine/editor/static.php -rw-r--r-- 1 bizon1 bizon1 12331 Sep 25 2012 /home/bizon1/data/www/bmx.on.kg/engine/editor/static.php root at nas2-backup:/# and on client: root at ispcp:/var/log/glusterfs# ls -al /home/bizon1/data/www/bmx.on.kg/engine/editor/static.php -rw-r--r-- 1 bizon1 bizon1 12331 Sep 25 2012 /home/bizon1/data/www/bmx.on.kg/engine/editor/static.php root at ispcp:/var/log/glusterfs# when i started rsync on client: rsync --log-file=/root/home_sync -altusgopr --inplace --exclude-from=/root/exclude.txt -e ssh --delete --delete-excluded root at 192.168.15.71:/storage/home/ /storage/home/ i checked logs storage.log on client and here is some lines: [2013-08-16 03:01:05.103442] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/.test.sh (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:05.103511] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/.test.sh under storage-client-0 failed (No data available) [2013-08-16 03:01:05.103558] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/mc-1181 (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:05.103571] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/mc-1181 under storage-client-0 failed (No data available) [2013-08-16 03:01:05.103593] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/mc-3031 (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:05.103607] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/mc-3031 under storage-client-0 failed (No data available) [2013-08-16 03:01:05.103710] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/doctrine_schema_22469.yml (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:05.103782] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/doctrine_schema_22469.yml under storage-client-0 failed (No data available) [2013-08-16 03:01:05.106094] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/mc-2290 (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:05.106163] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/mc-2290 under storage-client-0 failed (No data available) [2013-08-16 03:01:05.106205] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/mc-mad (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:05.106241] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/mc-mad under storage-client-0 failed (No data available) [2013-08-16 03:01:05.110690] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/mc-1181: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:05.110741] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/.test.sh: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:05.110777] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/mc-2290: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:05.111228] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/mc-3031: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:05.114187] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/mc-mad: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:05.114278] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/doctrine_schema_22469.yml: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:05.116824] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-storage-replicate-0: background entry self-heal failed on /home/tmp [2013-08-16 03:01:06.807915] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/.test.sh (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:06.807976] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/.test.sh under storage-client-0 failed (No data available) [2013-08-16 03:01:06.808030] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/mc-1181 (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:06.808057] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/mc-1181 under storage-client-0 failed (No data available) [2013-08-16 03:01:06.808115] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/mc-3031 (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:06.808138] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/mc-3031 under storage-client-0 failed (No data available) [2013-08-16 03:01:06.808176] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/doctrine_schema_22469.yml (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:06.808193] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/doctrine_schema_22469.yml under storage-client-0 failed (No data available) [2013-08-16 03:01:06.809869] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/mc-2290 (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:06.809910] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/mc-2290 under storage-client-0 failed (No data available) [2013-08-16 03:01:06.809972] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-storage-client-0: remote operation failed: No data available. Path: /home/tmp/mc-mad (00000000-0000-0000-0000-000000000000) [2013-08-16 03:01:06.809987] I [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk] 0-storage-replicate-0: looking up /home/tmp/mc-mad under storage-client-0 failed (No data available) [2013-08-16 03:01:06.813612] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/mc-1181: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:06.813681] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/.test.sh: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:06.813711] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/mc-2290: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:06.813740] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/mc-3031: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:06.816162] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/mc-mad: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:06.816261] I [afr-self-heal-entry.c:1837:afr_sh_entry_common_lookup_done] 0-storage-replicate-0: /home/tmp/doctrine_schema_22469.yml: Skipping entry self-heal because of gfid absence [2013-08-16 03:01:06.817643] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-storage-replicate-0: background entry self-heal failed on /home/tmp Ok then can i do next before run rsync on peer i will shutdown glusterfs-server on both peers.. then I will do rsync, and after finishing rsync i will start glusterfs-server on bith peer ,,, but i do not think and files will be updated on second peer ------------------------------------------------------- ??????? ????????? ????????????? ??????? ????? ???? "?????? kg" - http://www.hoster.kg ??. ????????? 123 (?????? ????) help at hoster.kg