Brian, These logs related to rsync need not be worried. Is rsync actually completing? these log messages will be fixed soon. Thanks, avati > Hi, > > Thanks for your help thus far. I'm continuing my testing and have run > across some errors in my logfiles. I'm using rsync from a local directory > to the gluster volume to load up some test data. This is the only > processing modifying the gluster filesystem. Occasionally, I'll see > self-heals triggered while the the rsync is running. With the exception of > this one, they all finish without any errors reported. > > I thought it was strange that selfheals were being triggered since all of > the servers are up and rsync is the only process modifying the gluster > filesystem, but since it seemed harmless I ignored them. This time it > seemed to be more of an error. While rsync is copying files, it uses a > different filename than the file that's actually being copied, and once the > copy is complete, the file is renamed to it's original name. There seems > to have been a race condition here where a self-heal is triggered and during > that time, the file is renamed. The logs on brick1 and brick3 seem > consistent with this theory. The failed locks seem to make sense also with > the error "No such file or directory" (the old name is no longer in use). > I see 2 potential problems here. The first is that self-heals are > occasionally being triggered during a copy when everything is up and > functioning properly. The 2nd issue seems to be that during a self-heal, > there is a race condition where a file can be renamed and causes these > errors below. > > Running 1.4RC3, logs and volfiles are below. Nothing of mention was in > glusterfs.log except on brick1. Please let me know what i can do to help. > > server x.x.x.102 (brick1) > ==> glusterfsd.log <== > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths > differ for inode(19824657): client path = > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path = > /mobygames/htdocs/images/i/22/23/139623.png > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths > differ for inode(19824657): client path = > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path = > /mobygames/htdocs/images/i/22/23/139623.png > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths > differ for inode(19824657): client path = > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path = > /mobygames/htdocs/images/i/22/23/139623.png > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths > differ for inode(19824657): client path = > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path = > /mobygames/htdocs/images/i/22/23/139623.png > > ==> glusterfs.log <== > 2008-12-18 11:16:20 W [afr-self-heal-common.c:1005:afr_self_heal] afr-ns: > performing self heal on /mobygames/htdocs/images/i/22/23/.139623.png.S254kc > (metadata=1 data=1 entry=1) > 2008-12-18 11:16:20 E [afr-self-heal-metadata.c:686:afr_sh_metadata_lk_cbk] > afr-ns: locking of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc on > child 3 failed: No such file or directory > 2008-12-18 11:16:20 E [afr-self-heal-metadata.c:686:afr_sh_metadata_lk_cbk] > afr-ns: locking of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc on > child 1 failed: No such file or directory > 2008-12-18 11:16:20 E [afr-self-heal-data.c:900:afr_sh_data_lock_cbk] > afr-ns: locking of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc on > child 3 failed: No such file or directory > 2008-12-18 11:16:20 E [afr-self-heal-data.c:900:afr_sh_data_lock_cbk] > afr-ns: locking of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc on > child 1 failed: No such file or directory > 2008-12-18 11:16:20 E [afr-self-heal-data.c:182:afr_sh_data_unlck_cbk] > afr-ns: locking inode of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc > on child 1 failed: No such file or directory > 2008-12-18 11:16:20 E [afr-self-heal-data.c:182:afr_sh_data_unlck_cbk] > afr-ns: locking inode of /mobygames/htdocs/images/i/22/23/.139623.png.S254kc > on child 3 failed: No such file or directory > > > > server x.x.x.106 (brick2) > ==> glusterfsd.log <== > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server: > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns > returning error: -1 (2) > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server: > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns > returning error: -1 (2) > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server: > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns > returning error: -1 (2) > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server: > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns > returning error: -1 (2) > > > server x.x.x107 (brick3) > ==> glusterfsd.log <== > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths > differ for inode(5570577): client path = > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path = > /mobygames/htdocs/images/i/22/23/139623.png > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths > differ for inode(5570577): client path = > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path = > /mobygames/htdocs/images/i/22/23/139623.png > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths > differ for inode(5570577): client path = > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path = > /mobygames/htdocs/images/i/22/23/139623.png > 2008-12-18 11:16:20 E [server-helpers.c:98:server_loc_fill] brick-ns: paths > differ for inode(5570577): client path = > /mobygames/htdocs/images/i/22/23/.139623.png.S254kc. dentry path = > /mobygames/htdocs/images/i/22/23/139623.png > > server x.x.x.103 (brick4) > ==> glusterfsd.log <== > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server: > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns > returning error: -1 (2) > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server: > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns > returning error: -1 (2) > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server: > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns > returning error: -1 (2) > 2008-12-18 11:16:20 E [server-protocol.c:3226:server_stub_resume] server: > INODELK (/mobygames/htdocs/images/i/22/23/.139623.png.S254kc) on brick-ns > returning error: -1 (2) > > > VOLFILES > > ****Server**** > > volume brick-store > type storage/posix > option directory /moby/gluster/ > end-volume > > volume brick > type features/locks > subvolumes brick-store > end-volume > > volume brick-store-ns > type storage/posix > option directory /moby/gluster-ns/ > end-volume > > volume brick-ns > type features/locks > subvolumes brick-store-ns > end-volume > > volume server > type protocol/server > option transport-type tcp/server > subvolumes brick brick-ns > option auth.addr.brick.allow x.x.x.* # Allow access to "brick" volume > option auth.addr.brick-ns.allow x.x.x.* # Allow access to "brick-ns" volume > end-volume > > > ****Client**** > > volume brick1 > type protocol/client > option transport-type tcp/client > option remote-host x.x.x.102 > option remote-subvolume brick > end-volume > > volume brick2 > type protocol/client > option transport-type tcp/client > option remote-host x.x.x.106 > option remote-subvolume brick > end-volume > > volume brick3 > type protocol/client > option transport-type tcp/client > option remote-host x.x.x.107 > option remote-subvolume brick > end-volume > > volume brick4 > type protocol/client > option transport-type tcp/client > option remote-host x.x.x.103 > option remote-subvolume brick > end-volume > > volume brick-ns1 > type protocol/client > option transport-type tcp/client > option remote-host x.x.x.102 > option remote-subvolume brick-ns > end-volume > > volume brick-ns2 > type protocol/client > option transport-type tcp/client > option remote-host x.x.x.106 > option remote-subvolume brick-ns > end-volume > > volume brick-ns3 > type protocol/client > option transport-type tcp/client > option remote-host x.x.x.107 > option remote-subvolume brick-ns > end-volume > > volume brick-ns4 > type protocol/client > option transport-type tcp/client > option remote-host x.x.x.103 > option remote-subvolume brick-ns > end-volume > > volume afr1 > type cluster/afr > subvolumes brick1 brick3 > end-volume > > volume afr2 > type cluster/afr > subvolumes brick2 brick4 > end-volume > > volume afr-ns > type cluster/afr > subvolumes brick-ns1 brick-ns2 brick-ns3 brick-ns4 > end-volume > > volume unify > type cluster/unify > option namespace afr-ns > option scheduler rr > subvolumes afr1 afr2 > end-volume > > > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@xxxxxxxxxx > http://lists.nongnu.org/mailman/listinfo/gluster-devel > > > > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@xxxxxxxxxx > http://lists.nongnu.org/mailman/listinfo/gluster-devel >
