On 09/16/2011 06:36 PM, Robert Krig wrote: > > Hi there. I'm new to GlusterFS. I'm currently evaluating it for > production usage. > > I have two Storage Servers which use JFS as a filesystem for the > underlying export. > > The setup is supposed to be replicated. > > I've been experimenting with various settings for benchmarking and such, > as well as trying out different failure scenarios. > > Anyways, the export directory on node 1 is out of sync with node 2. > So I mounted the storage volume via glusterfs client on node1 in another > directory. > > The fuse mounted directory is /storage > > As per Manual I tried doing the "find <gluster-mount> -noleaf -print0 | > xargs --null stat >/dev/null" dance, however the logs throw a bunch of > errors: > ######################################################################### > [2011-09-16 18:29:33.759729] E > [client3_1-fops.c:1216:client3_1_inodelk_cbk] 0-GLSTORAGE-client-0: error > [2011-09-16 18:29:33.759747] I > [client3_1-fops.c:1226:client3_1_inodelk_cbk] 0-GLSTORAGE-client-0: > remote operation failed: Invalid argument > [2011-09-16 18:29:33.759942] E > [afr-self-heal-metadata.c:672:afr_sh_metadata_post_nonblocking_inodelk_cbk] > 0-GLSTORAGE-replicate-0: Non Blocking metadata inodelks failed for /. > [2011-09-16 18:29:33.759961] E > [afr-self-heal-metadata.c:674:afr_sh_metadata_post_nonblocking_inodelk_cbk] > 0-GLSTORAGE-replicate-0: Metadata self-heal failed for /. > [2011-09-16 18:29:33.760167] W [rpc-common.c:64:xdr_to_generic] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x7d) [0x7f4702a751ad] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) > [0x7f4702a74de5] > (-->/usr/local/lib/glusterfs/3.2.3/xlator/protocol/client.so(client3_1_entrylk_cbk+0x52) > [0x7f46ff88a572]))) 0-xdr: XDR decoding failed > [2011-09-16 18:29:33.760200] E > [client3_1-fops.c:1292:client3_1_entrylk_cbk] 0-GLSTORAGE-client-0: error > [2011-09-16 18:29:33.760215] I > [client3_1-fops.c:1303:client3_1_entrylk_cbk] 0-GLSTORAGE-client-0: > remote operation failed: Invalid argument > [2011-09-16 18:29:33.760417] E > [afr-self-heal-entry.c:2292:afr_sh_post_nonblocking_entry_cbk] > 0-GLSTORAGE-replicate-0: Non Blocking entrylks failed for /. > [2011-09-16 18:29:33.760447] E > [afr-self-heal-common.c:1554:afr_self_heal_completion_cbk] > 0-GLSTORAGE-replicate-0: background meta-data entry self-heal failed on / > [2011-09-16 18:29:33.760808] I > [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: > remote operation failed: Invalid argument > ################################################################################### > > > Is this normal? The directory in question already has 150GB of data, so > the find command is still running. Will it be ok once it finishes? > from what I understand from the manual, the files should repair as the > find process runs, or did I misinterpret that? > > If self heal should fail, is there a failsafe method to ensure that both > nodes are in sync again? > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > Well, the find process has finished in the meantime, and as expected, it didn't fix anything. here are the last few lines of the client mount log: ###################################################################################### 2011-09-16 18:48:45.287954] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.288394] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.288921] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.289535] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.290063] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.290649] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 18:48:45.291126] I [client3_1-fops.c:2228:client3_1_lookup_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument [2011-09-16 20:14:52.289901] W [rpc-common.c:64:xdr_to_generic] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x7d) [0x7f4702a751ad] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7f4702a74de5] (-->/usr/local/lib/glusterfs/3.2.3/xlator/protocol/client.so(client3_1_statfs_cbk+0x71) [0x7f46ff88b741]))) 0-xdr: XDR decoding failed [2011-09-16 20:14:52.289928] E [client3_1-fops.c:624:client3_1_statfs_cbk] 0-GLSTORAGE-client-0: error [2011-09-16 20:14:52.289939] I [client3_1-fops.c:637:client3_1_statfs_cbk] 0-GLSTORAGE-client-0: remote operation failed: Invalid argument #######################################################################################
