Thank you Pranith Kumar K. ------172.23.0.1------------------------------- # getfattr -d -m . /home/syncdata/ getfattr: Removing leading '/' from absolute path names # file: home/syncdata trusted.afr.syncdata-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.syncdata-client-1=0sAAAAAAAAAAAAAAAA trusted.gfid=0sAAAAAAAAAAAAAAAAAAAAAQ== trusted.glusterfs.quota.dirty=0sMAA= trusted.glusterfs.quota.size=0sAAAAAAAAAAA= trusted.glusterfs.test="working\000" # getfattr -d -m . /home/syncdata/testdata/ getfattr: Removing leading '/' from absolute path names # file: home/syncdata/testdata trusted.afr.syncdata-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.syncdata-client-1=0sAAAAAAAAAAAAAAAA trusted.gfid=0st0UDRLu7TEqt2W8wc30mCQ== ------172.23.0.2------------------------------- # getfattr -d -m . /home/syncdata/ getfattr: Removing leading '/' from absolute path names # file: home/syncdata trusted.afr.syncdata-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.syncdata-client-1=0sAAAAAAAAAAAAAAAA trusted.gfid=0sAAAAAAAAAAAAAAAAAAAAAQ== trusted.glusterfs.quota.dirty=0sMAA= trusted.glusterfs.quota.size=0sAAAAAAAAAAA= trusted.glusterfs.test="working\000" # getfattr -d -m . /home/syncdata/testdata/ getfattr: Removing leading '/' from absolute path names # file: home/syncdata/testdata trusted.afr.syncdata-client-0=0sAAAAAAAAAAAAAAAA trusted.afr.syncdata-client-1=0sAAAAAAAAAAAAAAAA trusted.gfid=0shkqegy6JT0KgZjAlx3Db0w== thanks. 2011/8/24 Pranith Kumar K <pranithk at gluster.com>: > hi siga hiro, > ? ? Can you provide the output of: > getfattr -d -m . /home/syncdata > getfattr -d -m . /home/syncdata/testdata > > On both the machines. > Pranith > > On 08/24/2011 02:11 PM, siga hiro wrote: >> >> Thank you for the quick answer. >> >>> 1) http://bugs.gluster.com/show_bug.cgi?id=2921 (most likely this) >> >> Isn't this solved in GlusterFS 3.2.3? >> >> I have installed GlusterFS 3.2.3 in 172.23.0.2. >> (get from >> http://download.gluster.com/pub/gluster/glusterfs/LATEST/CentOS/) >> And It confirmed that md5sum corresponded with 172.23.0.1 and 172.23.0.2. >> # md5sum * >> 8012eaf68e8ee8153d1b4f317dea385d ?error_log.txt >> 88f70311135f82578a69866bce0564ba ?error.log >> >> mount 172.23.0.2 >> ? -> ?mount -t glusterfs -o tcp,soft,timeo=3 172.23.0.2:/syncdata >> /syncdata >> >> But... >> [root at 172.23.0.2 /]# ls -al /syncdata/testdata/ >> ls: reading directory /syncdata/testdata/: Input/output error >> >> /var/log/glusterfs/nfs.log >> [2011-08-24 17:06:14.447688] I [rpc-clnt.c:1531:rpc_clnt_reconfig] >> 0-syncdata-client-0: changing port to 24009 (from 0) >> [2011-08-24 17:06:17.453688] I >> >> [client-handshake.c:1082:select_server_supported_programs]0-syncdata-client-1: >> Using Program GlusterFS-3.1.0, Num (1298437), Version (310) >> [2011-08-24 17:06:17.456448] I >> [client-handshake.c:913:client_setvolume_cbk] 0-syncdata-client-1: >> Connected to 172.23.11.121:24009, attached to remote volume >> '/home/syncdata'. >> [2011-08-24 17:06:17.456517] I [afr-common.c:2611:afr_notify] >> 0-syncdata-replicate-0: Subvolume 'syncdata-client-1' came back up; >> going online. >> [2011-08-24 17:06:17.456957] I >> >> [client-handshake.c:1082:select_server_supported_programs]0-syncdata-client-0: >> Using Program GlusterFS-3.1.0, Num (1298437), Version (310) >> [2011-08-24 17:06:17.457937] I >> [client-handshake.c:913:client_setvolume_cbk] 0-syncdata-client-0: >> Connected to 172.23.3.4:24009, attached to remote volume >> '/home/syncdata'. >> [2011-08-24 17:06:17.458478] I [afr-common.c:912:afr_fresh_lookup_cbk] >> 0-syncdata-replicate-0: added root inode >> [2011-08-24 17:06:52.479588] W >> [afr-common.c:656:afr_lookup_self_heal_check] 0-syncdata-replicate-0: >> /fastask: gfid different on subvolume >> [2011-08-24 17:06:52.480560] I >> [client3_1-fops.c:411:client3_1_stat_cbk] 0-syncdata-client-0: remote >> operation failed: No such file or directory >> [2011-08-24 17:06:52.481555] I >> [client3_1-fops.c:1099:client3_1_access_cbk] 0-syncdata-client-0: >> remote operation failed: No such file or directory >> [2011-08-24 17:06:52.482554] I >> [client3_1-fops.c:2132:client3_1_opendir_cbk] 0-syncdata-client-0: >> remote operation failed: No such file or directory >> [2011-08-24 17:06:52.482577] W >> [client3_1-fops.c:5136:client3_1_readdir] 0-syncdata-client-0: >> (689897478): failed to get fd ctx. EBADFD >> [2011-08-24 17:06:52.482592] W >> [client3_1-fops.c:5201:client3_1_readdir] 0-syncdata-client-0: failed >> to send the fop: File descriptor in bad state >> [2011-08-24 17:06:52.482608] I >> [afr-dir-read.c:120:afr_examine_dir_readdir_cbk] >> 0-syncdata-replicate-0: /fastask: failed to do opendir on >> syncdata-client-0 >> [2011-08-24 17:06:52.482811] I >> [afr-dir-read.c:174:afr_examine_dir_readdir_cbk] >> 0-syncdata-replicate-0: ?entry self-heal triggered. path: /fastask, >> reason: checksums of directory differ, forced merge option set >> [2011-08-24 17:06:52.483553] I >> [client3_1-fops.c:1303:client3_1_entrylk_cbk] 0-syncdata-client-0: >> remote operation failed: No such file or directory >> [2011-08-24 17:06:52.483642] E >> [afr-self-heal-entry.c:2292:afr_sh_post_nonblocking_entry_cbk] >> 0-syncdata-replicate-0: Non Blocking entrylks failed for /fastask. >> [2011-08-24 17:06:52.483839] W [afr-common.c:122:afr_set_split_brain] >> >> (-->/opt/glusterfs/3.2.3/lib64/glusterfs/3.2.3/xlator/cluster/replicate.so(afr_sh_post_nonblocking_entry_cbk+0xf5) >> [0x2aaaaad137f5] >> >> (-->/opt/glusterfs/3.2.3/lib64/glusterfs/3.2.3/xlator/cluster/replicate.so(afr_sh_entry_done+0x46) >> [0x2aaaaad13646] >> >> (-->/opt/glusterfs/3.2.3/lib64/glusterfs/3.2.3/xlator/cluster/replicate.so(afr_self_heal_completion_cbk+0x246) >> [0x2aaaaad0cac6]))) 0-syncdata-replicate-0: invalid argument: inode >> [2011-08-24 17:06:52.483864] E >> [afr-self-heal-common.c:1554:afr_self_heal_completion_cbk] >> 0-syncdata-replicate-0: background ?entry entry self-heal failed on >> /fastask >> [2011-08-24 17:06:52.483898] W >> [client3_1-fops.c:5253:client3_1_readdirp] 0-syncdata-client-0: >> (689897478): failed to get fd ctx. EBADFD >> [2011-08-24 17:06:52.483913] W >> [client3_1-fops.c:5317:client3_1_readdirp] 0-syncdata-client-0: failed >> to send the fop: File descriptor in bad state >> >> thanks. >> >>> hi siga hiro, >>> ? ?I see the following warning: >>> [2011-08-24 11:36:04.695145] W >>> [afr-common.c:656:afr_lookup_self_heal_check] >>> 0-syncdata-replicate-0: /testdata: gfid different on subvolume >>> >>> I also see that you have more than one mount on the volume. Most probably >>> you are running into one of the following bugs: >>> 1) http://bugs.gluster.com/show_bug.cgi?id=2921 (most likely this) >>> 2) http://bugs.gluster.com/show_bug.cgi?id=2745 >>> >>> If it is not the bug 2745, you can confirm it is the bug 2921 if the >>> md5sums >>> on the files match on both the machines 172.23.0.1, 172.23.0.2 >>> >>> pranith. >>> >>> On 08/24/2011 11:48 AM, siga hiro wrote: >>> >>> Hi, everyone. >>> Its nice meeting you. >>> I am poor at English.... >>> >>> I am writing this because I'd like to update GlusterFS to 3.2.2-1,and I >>> want >>> to change from gluster mount to nfs mount. >>> >>> I have installed GlusterFS 3.2.1 one week ago,and replication 2 server. >>> >>> OS:CentOS5.5 64bit >>> RPM:glusterfs-core-3.2.1-1 >>> ? ? glusterfs-fuse-3.2.1-1 >>> >>> command >>> ?gluster volume create syncdata replica 2 ?transport tcp >>> 172.23.0.1:/home/syncdata 172.23.0.2:/home/syncdata >>> >>> mount command >>> ?172.23.0.1 -> ?mount -t glusterfs -o tcp,soft,timeo=3 >>> 172.23.0.1:/syncdata >>> /syncdata >>> ?172.23.0.2 -> ?mount -t glusterfs -o tcp,soft,timeo=3 >>> 172.23.0.2:/syncdata >>> /syncdata >>> >>> So,Yesterday I update GlusterFS to 3.2.2-1 and use nfs mount. >>> ?172.23.0.2 -> ?mount -t nfs ?-o nolock,nfsvers=3,tcp,hard,intr >>> 172.23.0.2:/syncdata /syncdata >>> >>> [root at 172.23.0.2 /]# ls -al /syncdata/testdata/ >>> ls: reading directory /syncdata/testdata/: Input/output error >>> >>> /var/log/glusterfs/nfs.log >>> [2011-08-24 11:35:16.319379] I >>> [client-handshake.c:1082:select_server_supported_programs] >>> 0-syncdata-client-1: Using Program GlusterFS-3.1.0, Num (1298437), >>> Version >>> (310) >>> [2011-08-24 11:35:16.322126] I >>> [client-handshake.c:913:client_setvolume_cbk] >>> 0-syncdata-client-1: Connected to 172.23.0.2:24009, attached to remote >>> volume '/home/syncdata'. >>> [2011-08-24 11:35:16.322191] I [afr-common.c:2611:afr_notify] >>> 0-syncdata-replicate-0: Subvolume 'syncdata-client-1' came back up; going >>> online. >>> [2011-08-24 11:35:16.323281] I >>> [client-handshake.c:1082:select_server_supported_programs] >>> 0-syncdata-client-0: Using Program GlusterFS-3.1.0, Num (1298437), >>> Version >>> (310) >>> [2011-08-24 11:35:16.324274] I >>> [client-handshake.c:913:client_setvolume_cbk] >>> 0-syncdata-client-0: Connected to 172.23.0.1:24009, attached to remote >>> volume '/home/syncdata'. >>> [2011-08-24 11:35:16.324801] I [afr-common.c:912:afr_fresh_lookup_cbk] >>> 0-syncdata-replicate-0: added root inode >>> [2011-08-24 11:36:04.695145] W >>> [afr-common.c:656:afr_lookup_self_heal_check] >>> 0-syncdata-replicate-0: /testdata: gfid different on subvolume >>> [2011-08-24 11:36:04.696121] I [client3_1-fops.c:411:client3_1_stat_cbk] >>> 0-syncdata-client-0: remote operation failed: No such file or directory >>> [2011-08-24 11:36:04.697121] I >>> [client3_1-fops.c:1099:client3_1_access_cbk] >>> 0-syncdata-client-0: remote operation failed: No such file or directory >>> [2011-08-24 11:36:04.698118] I >>> [client3_1-fops.c:2132:client3_1_opendir_cbk] >>> 0-syncdata-client-0: remote operation failed: No such file or directory >>> [2011-08-24 11:36:04.698140] W [client3_1-fops.c:5136:client3_1_readdir] >>> 0-syncdata-client-0: (689897478): failed to get fd ctx. EBADFD >>> [2011-08-24 11:36:04.698155] W [client3_1-fops.c:5201:client3_1_readdir] >>> 0-syncdata-client-0: failed to send the fop: File descriptor in bad state >>> [2011-08-24 11:36:04.698168] I >>> [afr-dir-read.c:120:afr_examine_dir_readdir_cbk] 0-syncdata-replicate-0: >>> /fastask: failed to do opendir on syncdata-client-0 >>> >>> # gluster volume info all >>> >>> Volume Name: syncdata >>> Type: Replicate >>> Status: Started >>> Number of Bricks: 2 >>> Transport-type: tcp >>> Bricks: >>> Brick1: 172.23.0.1:/home/syncdata >>> Brick2: 172.23.0.2:/home/syncdata >>> >>> >>> After an 172.23.0.2 server is made to work as usual, I want to do the >>> work >>> of the 172.23.0.1 server. >>> >>> Any ideas? >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >>> >>> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users > >
